Skip to content

Mute chatty xet in migration service #594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
seanses merged 4 commits into from
Dec 12, 2025
Merged

Mute chatty xet in migration service #594

seanses merged 4 commits into from
Dec 12, 2025

Conversation

@seanses
Copy link
Collaborator

@seanses seanses commented Dec 11, 2025 •
edited
Loading

The repo scanner migration services calls

xet-core/data/src/migration_tool/migrate.rs

Line 29 in 23f691b

pub async fn migrate_with_external_runtime(
with tracing subscriber set to logging at INFO level. As a result, xet-core has been too chatty for repo scanner because we log at INFO level at many places because we wanted to log more info in hf-xet.

This PR employs a build feature elevated_information_level which is enabled only for hf-xet, such that the information we want to log in hf-xet are still emitted at INFO level, but for others emitted at DEBUG level, so not to clutter the repo scanner log. It's not viable to pass in a tracing::Level variable to use at event!() because the macro expects a constant level value.

Log size of a test that uploads a 10 MB file:
With migrate_with_external_runtime:

{"timestamp":"2025-12-11T00:21:06.665382Z","level":"INFO","fields":{"message":"HTTP client configured","idle_timeout":"60s","max_idle_connections":16,"user_agent":"Some(\"hf_xet/1.2.1\")"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":126,"span":{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:21:06.665518Z","level":"INFO","fields":{"message":"HTTP client configured","idle_timeout":"60s","max_idle_connections":16,"user_agent":"Some(\"hf_xet/1.2.1\")"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":126,"span":{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:21:06.665553Z","level":"INFO","fields":{"message":"HTTP client configured","idle_timeout":"60s","max_idle_connections":16,"user_agent":"Some(\"hf_xet/1.2.1\")"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":126,"span":{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:21:06.665641Z","level":"INFO","fields":{"message":"Fixing maximum concurrency for upload at 8; adaptive concurrency disabled."},"filename":"/Users/di/hf/xet-core/cas_client/src/adaptive_concurrency/controller.rs","line_number":268,"span":{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:21:06.842576Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":true},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":204,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:21:06.842729Z","level":"INFO","fields":{"message":"Token refresh successful for CAS authentication"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":291,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:21:07.638856Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"01KC5CGX1QBH63PWF93W4FB04K","status_code":404},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":251,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:21:07.639050Z","level":"INFO","fields":{"message":"Fatal Error: \"cas::query_dedup\" api call failed (request id 01KC5CGX1QBH63PWF93W4FB04K): HTTP status client error (404 Not Found) for url (https://cas-server.xethub.hf.co/v1/chunks/default/2749288302764cba60f2c885e31e785d1b6bfa89de9e427800aea3572b185afb)"},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":132,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:21:07.645169Z","level":"INFO","fields":{"action":"clean","file_name":"/Users/di/tt/download/aa/zz","file_size_count":10485760,"new_bytes_count":10485760,"start_ts":"2025-12-11T00:21:06.680556+00:00","end_processing_ts":"2025-12-11T00:21:07.645165+00:00"},"filename":"/Users/di/hf/xet-core/data/src/file_cleaner.rs","line_number":175,"span":{"file_name":"/Users/di/tt/download/aa/zz","name":"FileCleaner::finish"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"file_name":"/Users/di/tt/download/aa/zz","name":"FileCleaner::finish"}]}
{"timestamp":"2025-12-11T00:21:07.727015Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":204,"span":{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","name":"FileUploadSession::upload_xorb_task"},{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:21:07.727195Z","level":"INFO","fields":{"message":"Token refresh successful for CAS authentication"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":291,"span":{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","name":"FileUploadSession::upload_xorb_task"},{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:21:10.351108Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"01KC5CGXCF6PTEKJPP9JVXPGZJ","status_code":200},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":251,"span":{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","name":"FileUploadSession::upload_xorb_task"},{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:21:10.351403Z","level":"INFO","fields":{"message":"Request Success: cas::upload_xorb api call succeeded (request id 01KC5CGXCF6PTEKJPP9JVXPGZJ)."},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":165,"span":{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","name":"FileUploadSession::upload_xorb_task"},{"key":"default/1a7497652e255f98785e725528a86c7e4c35ed8777ecceb55089a082115f50fb","xorb.len":10487176,"xorb.num_chunks":177,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:21:10.356387Z","level":"INFO","fields":{"message":"Shard manager flushed new shard to \"/Users/di/.cache/huggingface/xet/https___cas_serv-tGqkUaZf_CBPHQ6h/staging/shard-session/.tmpbJN77b/d3384a0fa312ab06e24c024724ff0b40595847bb0c91b504b606feb71200b857.mdb\"."},"filename":"/Users/di/hf/xet-core/mdb_shard/src/shard_file_manager.rs","line_number":518,"span":{"name":"ShardFileManager::flush"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"ShardFileManager::flush"}]}
{"timestamp":"2025-12-11T00:21:10.358392Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":204,"span":{"shard.len":8880,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8880,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:21:10.358460Z","level":"INFO","fields":{"message":"Token refresh successful for CAS authentication"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":291,"span":{"shard.len":8880,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8880,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:21:10.764815Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"01KC5CH00E6BKF3ZHXKKFNQRM2","status_code":200},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":251,"span":{"shard.len":8880,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8880,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:21:10.765192Z","level":"INFO","fields":{"message":"Request Success: cas::upload_shard api call succeeded (request id 01KC5CH00E6BKF3ZHXKKFNQRM2)."},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":165,"span":{"shard.len":8880,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8880,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:21:10.765631Z","level":"INFO","fields":{"message":"Shard d3384a0fa312ab06e24c024724ff0b40595847bb0c91b504b606feb71200b857 upload + sync completed successfully."},"filename":"/Users/di/hf/xet-core/data/src/shard_interface.rs","line_number":284,"span":{"name":"shard_session::upload_shard_task"},"spans":[{"num_files":1,"session_id":"01KC5CGVTWRRSDW8RSR7Z74MA1","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"}]}

With hf-xet:

{"timestamp":"2025-12-11T00:20:15.885922Z","level":"INFO","fields":{"message":"python/3.13.3; huggingface_hub/1.2.1; , xet-core revision v1.0.4-rc0-221-g890c061"},"filename":"/Users/di/hf/xet-core/xet_logging/src/logging.rs","line_number":65}
{"timestamp":"2025-12-11T00:20:15.886172Z","level":"INFO","fields":{"message":"starting log cleanup in \"/Users/di/.cache/huggingface/xet/logs\" (min_age=86400s, max_retention=1209600s, max_size=250MB bytes)"},"filename":"/Users/di/hf/xet-core/xet_logging/src/logging.rs","line_number":221}
{"timestamp":"2025-12-11T00:20:15.886234Z","level":"INFO","fields":{"message":"hf_xet logging cofigured."},"filename":"src/logging.rs","line_number":43}
{"timestamp":"2025-12-11T00:20:15.886259Z","level":"INFO","fields":{"message":"Increased RLIMIT_NOFILE soft limit from 256 to 9223372036854775807"},"filename":"/Users/di/hf/xet-core/xet_runtime/src/file_handle_limits.rs","line_number":31}
{"timestamp":"2025-12-11T00:20:15.904782Z","level":"INFO","fields":{"message":"Log Directory Cleanup: found 926B of logs in 1 log files, with 0B in 0 files eligible for deletion."},"filename":"/Users/di/hf/xet-core/xet_logging/src/logging.rs","line_number":299}
{"timestamp":"2025-12-11T00:20:15.904843Z","level":"INFO","fields":{"message":"Log Directory Cleanup: deleted 0B in 0 files"},"filename":"/Users/di/hf/xet-core/xet_logging/src/logging.rs","line_number":363}
{"timestamp":"2025-12-11T00:20:16.198360Z","level":"INFO","fields":{"message":"Config: min_spacing_between_global_dedup_queries = 256 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198459Z","level":"INFO","fields":{"message":"Config: local_cas_scheme = \"local://\" (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198490Z","level":"INFO","fields":{"message":"Config: max_concurrent_file_ingestion = 8 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198508Z","level":"INFO","fields":{"message":"Config: max_concurrent_file_downloads = 8 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198527Z","level":"INFO","fields":{"message":"Config: ingestion_block_size = 8MB (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198545Z","level":"INFO","fields":{"message":"Config: progress_update_interval = 200ms (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198562Z","level":"INFO","fields":{"message":"Config: progress_update_speed_sampling_window = 10s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198578Z","level":"INFO","fields":{"message":"Config: session_xorb_metadata_flush_interval = 20s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198594Z","level":"INFO","fields":{"message":"Config: session_xorb_metadata_flush_max_count = 64 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198613Z","level":"INFO","fields":{"message":"Config: default_cas_endpoint = \"http://localhost:8080\" (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198631Z","level":"INFO","fields":{"message":"Config: target_size = 67108864 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198665Z","level":"INFO","fields":{"message":"Config: max_target_size = 67108864 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198682Z","level":"INFO","fields":{"message":"Config: cache_size_limit = 16GB (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198698Z","level":"INFO","fields":{"message":"Config: chunk_index_table_max_size = 67108864 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198712Z","level":"INFO","fields":{"message":"Config: nranges_in_streaming_fragmentation_estimator = 128 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198729Z","level":"INFO","fields":{"message":"Config: min_n_chunks_per_range_hysteresis_factor = 0.5 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198747Z","level":"INFO","fields":{"message":"Config: min_n_chunks_per_range = 8.0 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198762Z","level":"INFO","fields":{"message":"Config: size_bytes = 0 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198777Z","level":"INFO","fields":{"message":"Config: retry_max_attempts = 5 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198795Z","level":"INFO","fields":{"message":"Config: retry_base_delay = 3s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198816Z","level":"INFO","fields":{"message":"Config: retry_max_duration = 360s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198833Z","level":"INFO","fields":{"message":"Config: idle_connection_timeout = 60s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198869Z","level":"INFO","fields":{"message":"Config: max_idle_connections = 16 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198895Z","level":"INFO","fields":{"message":"Config: connect_timeout = 60s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198910Z","level":"INFO","fields":{"message":"Config: read_timeout = 120s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198926Z","level":"INFO","fields":{"message":"Config: num_concurrent_range_gets = 48 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198941Z","level":"INFO","fields":{"message":"Config: upload_reporting_block_size = 524288 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198957Z","level":"INFO","fields":{"message":"Config: reconstruct_write_sequentially = false (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198973Z","level":"INFO","fields":{"message":"Config: num_range_in_segment_base = 16 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.198988Z","level":"INFO","fields":{"message":"Config: num_range_in_segment_delta = 1 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199004Z","level":"INFO","fields":{"message":"Config: num_range_in_segment_max = 400 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199019Z","level":"INFO","fields":{"message":"Config: enable_adaptive_concurrency = false (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199034Z","level":"INFO","fields":{"message":"Config: fixed_concurrency_max_uploads = 8 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199048Z","level":"INFO","fields":{"message":"Config: fixed_concurrency_max_downloads = 8 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199064Z","level":"INFO","fields":{"message":"Config: ac_min_adjustment_window_ms = 500 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199082Z","level":"INFO","fields":{"message":"Config: ac_min_bytes_required_for_adjustment = 20MB (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199098Z","level":"INFO","fields":{"message":"Config: ac_num_transmissions_required_for_adjustment = 1 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199113Z","level":"INFO","fields":{"message":"Config: ac_latency_rtt_half_life = 64.0 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199128Z","level":"INFO","fields":{"message":"Config: ac_success_tracking_half_life = 8.0 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199146Z","level":"INFO","fields":{"message":"Config: ac_target_rtt = 60s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199159Z","level":"INFO","fields":{"message":"Config: ac_max_healthy_rtt = 90s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199175Z","level":"INFO","fields":{"message":"Config: ac_rtt_success_max_quantile = 0.95 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199202Z","level":"INFO","fields":{"message":"Config: ac_healthy_success_ratio_threshold = 0.8 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199222Z","level":"INFO","fields":{"message":"Config: ac_unhealthy_success_ratio_threshold = 0.5 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199238Z","level":"INFO","fields":{"message":"Config: ac_target_rtt_transmission_size = 67108864 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199285Z","level":"INFO","fields":{"message":"Config: ac_logging_interval_ms = 10000 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199329Z","level":"INFO","fields":{"message":"Config: ac_max_upload_concurrency = 64 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199354Z","level":"INFO","fields":{"message":"Config: ac_min_upload_concurrency = 1 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199377Z","level":"INFO","fields":{"message":"Config: ac_initial_upload_concurrency = 1 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199400Z","level":"INFO","fields":{"message":"Config: ac_max_download_concurrency = 64 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199423Z","level":"INFO","fields":{"message":"Config: ac_min_download_concurrency = 1 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199441Z","level":"INFO","fields":{"message":"Config: ac_initial_download_concurrency = 1 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199465Z","level":"INFO","fields":{"message":"Config: dest = None (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199487Z","level":"INFO","fields":{"message":"Config: format = None (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199525Z","level":"INFO","fields":{"message":"Config: prefix = \"xet\" (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199551Z","level":"INFO","fields":{"message":"Config: dir_disable_cleanup = false (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199579Z","level":"INFO","fields":{"message":"Config: dir_max_size = 250MB (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199604Z","level":"INFO","fields":{"message":"Config: dir_min_deletion_age = 86400s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199628Z","level":"INFO","fields":{"message":"Config: dir_max_retention_age = 1209600s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199651Z","level":"INFO","fields":{"message":"Config: compression_scheme_retest_interval = 32 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.199845Z","level":"INFO","fields":{"message":"Using 12 async threads for tokio runtime"},"filename":"/Users/di/hf/xet-core/xet_runtime/src/runtime.rs","line_number":56}
{"timestamp":"2025-12-11T00:20:16.201876Z","level":"INFO","fields":{"message":"Using cache path \"/Users/di/.cache/huggingface/xet\"."},"filename":"/Users/di/hf/xet-core/data/src/data_client.rs","line_number":47,"span":{"num_files":1,"name":"data_client::upload_files"},"spans":[{"num_files":1,"name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:20:16.202849Z","level":"INFO","fields":{"message":"Chunk cache size set to 0, disabling chunk cache"},"filename":"/Users/di/hf/xet-core/cas_client/src/remote_client.rs","line_number":201,"span":{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:20:16.345534Z","level":"INFO","fields":{"message":"HTTP client configured","idle_timeout":"60s","max_idle_connections":16,"user_agent":"Some(\"hf_xet/1.2.1\")"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":126,"span":{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:20:16.345709Z","level":"INFO","fields":{"message":"HTTP client configured","idle_timeout":"60s","max_idle_connections":16,"user_agent":"Some(\"hf_xet/1.2.1\")"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":126,"span":{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:20:16.345749Z","level":"INFO","fields":{"message":"HTTP client configured","idle_timeout":"60s","max_idle_connections":16,"user_agent":"Some(\"hf_xet/1.2.1\")"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":126,"span":{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:20:16.345846Z","level":"INFO","fields":{"message":"Fixing maximum concurrency for upload at 8; adaptive concurrency disabled."},"filename":"/Users/di/hf/xet-core/cas_client/src/adaptive_concurrency/controller.rs","line_number":268,"span":{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"}]}
{"timestamp":"2025-12-11T00:20:16.346200Z","level":"INFO","fields":{"message":"Config: MDB_SHARD_EXPIRATION_BUFFER = 604800s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37}
{"timestamp":"2025-12-11T00:20:16.360239Z","level":"INFO","fields":{"message":"Config: TARGET_CHUNK_SIZE = 65536 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37,"span":{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"}]}
{"timestamp":"2025-12-11T00:20:16.360270Z","level":"INFO","fields":{"message":"Config: MINIMUM_CHUNK_DIVISOR = 8 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37,"span":{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"}]}
{"timestamp":"2025-12-11T00:20:16.360295Z","level":"INFO","fields":{"message":"Config: MAXIMUM_CHUNK_MULTIPLIER = 2 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37,"span":{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"}]}
{"timestamp":"2025-12-11T00:20:16.524753Z","level":"INFO","fields":{"message":"Config: MDB_SHARD_GLOBAL_DEDUP_CHUNK_MODULUS = 1024 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37,"span":{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"}]}
{"timestamp":"2025-12-11T00:20:16.525184Z","level":"INFO","fields":{"message":"Starting query_dedup API call","call_id":1,"prefix":"default","chunk_hash":"48328c1471cf1d60f2cfecdb7f6c6d63c54abc2e6d14d081ab7e8a6094dadc87"},"filename":"/Users/di/hf/xet-core/cas_client/src/remote_client.rs","line_number":241,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:20:16.525274Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":true},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":204,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:20:16.525398Z","level":"INFO","fields":{"message":"Token refresh successful for CAS authentication"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":291,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:20:17.324301Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"01KC5CFBXDTV9T7MH1D8NM9KJD","status_code":404},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":251,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:20:17.324527Z","level":"INFO","fields":{"message":"Fatal Error: \"cas::query_dedup\" api call failed (request id 01KC5CFBXDTV9T7MH1D8NM9KJD): HTTP status client error (404 Not Found) for url (https://cas-server.xethub.hf.co/v1/chunks/default/48328c1471cf1d60f2cfecdb7f6c6d63c54abc2e6d14d081ab7e8a6094dadc87)"},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":132,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:20:17.324698Z","level":"INFO","fields":{"message":"Completed query_dedup API call","call_id":1,"prefix":"default","chunk_hash":"48328c1471cf1d60f2cfecdb7f6c6d63c54abc2e6d14d081ab7e8a6094dadc87","result":"not_found"},"filename":"/Users/di/hf/xet-core/cas_client/src/remote_client.rs","line_number":259,"span":{"name":"UploadSessionDataManager::dedup_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"name":"UploadSessionDataManager::dedup_task"}]}
{"timestamp":"2025-12-11T00:20:17.325973Z","level":"INFO","fields":{"message":"Config: MAX_XORB_BYTES = 67108864 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37,"span":{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"}]}
{"timestamp":"2025-12-11T00:20:17.326076Z","level":"INFO","fields":{"message":"Config: MAX_XORB_CHUNKS = 8192 (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37,"span":{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"}]}
{"timestamp":"2025-12-11T00:20:17.330749Z","level":"INFO","fields":{"action":"clean","file_name":"/Users/di/tt/download/aa/zz","file_size_count":10485760,"new_bytes_count":10485760,"start_ts":"2025-12-11T00:20:16.360320+00:00","end_processing_ts":"2025-12-11T00:20:17.330743+00:00"},"filename":"/Users/di/hf/xet-core/data/src/file_cleaner.rs","line_number":175,"span":{"file_name":"/Users/di/tt/download/aa/zz","name":"FileCleaner::finish"},"spans":[{"file.len":10485760,"file.name":"/Users/di/tt/download/aa/zz","name":"clean_file_task"},{"file_name":"/Users/di/tt/download/aa/zz","name":"FileCleaner::finish"}]}
{"timestamp":"2025-12-11T00:20:17.432794Z","level":"INFO","fields":{"message":"Starting upload_xorb API call","call_id":2,"prefix":"default","hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","size":10487032,"num_chunks":159},"filename":"/Users/di/hf/xet-core/cas_client/src/remote_client.rs","line_number":825,"span":{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","name":"FileUploadSession::upload_xorb_task"},{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:20:17.435403Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":204,"span":{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","name":"FileUploadSession::upload_xorb_task"},{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:20:17.435514Z","level":"INFO","fields":{"message":"Token refresh successful for CAS authentication"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":291,"span":{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","name":"FileUploadSession::upload_xorb_task"},{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:20:19.378461Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"01KC5CFC8Y5Q14YEW5B89FSSV5","status_code":200},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":251,"span":{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","name":"FileUploadSession::upload_xorb_task"},{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:20:19.378645Z","level":"INFO","fields":{"message":"Request Success: cas::upload_xorb api call succeeded (request id 01KC5CFC8Y5Q14YEW5B89FSSV5)."},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":165,"span":{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","name":"FileUploadSession::upload_xorb_task"},{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:20:19.378847Z","level":"INFO","fields":{"message":"Completed upload_xorb API call","call_id":2,"prefix":"default","hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","size":10487032,"result":"inserted"},"filename":"/Users/di/hf/xet-core/cas_client/src/remote_client.rs","line_number":905,"span":{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"xorb_len":10485760,"name":"FileUploadSession::register_new_xorb_for_upload"},{"xorb.hash":"d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","name":"FileUploadSession::upload_xorb_task"},{"key":"default/d5cc1c4544a210616c50fe21dbcf7d1aed997d83e3176e5cae5557587edab963","xorb.len":10487032,"xorb.num_chunks":159,"name":"RemoteClient::upload_xorb"}]}
{"timestamp":"2025-12-11T00:20:19.382154Z","level":"INFO","fields":{"message":"Shard manager flushed new shard to \"/Users/di/.cache/huggingface/xet/https___cas_serv-tGqkUaZf_CBPHQ6h/staging/shard-session/.tmp1gug7B/f3ac8cffba104d40c51c426ff8b3cd2aca6a9713b93f8abd0ed252e54efa3997.mdb\"."},"filename":"/Users/di/hf/xet-core/mdb_shard/src/shard_file_manager.rs","line_number":518,"span":{"name":"ShardFileManager::flush"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"ShardFileManager::flush"}]}
{"timestamp":"2025-12-11T00:20:19.384636Z","level":"INFO","fields":{"message":"Starting upload_shard API call","call_id":3,"size":8016},"filename":"/Users/di/hf/xet-core/cas_client/src/remote_client.rs","line_number":763,"span":{"shard.len":8016,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8016,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:20:19.384718Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":204,"span":{"shard.len":8016,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8016,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:20:19.384802Z","level":"INFO","fields":{"message":"Token refresh successful for CAS authentication"},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":291,"span":{"shard.len":8016,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8016,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:20:19.942796Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"01KC5CFE7MYNW8RMD8F5WCZHCH","status_code":200},"filename":"/Users/di/hf/xet-core/cas_client/src/http_client.rs","line_number":251,"span":{"shard.len":8016,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8016,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:20:19.943227Z","level":"INFO","fields":{"message":"Request Success: cas::upload_shard api call succeeded (request id 01KC5CFE7MYNW8RMD8F5WCZHCH)."},"filename":"/Users/di/hf/xet-core/cas_client/src/retry_wrapper.rs","line_number":165,"span":{"shard.len":8016,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8016,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:20:19.943532Z","level":"INFO","fields":{"message":"Completed upload_shard API call","call_id":3,"size":8016,"result":"sync performed"},"filename":"/Users/di/hf/xet-core/cas_client/src/remote_client.rs","line_number":793,"span":{"shard.len":8016,"name":"RemoteClient::upload_shard"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"},{"shard.len":8016,"name":"RemoteClient::upload_shard"}]}
{"timestamp":"2025-12-11T00:20:19.943673Z","level":"INFO","fields":{"message":"Shard f3ac8cffba104d40c51c426ff8b3cd2aca6a9713b93f8abd0ed252e54efa3997 upload + sync completed successfully."},"filename":"/Users/di/hf/xet-core/data/src/shard_interface.rs","line_number":284,"span":{"name":"shard_session::upload_shard_task"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"}]}
{"timestamp":"2025-12-11T00:20:19.943772Z","level":"INFO","fields":{"message":"Config: MDB_SHARD_LOCAL_CACHE_EXPIRATION = 1814400s (default)"},"filename":"/Users/di/hf/xet-core/utils/src/configuration_utils.rs","line_number":37,"span":{"name":"shard_session::upload_shard_task"},"spans":[{"num_files":1,"session_id":"01KC5CFAPAJB07T4RZ6QDGF2CQ","name":"data_client::upload_files"},{"name":"FileUploadSession::finalize"},{"name":"shard_session::upload_shard_task"}]}

@seanses seanses changed the base branch from main to di/clean-up-dead-code December 11, 2025 00:43
@seanses seanses requested review from rajatarya and sirahd December 11, 2025 00:45
@seanses seanses marked this pull request as ready for review December 11, 2025 00:48
sirahd
sirahd approved these changes Dec 11, 2025
View reviewed changes
Copy link
Collaborator

@sirahd sirahd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for fixing this.

Base automatically changed from di/clean-up-dead-code to main December 11, 2025 18:55
@seanses seanses merged commit b719b6e into main Dec 12, 2025
6 checks passed
@seanses seanses deleted the di/mute-chatty-xet branch December 12, 2025 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sirahd sirahd sirahd approved these changes

@rajatarya rajatarya Awaiting requested review from rajatarya

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@seanses @sirahd