Skip to content

Fix telemetry opt-out wiring and action sampling - #953

Open
bmehta001 wants to merge 5 commits into
mainfrom
bhamehta/flcore/1ds-telemetry-core-minimal
Open

Fix telemetry opt-out wiring and action sampling#953
bmehta001 wants to merge 5 commits into
mainfrom
bhamehta/flcore/1ds-telemetry-core-minimal

Conversation

@bmehta001

Copy link
Copy Markdown
Contributor

Summary

  • Normalize DisableNonessentialTelemetry from native configuration and additional_options.
  • Sample OpenAIAudioTranscribe action events at 2% while retaining existing sampling for other actions.
  • Add focused telemetry sampling coverage.

Validation

  • C++ RelWithDebInfo build passed.
  • Focused telemetry tests passed: 18 tests.

This PR includes the latest origin/main merge.

bmehta001 and others added 3 commits August 3, 2026 21:57
Keep the first telemetry slice focused on enable/disable controls, native 1DS initialization, startup ProcessInfo, and redacted Error logging.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 297bc539-5f7f-4a70-9411-48e91a5bf532
…ling

- Modified Manager::Initialize to normalize DisableNonessentialTelemetry from both
  config field and additional_options, ensuring opt-out from public SDKs reaches
  native initialization (C#, JS, Python SetDisableNonessentialTelemetry forwarded
  via additional_options were being ignored)

- Added action-specific sampling support:
  - Added kCoreAudioTranscribeSampleRatePercent = 2.0 constant
  - Added SampleRateForAction() to map action names to sample rates
  - Updated RecordAction() to pass action-specific rate to ShouldSampleEvent()
  - OpenAIAudioTranscribe (action 203) now sampled at 2%, all others at 100%

- Added test SamplesCoreAudioTranscribeAtTwoPercent to verify behavior

Fixes cross-language telemetry opt-out parity and implements targeted sampling
for high-volume action as requested in security/architecture review.

All 18 telemetry unit tests pass.

Files changed:
- sdk_v2/cpp/src/manager.cc: opt-out normalization at native init
- sdk_v2/cpp/src/telemetry/telemetry_sampling.h: rate constants and mapper
- sdk_v2/cpp/src/telemetry/one_ds_telemetry.cc: action-specific rate dispatch
- sdk_v2/cpp/test/internal_api/telemetry_test.cc: new sampling test

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5a9051ad-9c32-4dc3-bd66-d33d416145a4
Copilot AI balanced review requested due to automatic review settings August 5, 2026 22:22
@vercel

vercel Bot commented Aug 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
foundry-local Ready Ready Preview Aug 6, 2026 12:44am

Request Review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Wires telemetry opt-out across SDKs and adds 1DS telemetry sampling infrastructure to the C++ core.

Changes:

  • Exposes non-essential telemetry opt-out in C++, C#, Python, and JavaScript.
  • Adds 1DS metadata, device identity, redaction, lifecycle, and 2% audio-action sampling.
  • Expands telemetry tests and build dependencies.

Reviewed changes

Copilot reviewed 44 out of 44 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
sdk_v2/python/src/foundry_local_sdk/configuration.py Adds Python opt-out wiring.
sdk_v2/js/src/detail/native.ts Extends native manager options.
sdk_v2/js/src/configuration.ts Exposes JavaScript opt-out.
sdk_v2/js/native/src/manager.cc Maps JavaScript telemetry settings.
sdk_v2/cs/src/FoundryLocalManager.cs Applies C# opt-out setting.
sdk_v2/cs/src/Configuration.cs Exposes C# opt-out property.
sdk_v2/cpp/vcpkg.json Adds telemetry feature dependencies.
sdk_v2/cpp/test/internal_api/web_service_test.cc Updates telemetry test fixtures.
sdk_v2/cpp/test/internal_api/test_helpers.h Includes telemetry test support.
sdk_v2/cpp/test/internal_api/telemetry_test.cc Expands telemetry coverage.
sdk_v2/cpp/test/internal_api/session_manager_test.cc Updates telemetry fixture.
sdk_v2/cpp/test/internal_api/null_telemetry.h Removes obsolete test fake.
sdk_v2/cpp/test/internal_api/chat/chat_session_test.cc Updates chat telemetry fixture.
sdk_v2/cpp/test/internal_api/audio/audio_session_test.cc Updates audio telemetry fixtures.
sdk_v2/cpp/src/util/sha256.h Declares string hashing.
sdk_v2/cpp/src/util/sha256.cc Implements cross-platform hashing.
sdk_v2/cpp/src/telemetry/telemetry.h Expands telemetry contracts.
sdk_v2/cpp/src/telemetry/telemetry.cc Maps actions and statuses.
sdk_v2/cpp/src/telemetry/telemetry_sampling.h Implements deterministic sampling.
sdk_v2/cpp/src/telemetry/telemetry_redaction.h Adds telemetry redaction.
sdk_v2/cpp/src/telemetry/telemetry_metadata.h Defines process metadata.
sdk_v2/cpp/src/telemetry/telemetry_metadata.cc Collects platform metadata.
sdk_v2/cpp/src/telemetry/telemetry_logger.h Expands logger interface.
sdk_v2/cpp/src/telemetry/telemetry_logger.cc Formats telemetry events.
sdk_v2/cpp/src/telemetry/telemetry_environment.h Declares runtime gating.
sdk_v2/cpp/src/telemetry/telemetry_environment.cc Implements CI/env gating.
sdk_v2/cpp/src/telemetry/telemetry_action_tracker.h Adds invocation contexts.
sdk_v2/cpp/src/telemetry/telemetry_action_tracker.cc Improves action tracking.
sdk_v2/cpp/src/telemetry/one_ds_tenant_token.h.in Templates the tenant token.
sdk_v2/cpp/src/telemetry/one_ds_telemetry.h Declares the 1DS backend.
sdk_v2/cpp/src/telemetry/one_ds_telemetry.cc Implements 1DS telemetry.
sdk_v2/cpp/src/telemetry/invocation_context.h Defines correlation context.
sdk_v2/cpp/src/telemetry/invocation_context.cc Implements GUID/user-agent helpers.
sdk_v2/cpp/src/telemetry/device_id.h Defines device identity storage.
sdk_v2/cpp/src/telemetry/device_id.cc Implements platform device IDs.
sdk_v2/cpp/src/telemetry/android_telemetry_bridge.cc Adds Android readiness bridge.
sdk_v2/cpp/src/manager.cc Initializes 1DS and opt-out behavior.
sdk_v2/cpp/src/configuration.h Adds native opt-out field.
sdk_v2/cpp/ports/cpp-client-telemetry/vcpkg.json Defines the 1DS overlay port.
sdk_v2/cpp/ports/cpp-client-telemetry/portfile.cmake Builds the 1DS dependency.
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.inline.h Implements C++ opt-out setter.
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.h Declares C++ opt-out API.
sdk_v2/cpp/CMakeLists.txt Integrates telemetry sources and linking.
sdk_v2/cpp/build.py Enables overlay ports and features.
Suppressed comments (1)

sdk_v2/cs/src/FoundryLocalManager.cs:253

  • The user dictionary is copied after the dedicated opt-out flag, so an entry of DisableNonessentialTelemetry=false silently defeats DisableNonessentialTelemetry = true. Make the public opt-out monotonic by applying it after AdditionalSettings; otherwise this API can send non-essential telemetry despite the caller opting out.

EventProperties ev(name);
ev.SetPriority(EventPriority::EventPriority_Normal);
ev.SetPolicyBitFlags(kCriticalData);
ev.SetPopsample(TelemetryInternal::kTelemetrySampleRatePercent);
Comment thread sdk_v2/cpp/src/telemetry/one_ds_telemetry.cc Outdated
Comment thread sdk_v2/python/src/foundry_local_sdk/configuration.py
Comment on lines +273 to +279
try {
impl_->log_manager->Flush();
impl_->log_manager->FlushAndTeardown();
LogManagerProvider::Release(impl_->config);
} catch (...) {
// Best-effort: never throw from a destructor.
}
- Preserve the selected action sample rate in the emitted popSample field.
- Make 1DS manager cleanup independent so Release runs after flush failures.
- Mark the manager acquired immediately after CreateLogManager succeeds.
- Keep dedicated telemetry opt-out settings authoritative over user overrides in C# and Python.
- Replace the stale NullTelemetry audio test fixture after merging main.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5a9051ad-9c32-4dc3-bd66-d33d416145a4

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.

Suppressed comments (7)

sdk_v2/cpp/src/manager.cc:314

  • This added line exceeds the repository's 120-character C++ line limit.
    logger_->Log(LogLevel::Warning, fmt::format("telemetry ProcessInfo failed during Manager initialization: {}", ex.what()));

sdk_v2/cpp/src/telemetry/one_ds_telemetry.cc:181

  • This gate suppresses CI only; local foundry_local_tests, sdk_integration_tests, and cache_only_tests runs do not set this or ORT_TELEMETRY_DISABLED, so they initialize the production 1DS client and can upload test events. That contradicts the class contract that unit-test processes are fully suppressed and pollutes operational telemetry. Add an explicit test-process/build gate or set the opt-out environment for every test registration before initialization.
  if (TelemetryEnvironment::IsCiEnvironment()) {
    logger_.Log(LogLevel::Information,
                "[Telemetry] CI environment detected; 1DS upload disabled (events still logged locally)");
    return;

sdk_v2/js/native/src/manager.cc:218

  • An explicit disableNonessentialTelemetry: false overwrites additionalSettings.DisableNonessentialTelemetry = "true", re-enabling uploads despite the legacy opt-out. The core and the Python/C# bindings treat either opt-out source as authoritative, so only inject this key when the typed flag is true.
    sdk_v2/cpp/src/telemetry/telemetry.h:213
  • This compatibility overload emits an empty correlation ID. Manager::Create still calls this overload, so its CoreInitialize event cannot satisfy the new correlation contract even though the other compatibility overloads create a direct context. Construct a direct context here and then apply the requested indirect flag.
  void RecordAction(Action action, ActionStatus status, const std::string& user_agent,
                    bool indirect, int64_t duration_ms) {
    RecordAction(action, status, InvocationContext{user_agent, {}, indirect}, duration_ms);
  }

sdk_v2/cpp/src/manager.cc:308

  • The tests exercise OneDsTelemetry(true) directly, but none verifies this normalization point. Add coverage proving both the native configuration field and additional_options["DisableNonessentialTelemetry"] suppress uploads, including accepted truthy values, so the primary wiring fixed by this PR cannot regress.

This issue also appears on line 314 of the same file.

  const bool disable_nonessential_telemetry =
      config_.disable_nonessential_telemetry ||
      IsAdditionalOptionEnabled(config_, "DisableNonessentialTelemetry");

sdk_v2/cs/src/FoundryLocalManager.cs:260

  • The C# manager tests do not cover this new telemetry-consent wiring. Add a configuration/native handoff test that verifies the default, enabled state, and precedence against a conflicting AdditionalSettings entry.
    sdk_v2/python/src/foundry_local_sdk/configuration.py:249
  • The Python configuration suite covers additional-setting serialization and native construction, but no test exercises this new opt-out branch or its precedence over a conflicting additional setting. Add unit/native-configuration coverage for the default, enabled, and conflict cases.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk_v2/cpp/test/internal_api/telemetry_test.cc:150

  • This does not exercise the opt-out wiring: it constructs OneDsTelemetry directly, and in CI the constructor exits because CI/GITHUB_ACTIONS is set, so IsUploadEnabled() is false even if the configuration flag is ignored. Add a test that drives DisableNonessentialTelemetry through Manager/additional_options and can distinguish configuration suppression from CI, token, or initialization failure; it should also verify that nonessential events are blocked while ProcessInfo remains eligible.
    sdk_v2/cpp/test/internal_api/telemetry_test.cc:361
  • This only verifies the lookup constant, not the client-side sampling behavior used by RecordAction. A broken threshold/hash path could still retain 0% or 100% while this passes. Exercise ShouldSampleTelemetryEvent at the returned 2% rate over deterministic keys (or fixed sampled/rejected vectors) and assert the expected retained proportion/decisions.
    sdk_v2/cpp/src/manager.cc:314
  • This added line exceeds the repository's 120-character C++ line limit. Wrap the logging call so the changed code conforms to the enforced project style.
    logger_->Log(LogLevel::Warning, fmt::format("telemetry ProcessInfo failed during Manager initialization: {}", ex.what()));

- Disable 1DS telemetry in both C++ test executables before test initialization.
- Preserve correlation IDs in the compatibility action overload.
- Keep JavaScript additional-setting opt-outs authoritative when the typed flag is false.
- Wrap the long Manager telemetry warning and add correlation regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5a9051ad-9c32-4dc3-bd66-d33d416145a4

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

sdk_v2/cpp/test/internal_api/telemetry_test.cc:152

  • This test cannot exercise the configuration opt-out: test_main.cc sets ORT_TELEMETRY_DISABLED=1 for the entire process, so the constructor exits at the environment gate and IsUploadEnabled() is false even if this argument is changed to false. Add an observable test for the configuration gate or run this case in an isolated process without the global suppression.

Comment on lines +219 to +221
if (const auto cache_dir = TelemetryDeviceId::EnsureCacheDirectory(); !cache_dir.empty()) {
config[CFG_STR_CACHE_FILE_PATH] = (cache_dir / "foundry-local.db").string();
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants