From 10137c2ffa9a52e02d19ba6f64609e3309693ff8 Mon Sep 17 00:00:00 2001 From: inventarSarah Date: Thu, 18 Jun 2026 15:33:30 +0200 Subject: [PATCH 1/5] python stream spans tracing pages update progress --- .../tracing/configure-sampling/index.mdx | 18 +- .../automatic-instrumentation.mdx | 8 +- .../custom-instrumentation/index.mdx | 11 + .../python/tracing/instrumentation/index.mdx | 4 +- .../python/tracing/span-lifecycle/index.mdx | 308 ++++++++-- .../python/tracing/span-metrics/examples.mdx | 570 ++++++++++++++++-- .../python/tracing/span-metrics/index.mdx | 72 +-- .../python/tracing/troubleshooting/index.mdx | 6 + .../custom-instrumentation/python.mdx | 52 +- .../span-metrics/add-all-spans/python.mdx | 58 ++ 10 files changed, 942 insertions(+), 165 deletions(-) create mode 100644 platform-includes/tracing/span-metrics/add-all-spans/python.mdx diff --git a/docs/platforms/python/tracing/configure-sampling/index.mdx b/docs/platforms/python/tracing/configure-sampling/index.mdx index 0627639b8f1a4..d2e363d691bc0 100644 --- a/docs/platforms/python/tracing/configure-sampling/index.mdx +++ b/docs/platforms/python/tracing/configure-sampling/index.mdx @@ -8,6 +8,12 @@ If you find that Sentry's tracing functionality is generating too much data, for Effective sampling is key to getting the most value from Sentry's performance monitoring while minimizing overhead. The Python SDK provides two ways to control the sampling rate. You can review the options and [examples](#traces-sampler-examples) below. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + ## Sampling Configuration Options ### 1. Uniform Sample Rate (`traces_sample_rate`) @@ -16,15 +22,15 @@ Effective sampling is key to getting the most value from Sentry's performance mo -With `traces_sample_rate` set to `0.25`, each transaction in your application is randomly sampled with a probability of `0.25`, so you can expect that one in every four transactions will be sent to Sentry. +With `traces_sample_rate` set to `0.25`, each transaction/service span in your application is randomly sampled with a probability of `0.25`, so you can expect that one in every four transactions/service spans will be sent to Sentry. ### 2. Sampling Function (`traces_sampler`) For more granular control, you can provide a `traces_sampler` function. This approach allows you to: -- Apply different sampling rates to different types of transactions -- Filter out specific transactions entirely -- Make sampling decisions based on transaction data +- Apply different sampling rates to different types of transactions/service spans +- Filter out specific transactions/service spans entirely +- Make sampling decisions based on transaction/service span data - Control the inheritance of sampling decisions in distributed traces - Use custom attributes to modify sampling @@ -291,6 +297,7 @@ with sentry_sdk.start_transaction( ) as transaction: # Your code here ``` + ## The Sampling Context Object @@ -315,12 +322,14 @@ When the `traces_sampler` function is called, the Sentry SDK passes a `sampling_ The sampling context contains both SDK-provided attributes and custom attributes: **SDK-Provided Attributes:** + - `transaction_context.name`: The name of the transaction - `transaction_context.op`: The operation type - `parent_sampled`: Whether the parent transaction was sampled - `parent_sample_rate`: The sample rate used by the parent **Custom Attributes:** + - Any data you add to the `custom_sampling_context` parameter in `start_transaction`. Use this for data that you want to use for sampling decisions but don't want to include in the transaction data that gets sent to Sentry. Read more about sampling context [here](/platforms/python/configuration/sampling/#sampling-context). ## Sampling Decision Precedence @@ -341,6 +350,7 @@ Sentry uses a "head-based" sampling approach: - This decision is propagated to all downstream services The two key headers are: + - `sentry-trace`: Contains trace ID, span ID, and sampling decision - `baggage`: Contains additional trace metadata including sample rate diff --git a/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx b/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx index fc44a217d9a72..4d4387c3425d5 100644 --- a/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx +++ b/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx @@ -17,7 +17,7 @@ supported: description: "Learn what instrumentation automatically captures transactions." --- -Many integrations for popular frameworks automatically capture transactions. If you already have any of the following frameworks set up for Sentry error reporting, you will start to see traces immediately: +Many integrations for popular frameworks automatically capture transactions (or service spans in stream mode). If you already have any of the following frameworks set up for Sentry error reporting, you will start to see traces immediately: - All WSGI-based web frameworks (Django, Flask, Pyramid, Falcon, Bottle) - Celery @@ -26,11 +26,13 @@ Many integrations for popular frameworks automatically capture transactions. If See the full [list of available integrations](/platforms/python/integrations/). -Spans are instrumented for the following operations within a transaction: +Spans are instrumented for the following operations within a transaction/service span: - Database queries that use SQLAlchemy or the Django ORM - HTTP requests made with HTTPX, requests, the stdlib, AIOHTTP, or pyreqwest - Spawned subprocesses - Redis operations -Spans are only created within an existing transaction. If you're not using any of the supported frameworks, you'll need to create transactions manually. +In transaction mode, spans are only created within an existing transaction. If you're not using any of the supported frameworks, you'll need to create transactions manually. + +In stream mode, any span without a parent is a service span (equivalent to transactions). You can force any span to be a service by removing its parent. See Custom Instrumentation to learn more. diff --git a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx index 509d7af43d2ec..24790f07d18de 100644 --- a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx +++ b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx @@ -5,8 +5,19 @@ description: "Learn how to capture performance data on any action in your app." The Sentry SDK for Python does a very good job of auto instrumenting your application. If you use one of the popular frameworks, we've got you covered because well-known operations like HTTP calls and database queries will be instrumented out of the box. The Sentry SDK will also check your installed Python packages and auto-enable the matching SDK integrations. If you want to enable tracing in a piece of code that performs some other operations, add the `@sentry_sdk.trace` decorator. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + ## Add a Transaction + + Transactions are only available in transaction mode. In stream mode, + `start_transaction()` isn't available and you create spans instead. + + Adding transactions will allow you to instrument and capture certain regions of your code. diff --git a/docs/platforms/python/tracing/instrumentation/index.mdx b/docs/platforms/python/tracing/instrumentation/index.mdx index be7475828ecfc..cbc92494f3ae8 100644 --- a/docs/platforms/python/tracing/instrumentation/index.mdx +++ b/docs/platforms/python/tracing/instrumentation/index.mdx @@ -14,8 +14,8 @@ There are two ways that instrumentation is applied to your application: ## Automatic Instrumentation -Many integrations for popular frameworks automatically capture transactions that can be sent to Sentry. Read more about automatic instrumentation [here](/platforms/python/tracing/instrumentation/automatic-instrumentation/). +Many integrations for popular frameworks automatically capture transactions and spans that can be sent to Sentry. Read more about automatic instrumentation [here](/platforms/python/tracing/instrumentation/automatic-instrumentation/). ## Custom Instrumentation -To add custom performance data to your application, you need to add custom instrumentation in the form of [spans](/concepts/key-terms/tracing/distributed-tracing/#traces-transactions-and-spans). Spans are a way to measure the time it takes for a specific action to occur. For example, you can create a span to measure the time it takes for a function to execute. Learn more about span lifecycles [here](/platforms/python/tracing/span-lifecycle/). +To add custom performance data to your application, you need to add custom instrumentation in the form of [spans](/concepts/key-terms/tracing/distributed-tracing/#traces-transactions-and-spans). Spans are a way to measure the time it takes for a specific action to occur. For example, you can create a span to measure the time it takes for a function to execute. Learn more about span lifecycles [here](/platforms/python/tracing/span-lifecycle/). diff --git a/docs/platforms/python/tracing/span-lifecycle/index.mdx b/docs/platforms/python/tracing/span-lifecycle/index.mdx index 571bf58725f73..48116b4ac34c7 100644 --- a/docs/platforms/python/tracing/span-lifecycle/index.mdx +++ b/docs/platforms/python/tracing/span-lifecycle/index.mdx @@ -12,13 +12,19 @@ To capture transactions and spans customized to your organization's needs, you m To add custom performance data to your application, you need to add custom instrumentation in the form of [spans](/concepts/key-terms/tracing/distributed-tracing/#traces-transactions-and-spans). Spans are a way to measure the time it takes for a specific action to occur. For example, you can create a span to measure the time it takes for a function to execute. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + ## Span Lifecycle In Python, spans are typically created using a context manager, which automatically manages the span's lifecycle. When you create a span using a context manager, the span automatically starts when entering the context and ends when exiting it. This is the recommended approach for most scenarios. -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk # Start a span for a task @@ -30,13 +36,28 @@ with sentry_sdk.start_span(op="task", name="Create User"): # The span automatically ends here when the 'with' block exits ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +# Start a span for a task +with sentry_sdk.traces.start_span( + name="Create User", + attributes={"sentry.op": "task"}, +): + # Your code here + # The span will automatically end when exiting this block + user = create_user(email="user@example.com") + send_welcome_email(user) + # The span automatically ends here when the 'with' block exits +``` + You can call the context manager's `__enter__` and `__exit__` methods to more explicitly control the span's lifecycle. ## Span Context and Nesting When you create a span, it becomes the child of the current active span. This allows you to build a hierarchy of spans that represent the execution path of your application: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk with sentry_sdk.start_span(op="process", name="Process Data"): @@ -51,21 +72,61 @@ with sentry_sdk.start_span(op="process", name="Process Data"): transform_data() ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="Process Data", + attributes={"sentry.op": "process"}, +): + # This code is tracked in the "Process Data" span + + with sentry_sdk.traces.start_span( + name="Validate Input", + attributes={"sentry.op": "task"}, + ): + # This is now a child span of "Process Data" + validate_data() + + with sentry_sdk.traces.start_span( + name="Transform Data", + attributes={"sentry.op": "task"}, + ): + # Another child span + transform_data() +``` + ## Span Starting Options The following options can be used when creating spans: -| Option | Type | Description | -| ------------- | --------------- | ----------------------------------------------- | -| `op` | `string` | The operation of the span. | -| `name` | `string` | The name of the span. | -| `start_timestamp` | `datetime/float`| The start time of the span. | +### Transaction Mode (Default) + +| Option | Type | Description | +| ----------------- | ---------------- | --------------------------- | +| `op` | `string` | The operation of the span. | +| `name` | `string` | The name of the span. | +| `start_timestamp` | `datetime/float` | The start time of the span. | + +### Stream mode + +| Option | Type | Description | +| ----------------- | ---------------- | -------------------------------------------------------------------------- | +| `name` | `string` | The name of the span. | +| `attributes` | `dict` | Structured metadata for the span, including `sentry.op` for the operation. | +| `start_timestamp` | `datetime/float` | The parent to attach to. Pass None to force a service span. | +| `parent_span` | `Span/None` | The start time of the span. | + + + `op` is not a dedicated argument in stream mode. Set it as the `sentry.op` + attribute instead. + ## Using the Context Manager -For most scenarios, we recommend using the context manager approach with `sentry_sdk.start_span()`. This creates a new span that automatically starts when entering the context and ends when exiting it. +For most scenarios, we recommend using the context manager approach. This creates a new span that automatically starts when entering the context and ends when exiting it. -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk with sentry_sdk.start_span(op="db", name="Query Users") as span: @@ -76,9 +137,23 @@ with sentry_sdk.start_span(op="db", name="Query Users") as span: span.set_data("user_count", len(users)) ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="Query Users", + attributes={"sentry.op": "db"}, +) as span: + # Perform a database query + users = db.query("SELECT * FROM users") + + # You can set attributes on the span + span.set_attribute("user_count", len(users)) +``` + The context manager also correctly handles exceptions, marking the span as failed if an exception occurs: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk try: @@ -91,11 +166,27 @@ except Exception: pass ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +try: + with sentry_sdk.traces.start_span( + name="Call External API", + attributes={"sentry.op": "http"}, + ): + # If this raises an exception, the span will be marked as failed + response = requests.get("https://api.example.com/data") + response.raise_for_status() +except Exception: + # The span is already marked as failed and has ended + pass +``` + ## Getting the Current Span -You can access the currently active span using `sentry_sdk.get_current_span()`: +You can access the currently active span: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk # Get the current active span @@ -104,8 +195,19 @@ if current_span: current_span.set_data("key", "value") ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +# Get the current active span +current_span = sentry_sdk.traces.get_current_span() +if current_span: + current_span.set_attribute("key", "value") +``` + ## Working with Transactions +`start_transaction` is only available in transaction mode. + [Transactions](/product/dashboards/sentry-dashboards/transaction-summary/#what-is-a-transaction) are a special type of span that represent a complete operation in your application, such as a web request. You can create transactions explicitly: ```python @@ -120,13 +222,38 @@ with sentry_sdk.start_transaction(name="Background Task", op="task") as transact pass ``` +## Working with Service Spans + +Service spans are only available in stream mode. + +In stream mode, a service span (the equivalent of a transaction) is just a span started with no parent. Pass `parent_span=None` to force this, even if a span is currently active: + +```python +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="Background Task", + attributes={"sentry.op": "task"}, + parent_span=None, +) as service_span: + # Your code here + + # You can add child spans to the service span + with sentry_sdk.traces.start_span( + name="Data Processing", + attributes={"sentry.op": "subtask"}, + ): + # Process data + pass +``` + ## Improving Span Data ### Adding Span Attributes Span attributes customize information you can get through tracing. This information can be found in the traces views in Sentry, once you drill into a span. You can capture additional context with span attributes. These can be key-value pairs of various Python types. -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk with sentry_sdk.start_span(op="db", name="Query Users") as span: @@ -137,9 +264,23 @@ with sentry_sdk.start_span(op="db", name="Query Users") as span: span.set_data("result_count", len(users)) ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="Query Users", + attributes={"sentry.op": "db"}, +) as span: + # Execute the query + users = db.query("SELECT * FROM users WHERE active = true") + + # You can add more attributes during execution + span.set_attribute("result_count", len(users)) +``` + You can also add attributes to an existing span: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk # Get the current span @@ -150,42 +291,22 @@ if span: span.set_data("request_size", len(request.body)) ``` -### Adding Attributes to All Spans - -To add attributes to all spans, use the `before_send_transaction` callback: - -```python +```python {tabTitle:Stream Mode} import sentry_sdk -from sentry_sdk.types import Event, Hint - -def before_send_transaction(event: Event, hint: Hint) -> Event | None: - # Add attributes to the root span (transaction) - if "trace" in event.get("contexts", {}): - if "data" not in event["contexts"]["trace"]: - event["contexts"]["trace"]["data"] = {} - event["contexts"]["trace"]["data"].update({ - "app_version": "1.2.3", - "environment_region": "us-west-2" - }) +# Get the current span +span = sentry_sdk.traces.get_current_span() +if span: + # Set individual attributes + span.set_attribute("user_id", user.id) + span.set_attribute("request_size", len(request.body)) +``` - # Add attributes to all child spans - for span in event.get("spans", []): - if "data" not in span: - span["data"] = {} +### Adding Attributes to All Spans - span["data"].update({ - "component_version": "2.0.0", - "deployment_stage": "production" - }) + - return event - -sentry_sdk.init( - # ... - before_send_transaction=before_send_transaction -) -``` +Note that `before_send_span` can only modify span data and not drop spans – use [`ignore_spans`](#dropping-spans) instead. ### Adding Span Operations ("op") @@ -193,7 +314,7 @@ Spans can have an operation associated with them, which helps Sentry understand Sentry maintains a [list of well-known span operations](https://develop.sentry.dev/sdk/performance/span-operations/#list-of-operations) that you should use when applicable: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk # HTTP client operation @@ -213,11 +334,41 @@ with sentry_sdk.start_span(op="file.read", name="Read Config"): config = json.load(f) ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +# HTTP client operation +with sentry_sdk.traces.start_span( + name="Fetch User Data", + attributes={"sentry.op": "http.client"}, +): + response = requests.get("https://api.example.com/users") + +# Database operation +with sentry_sdk.traces.start_span( + name="Save User", + attributes={"sentry.op": "db"}, +): + db.execute( + "INSERT INTO users (name, email) VALUES (%s, %s)", + (user.name, user.email), + ) + +# File I/O operation +with sentry_sdk.traces.start_span( + name="Read Config", + attributes={"sentry.op": "file.read"}, +): + with open("config.json", "r") as f: + config = json.load(f) +``` + ### Updating the Span Status -You can update the status of a span to indicate whether it succeeded or failed: +You can update the status of a span to indicate whether it succeeded or failed. +In stream mode, status can only be `ok` (default) or `error`. -```python +```python {tabTitle:Stream Mode} import sentry_sdk with sentry_sdk.start_span(op="task", name="Process Payment") as span: @@ -234,3 +385,64 @@ with sentry_sdk.start_span(op="task", name="Process Payment") as span: # Span will automatically be marked as failed when an exception occurs raise ``` + +```python {tabTitle:Stream Mode} +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="Process Payment", + attributes={"sentry.op": "task"}, +) as span: + try: + result = process_payment(payment_id) + if result.success: + # Mark the span as successful (this is also the default) + span.status = "ok" + else: + # Mark the span as failed + span.status = "error" + span.set_attribute("error_reason", result.error) + except Exception: + # Span will automatically be marked as failed when an exception occurs + raise +``` + +## Dropping Spans + +`ignore_span` is only available in stream mode. + +In stream mode, you can prevent specific spans from being created using `ignore_spans`, set under `_experiments`. It evaluates rules at span start, which can be strings, compiled regexes, or dictionaries with `name` and/or `attributes` conditions. If the dropped span is a service span, its children are dropped too. If it's a child span, only that span is dropped and its children are reparented to the nearest ancestor. + +```python +import re +import sentry_sdk + +sentry_sdk.init( + _experiments={ + "trace_lifecycle": "stream", + "ignore_spans": [ + # String match against span name + "/health", + + # Regex match against span name + re.compile(r"/flow/.*"), + + # Match by attributes (all must match) + { + "attributes": { + "service.id": "15def9a", + "flow.pipeline": "legacy", + } + }, + + # Match by name and attributes + { + "name": re.compile(r"/flow/.*"), + "attributes": { + "service.id": re.compile(r".*\.facade"), + }, + }, + ], + }, +) +``` diff --git a/docs/platforms/python/tracing/span-metrics/examples.mdx b/docs/platforms/python/tracing/span-metrics/examples.mdx index 5fe2452ebcc23..c6d363df8b9c5 100644 --- a/docs/platforms/python/tracing/span-metrics/examples.mdx +++ b/docs/platforms/python/tracing/span-metrics/examples.mdx @@ -12,6 +12,11 @@ These examples assume you have already set up traci This guide provides practical examples of using span attributes and metrics to solve common monitoring and debugging challenges in Python applications. Each example demonstrates how to instrument different components, showing how they work together within a distributed trace to provide end-to-end visibility. + + This page covers both transaction mode (default) and stream mode. See{" "} + New Spans to learn more. + + ## File Upload and Processing Pipeline **Challenge:** Understanding bottlenecks and failures in multi-step file processing operations across request handling and processing services. @@ -19,7 +24,8 @@ This guide provides practical examples of using span attributes and metrics to s **Solution:** Track the entire file processing pipeline with detailed metrics at each stage, from initial file handling through processing and storage. **Client Application Instrumentation:** -```python + +```python {tabTitle:Transaction Mode (Default)} # File upload request handling import sentry_sdk import time @@ -31,13 +37,13 @@ with sentry_sdk.start_span(op="upload", name="Upload File") as span: try: # Begin upload process with the requests library file_path = Path("/path/to/uploads/user-profile.jpg") - + # Track progress with a context manager @contextmanager def track_progress(total_size): start = time.time() bytes_sent = 0 - + class ProgressTracker: def __call__(self, monitor): nonlocal bytes_sent @@ -45,12 +51,12 @@ with sentry_sdk.start_span(op="upload", name="Upload File") as span: progress_percent = (bytes_sent / total_size) * 100 span.set_data("upload.percent_complete", progress_percent) span.set_data("upload.bytes_transferred", bytes_sent) - + yield ProgressTracker() - + # Record total time after context exits span.set_data("upload.total_time_ms", (time.time() - start) * 1000) - + # Use the progress tracker with requests with track_progress(file_path.stat().st_size) as progress_callback: with open(file_path, "rb") as f: @@ -61,15 +67,15 @@ with sentry_sdk.start_span(op="upload", name="Upload File") as span: stream=True, hooks={"response": progress_callback} ) - + # Set final data after completion span.set_data("upload.success", response.ok) if response.ok: result = response.json() span.set_data("upload.server_file_id", result["file_id"]) - + return response - + except Exception as error: # Record failure information span.set_data("upload.success", False) @@ -79,8 +85,72 @@ with sentry_sdk.start_span(op="upload", name="Upload File") as span: raise ``` +```python {tabTitle:Stream Mode} +# File upload request handling +import sentry_sdk +import time +from pathlib import Path +import requests +from contextlib import contextmanager + +with sentry_sdk.traces.start_span( + name="Upload File", + attributes={"sentry.op": "upload"}, +) as span: + try: + # Begin upload process with the requests library + file_path = Path("/path/to/uploads/user-profile.jpg") + + # Track progress with a context manager + @contextmanager + def track_progress(total_size): + start = time.time() + bytes_sent = 0 + + class ProgressTracker: + def __call__(self, monitor): + nonlocal bytes_sent + bytes_sent = monitor.bytes_read + progress_percent = (bytes_sent / total_size) * 100 + span.set_attribute("upload.percent_complete", progress_percent) + span.set_attribute("upload.bytes_transferred", bytes_sent) + + yield ProgressTracker() + + # Record total time after context exits + span.set_attribute("upload.total_time_ms", (time.time() - start) * 1000) + + # Use the progress tracker with requests + with track_progress(file_path.stat().st_size) as progress_callback: + with open(file_path, "rb") as f: + response = requests.post( + "https://api.example.com/upload", + files={"file": f}, + headers={"X-Sentry-Trace": sentry_sdk.get_traceparent()}, # Propagate trace context + stream=True, + hooks={"response": progress_callback} + ) + + # Set final data after completion + span.set_attribute("upload.success", response.ok) + if response.ok: + result = response.json() + span.set_attribute("upload.server_file_id", result["file_id"]) + + return response + + except Exception as error: + # Record failure information + span.set_attribute("upload.success", False) + span.set_attribute("upload.error_type", error.__class__.__name__) + span.set_attribute("upload.error_message", str(error)) + span.status = "error" + raise +``` + **Server Application Instrumentation:** -```python + +```python {tabTitle:Transaction Mode (Default)} # File processing service import sentry_sdk from pathlib import Path @@ -92,7 +162,7 @@ with sentry_sdk.start_span( ) as span: # File processing implementation file_path = Path("/tmp/uploads/user-profile.jpg") - + # Process the file try: # Track individual processing steps @@ -100,7 +170,7 @@ with sentry_sdk.start_span( # Virus scan implementation scan_span.set_data("scan.engine", "clamav") scan_span.set_data("scan.result", "clean") - + # Upload to S3 s3_client = boto3.client('s3', region_name='us-west-2') upload_start = time.time() @@ -109,16 +179,58 @@ with sentry_sdk.start_span( 'my-bucket', 'uploads/user-profile.jpg' ) - - span.set_data("storage.actual_upload_time_ms", + + span.set_data("storage.actual_upload_time_ms", (time.time() - upload_start) * 1000) - + except Exception as e: span.set_status(sentry_sdk.consts.SPANSTATUS.INTERNAL_ERROR) span.set_data("error.message", str(e)) raise ``` +```python {tabTitle:Stream Mode} +# File processing service +import sentry_sdk +from pathlib import Path +import boto3 + +with sentry_sdk.traces.start_span( + name="File Processing Service", + attributes={"sentry.op": "file.process.service"}, +) as span: + # File processing implementation + file_path = Path("/tmp/uploads/user-profile.jpg") + + # Process the file + try: + # Track individual processing steps + with sentry_sdk.traces.start_span( + name="Virus Scan", + attributes={"sentry.op": "scan"}, + ) as scan_span: + # Virus scan implementation + scan_span.set_attribute("scan.engine", "clamav") + scan_span.set_attribute("scan.result", "clean") + + # Upload to S3 + s3_client = boto3.client('s3', region_name='us-west-2') + upload_start = time.time() + s3_client.upload_file( + str(file_path), + 'my-bucket', + 'uploads/user-profile.jpg' + ) + + span.set_attribute("storage.actual_upload_time_ms", + (time.time() - upload_start) * 1000) + + except Exception as e: + span.status = "error" + span.set_attribute("error.message", str(e)) + raise +``` + **How the Trace Works Together:** The client application span initiates the trace and handles the file upload. It propagates the trace context to the server through the request headers. The server span continues the trace, processing the file and storing it. This creates a complete picture of the file's journey, allowing you to: @@ -134,7 +246,8 @@ The client application span initiates the trace and handles the file upload. It **Solution:** Tracking of the entire LLM interaction flow, from initial request through response processing. **Client Application Instrumentation:** -```python + +```python {tabTitle:Transaction Mode (Default)} # LLM request handling in a Flask application import sentry_sdk import time @@ -145,14 +258,14 @@ from flask import jsonify def handle_llm_request(): with sentry_sdk.start_span(op="gen_ai.chat", name="Generate Text") as span: start_time = time.time() * 1000 # Convert to milliseconds - + # Begin streaming response from LLM API user_input = request.json["question"] - + response_chunks = [] first_token_received = False tokens_received = 0 - + # Using OpenAI's streaming API try: for chunk in openai.ChatCompletion.create( @@ -163,25 +276,25 @@ def handle_llm_request(): tokens_received += 1 content = chunk.get("choices", [{}])[0].get("delta", {}).get("content", "") response_chunks.append(content) - + # Record time to first token if not first_token_received and content: first_token_received = True time_to_first_token = (time.time() * 1000) - start_time span.set_data("response.time_to_first_token_ms", time_to_first_token) - + # Record final metrics after stream completes total_request_time = (time.time() * 1000) - start_time - + span.set_data("response.total_time_ms", total_request_time) span.set_data("response.format", "text") span.set_data("response.tokens_received", tokens_received) - + return jsonify({ "response": "".join(response_chunks), "tokens": tokens_received }) - + except Exception as error: span.set_status(sentry_sdk.consts.SPANSTATUS.INTERNAL_ERROR) span.set_data("error.type", error.__class__.__name__) @@ -189,8 +302,69 @@ def handle_llm_request(): return jsonify({"error": str(error)}), 500 ``` +```python {tabTitle:Stream Mode} +# LLM request handling in a Flask application +import sentry_sdk +import time +import openai +from flask import jsonify + +@app.route("/ask", methods=["POST"]) +def handle_llm_request(): + with sentry_sdk.traces.start_span( + name="Generate Text", + attributes={"sentry.op": "gen_ai.chat"}, + ) as span: + start_time = time.time() * 1000 # Convert to milliseconds + + # Begin streaming response from LLM API + user_input = request.json["question"] + + response_chunks = [] + first_token_received = False + tokens_received = 0 + + # Using OpenAI's streaming API + try: + for chunk in openai.ChatCompletion.create( + model="gpt-4", + messages=[{"role": "user", "content": user_input}], + stream=True + ): + tokens_received += 1 + content = chunk.get("choices", [{}])[0].get("delta", {}).get("content", "") + response_chunks.append(content) + + # Record time to first token + if not first_token_received and content: + first_token_received = True + time_to_first_token = (time.time() * 1000) - start_time + span.set_attribute("response.time_to_first_token_ms", time_to_first_token) + + # Record final metrics after stream completes + total_request_time = (time.time() * 1000) - start_time + + span.set_attributes({ + "response.total_time_ms": total_request_time, + "response.format": "text", + "response.tokens_received": tokens_received + }) + + return jsonify({ + "response": "".join(response_chunks), + "tokens": tokens_received + }) + + except Exception as error: + span.status = "error" + span.set_attribute("error.type", error.__class__.__name__) + span.set_attribute("error.message", str(error)) + return jsonify({"error": str(error)}), 500 +``` + **Server Application Instrumentation:** -```python + +```python {tabTitle:Transaction Mode (Default)} # LLM processing service (e.g., in a separate microservice) import sentry_sdk import time @@ -202,15 +376,15 @@ def process_llm_request(request_data): name="Generate Text" ) as span: start_time = int(time.time() * 1000) # Current time in milliseconds - + try: # Check rate limits before processing rate_limits = check_rate_limits() span.set_data("llm.rate_limit_remaining", rate_limits["remaining"]) - + # Prepare the prompt with additional context prepared_prompt = enhance_prompt(request_data["question"]) - + # Make the actual API call to the LLM provider response = openai.ChatCompletion.create( model="gpt-4", @@ -219,13 +393,13 @@ def process_llm_request(request_data): temperature=0.7, max_tokens=4096 ) - + # Track token usage and performance metrics span.set_data("llm.prompt_tokens", response.usage.prompt_tokens) span.set_data("llm.completion_tokens", response.usage.completion_tokens) span.set_data("llm.total_tokens", response.usage.total_tokens) span.set_data("llm.api_latency_ms", int(time.time() * 1000) - start_time) - + # Calculate and record cost based on token usage cost = calculate_cost( response.usage.prompt_tokens, @@ -233,23 +407,88 @@ def process_llm_request(request_data): "gpt-4" ) span.set_data("llm.cost_usd", cost) - + return { "response": response.choices[0].message.content, "usage": response.usage } - + except Exception as error: # Track error information span.set_data("error", True) span.set_data("error.type", error.__class__.__name__) span.set_data("error.message", str(error)) - + # Check if it's a rate limit error is_rate_limit = "rate_limit" in str(error).lower() span.set_data("error.is_rate_limit", is_rate_limit) span.set_status(sentry_sdk.consts.SPANSTATUS.INTERNAL_ERROR) - + + raise +``` + +```python {tabTitle:Stream Mode} +# LLM processing service (e.g., in a separate microservice) +import sentry_sdk +import time +import openai + +def process_llm_request(request_data): + with sentry_sdk.traces.start_span( + name="Generate Text", + attributes={"sentry.op": "gen_ai.chat"}, + ) as span: + start_time = int(time.time() * 1000) # Current time in milliseconds + + try: + # Check rate limits before processing + rate_limits = check_rate_limits() + span.set_attribute("llm.rate_limit_remaining", rate_limits["remaining"]) + + # Prepare the prompt with additional context + prepared_prompt = enhance_prompt(request_data["question"]) + + # Make the actual API call to the LLM provider + response = openai.ChatCompletion.create( + model="gpt-4", + messages=[{"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": prepared_prompt}], + temperature=0.7, + max_tokens=4096 + ) + + # Track token usage and performance metrics + span.set_attributes({ + "llm.prompt_tokens": response.usage.prompt_tokens, + "llm.completion_tokens": response.usage.completion_tokens, + "llm.total_tokens": response.usage.total_tokens, + "llm.api_latency_ms": int(time.time() * 1000) - start_time + }) + + # Calculate and record cost based on token usage + cost = calculate_cost( + response.usage.prompt_tokens, + response.usage.completion_tokens, + "gpt-4" + ) + span.set_attribute("llm.cost_usd", cost) + + return { + "response": response.choices[0].message.content, + "usage": response.usage + } + + except Exception as error: + # Track error information + span.set_attribute("error", True) + span.set_attribute("error.type", error.__class__.__name__) + span.set_attribute("error.message", str(error)) + + # Check if it's a rate limit error + is_rate_limit = "rate_limit" in str(error).lower() + span.set_attribute("error.is_rate_limit", is_rate_limit) + span.status = "error" + raise ``` @@ -269,7 +508,8 @@ The client application span captures the initial request handling, while the ser **Solution:** Track the full transaction process from API request to order fulfillment. **Client Application Instrumentation:** -```python + +```python {tabTitle:Transaction Mode (Default)} # Django view handling checkout request import sentry_sdk import time @@ -282,28 +522,28 @@ class CheckoutView(View): # Validate the checkout request validation_start = time.time() validation_result = self.validate_checkout_data(request.POST) - span.set_data("request.validation_time_ms", + span.set_data("request.validation_time_ms", (time.time() - validation_start) * 1000) - + if not validation_result["valid"]: span.set_data("request.validation_success", False) span.set_data("request.validation_errors", validation_result["errors"]) return JsonResponse({"errors": validation_result["errors"]}, status=400) - + # Process the order try: order_result = self.process_order(request) - + # Update span with order results span.set_data("order.id", order_result["order_id"]) span.set_data("order.success", True) - + # Clear the cart and return success request.session["cart"] = [] request.session["cart_total"] = 0 - + return JsonResponse({"order_id": order_result["order_id"]}) - + except Exception as e: span.set_status(sentry_sdk.consts.SPANSTATUS.INTERNAL_ERROR) span.set_data("order.success", False) @@ -311,8 +551,54 @@ class CheckoutView(View): return JsonResponse({"error": str(e)}, status=500) ``` +```python {tabTitle:Stream Mode} +# Django view handling checkout request +import sentry_sdk +import time +from django.views import View +from django.http import JsonResponse + +class CheckoutView(View): + def post(self, request): + with sentry_sdk.traces.start_span( + name="Process Order", + attributes={"sentry.op": "order"}, + ) as span: + # Validate the checkout request + validation_start = time.time() + validation_result = self.validate_checkout_data(request.POST) + span.set_attribute("request.validation_time_ms", + (time.time() - validation_start) * 1000) + + if not validation_result["valid"]: + span.set_attribute("request.validation_success", False) + span.set_attribute("request.validation_errors", validation_result["errors"]) + return JsonResponse({"errors": validation_result["errors"]}, status=400) + + # Process the order + try: + order_result = self.process_order(request) + + # Update span with order results + span.set_attribute("order.id", order_result["order_id"]) + span.set_attribute("order.success", True) + + # Clear the cart and return success + request.session["cart"] = [] + request.session["cart_total"] = 0 + + return JsonResponse({"order_id": order_result["order_id"]}) + + except Exception as e: + span.status = "error" + span.set_attribute("order.success", False) + span.set_attribute("error.message", str(e)) + return JsonResponse({"error": str(e)}, status=500) +``` + **Server Application Instrumentation:** -```python + +```python {tabTitle:Transaction Mode (Default)} # Order processing service import sentry_sdk import stripe @@ -327,15 +613,15 @@ def process_order(order_data): # Check inventory availability inventory_start = time.time() inventory_result = check_inventory(order_data["items"]) - span.set_data("inventory.check_time_ms", + span.set_data("inventory.check_time_ms", (time.time() - inventory_start) * 1000) span.set_data("inventory.all_available", inventory_result["all_available"]) - + if not inventory_result["all_available"]: - span.set_data("inventory.unavailable_items", + span.set_data("inventory.unavailable_items", inventory_result["unavailable_items"]) raise ValueError("Some items are out of stock") - + # Process payment via Stripe payment_start = time.time() stripe.api_key = "sk_test_..." @@ -345,27 +631,27 @@ def process_order(order_data): payment_method=order_data["payment_method_id"], confirm=True ) - - span.set_data("payment.processing_time_ms", + + span.set_data("payment.processing_time_ms", (time.time() - payment_start) * 1000) span.set_data("payment.transaction_id", payment_intent.id) span.set_data("payment.success", payment_intent.status == "succeeded") - + # Create fulfillment record fulfillment = create_fulfillment(order_data["order_id"]) span.set_data("fulfillment.id", fulfillment["id"]) span.set_data("fulfillment.warehouse", fulfillment["warehouse"]) span.set_data("fulfillment.shipping_method", fulfillment["shipping_method"]) - span.set_data("fulfillment.estimated_delivery", + span.set_data("fulfillment.estimated_delivery", fulfillment["estimated_delivery"].isoformat()) - + return { "success": True, "order_id": order_data["order_id"], "payment_id": payment_intent.id, "fulfillment_id": fulfillment["id"] } - + except Exception as e: span.set_status(sentry_sdk.consts.SPANSTATUS.INTERNAL_ERROR) span.set_data("error.message", str(e)) @@ -373,6 +659,69 @@ def process_order(order_data): raise ``` +```python {tabTitle:Stream Mode} +# Order processing service +import sentry_sdk +import stripe +from decimal import Decimal + +def process_order(order_data): + with sentry_sdk.traces.start_span( + name="Check Inventory", + attributes={"sentry.op": "inventory"}, + ) as span: + try: + # Check inventory availability + inventory_start = time.time() + inventory_result = check_inventory(order_data["items"]) + span.set_attribute("inventory.check_time_ms", + (time.time() - inventory_start) * 1000) + span.set_attribute("inventory.all_available", inventory_result["all_available"]) + + if not inventory_result["all_available"]: + span.set_attribute("inventory.unavailable_items", + inventory_result["unavailable_items"]) + raise ValueError("Some items are out of stock") + + # Process payment via Stripe + payment_start = time.time() + stripe.api_key = "sk_test_..." + payment_intent = stripe.PaymentIntent.create( + amount=int(Decimal(order_data["total"]) * 100), # Convert to cents + currency="usd", + payment_method=order_data["payment_method_id"], + confirm=True + ) + + span.set_attributes({ + "payment.processing_time_ms": (time.time() - payment_start) * 1000, + "payment.transaction_id": payment_intent.id, + "payment.success": payment_intent.status == "succeeded" + }) + + # Create fulfillment record + fulfillment = create_fulfillment(order_data["order_id"]) + span.set_attributes({ + "fulfillment.id": fulfillment["id"], + "fulfillment.warehouse": fulfillment["warehouse"], + "fulfillment.shipping_method": fulfillment["shipping_method"], + "fulfillment.estimated_delivery": fulfillment["estimated_delivery"].isoformat() + }) + + return { + "success": True, + "order_id": order_data["order_id"], + "payment_id": payment_intent.id, + "fulfillment_id": fulfillment["id"] + } + + except Exception as e: + span.status = "error" + span.set_attribute("error.message", str(e)) + span.set_attribute("order.success", False) + raise +``` + **How the Trace Works Together:** The client application span tracks the initial order request, while the server span handles order processing and fulfillment. The distributed trace provides visibility into the entire purchase flow, allowing you to: @@ -389,7 +738,8 @@ The client application span tracks the initial order request, while the server s **Solution:** Comprehensive tracking of job lifecycle across queue management, processing stages, and worker performance. **Client Application Instrumentation:** -```python + +```python {tabTitle:Transaction Mode (Default)} # Celery task submission import sentry_sdk from celery import shared_task @@ -406,20 +756,20 @@ def submit_processing_job(data_file_path, priority="medium"): kwargs={"priority": priority}, queue="data_processing" ) - + span.set_data("job.id", task.id) span.set_data("job.submission_success", True) - + # Start monitoring task progress monitoring_result = setup_task_monitoring(task.id) span.set_data("monitor.callback_url", monitoring_result["callback_url"]) - + return { "job_id": task.id, "status": "submitted", "monitoring_url": monitoring_result["status_url"] } - + except Exception as e: span.set_status(sentry_sdk.consts.SPANSTATUS.INTERNAL_ERROR) span.set_data("job.submission_success", False) @@ -436,34 +786,34 @@ def process_data_file(file_path, priority="medium"): # Processing implementation with stage tracking span.set_data("processing.current_stage", "parse") data = parse_data_file(file_path) - + span.set_data("processing.current_stage", "transform") transformed_data = transform_data(data) - + span.set_data("processing.current_stage", "validate") validation_result = validate_data(transformed_data) span.set_data("validation.errors_count", len(validation_result["errors"])) - + span.set_data("processing.current_stage", "export") export_result = export_processed_data(transformed_data) - + # Record resource utilization span.set_data("resource.cpu_percent", psutil.cpu_percent()) - span.set_data("resource.memory_used_mb", + span.set_data("resource.memory_used_mb", psutil.Process().memory_info().rss / (1024 * 1024)) - + # Update final job outcome span.set_data("outcome.status", "completed") span.set_data("outcome.records_processed", len(data)) - span.set_data("outcome.output_size_bytes", + span.set_data("outcome.output_size_bytes", os.path.getsize(export_result["output_path"])) - + return { "success": True, "records_processed": len(data), "output_path": export_result["output_path"] } - + except Exception as e: span.set_status(sentry_sdk.consts.SPANSTATUS.INTERNAL_ERROR) span.set_data("outcome.status", "failed") @@ -472,6 +822,92 @@ def process_data_file(file_path, priority="medium"): raise ``` +```python {tabTitle:Stream Mode} +# Celery task submission +import sentry_sdk +from celery import shared_task +import time +from datetime import datetime, timedelta + +def submit_processing_job(data_file_path, priority="medium"): + with sentry_sdk.traces.start_span( + name="Process Job", + attributes={"sentry.op": "job"}, + ) as span: + # Submit job to Celery + try: + # Configure task and submit + task = process_data_file.apply_async( + args=[str(data_file_path)], + kwargs={"priority": priority}, + queue="data_processing" + ) + + span.set_attribute("job.id", task.id) + span.set_attribute("job.submission_success", True) + + # Start monitoring task progress + monitoring_result = setup_task_monitoring(task.id) + span.set_attribute("monitor.callback_url", monitoring_result["callback_url"]) + + return { + "job_id": task.id, + "status": "submitted", + "monitoring_url": monitoring_result["status_url"] + } + + except Exception as e: + span.status = "error" + span.set_attribute("job.submission_success", False) + span.set_attribute("error.message", str(e)) + raise + +@shared_task(name="tasks.process_data_file") +def process_data_file(file_path, priority="medium"): + with sentry_sdk.traces.start_span( + name="Process Data", + attributes={"sentry.op": "process"}, + ) as span: + try: + # Processing implementation with stage tracking + span.set_attribute("processing.current_stage", "parse") + data = parse_data_file(file_path) + + span.set_attribute("processing.current_stage", "transform") + transformed_data = transform_data(data) + + span.set_attribute("processing.current_stage", "validate") + validation_result = validate_data(transformed_data) + span.set_attribute("validation.errors_count", len(validation_result["errors"])) + + span.set_attribute("processing.current_stage", "export") + export_result = export_processed_data(transformed_data) + + # Record resource utilization + span.set_attribute("resource.cpu_percent", psutil.cpu_percent()) + span.set_attribute("resource.memory_used_mb", + psutil.Process().memory_info().rss / (1024 * 1024)) + + # Update final job outcome + span.set_attributes({ + "outcome.status": "completed", + "outcome.records_processed": len(data), + "outcome.output_size_bytes": os.path.getsize(export_result["output_path"]) + }) + + return { + "success": True, + "records_processed": len(data), + "output_path": export_result["output_path"] + } + + except Exception as e: + span.status = "error" + span.set_attribute("outcome.status", "failed") + span.set_attribute("error.message", str(e)) + raise +``` + **How the Trace Works Together:** This example shows both the job submission and processing as a single trace, which is common in Celery/distributed task patterns. The spans track the entire job lifecycle, enabling you to: diff --git a/docs/platforms/python/tracing/span-metrics/index.mdx b/docs/platforms/python/tracing/span-metrics/index.mdx index 4483c5c3be834..341840f5d45fe 100644 --- a/docs/platforms/python/tracing/span-metrics/index.mdx +++ b/docs/platforms/python/tracing/span-metrics/index.mdx @@ -17,11 +17,16 @@ Span metrics allow you to extend the default metrics that are collected by traci 1. [Adding metrics to existing spans](#adding-metrics-to-existing-spans) 2. [Creating dedicated spans with custom metrics](#creating-dedicated-metric-spans) + + This page covers both transaction mode (default) and stream mode. See{" "} + New Spans to learn more. + + ## Adding Metrics to Existing Spans You can enhance existing spans with custom metrics by adding data. This is useful when you want to augment automatic instrumentation or add contextual data to spans you've already created. -```python +```python {tabTitle:Transaction Mode (Default)} span = sentry_sdk.get_current_span() if span: # Add individual metrics @@ -32,6 +37,17 @@ if span: span.set_data("processing.duration_ms", 127) ``` +```python {tabTitle:Stream Mode} +span = sentry_sdk.traces.get_current_span() +if span: + # Add individual metrics + span.set_attribute("database.rows_affected", 42) + span.set_attribute("cache.hit_rate", 0.85) + span.set_attribute("memory.heap_used", 1024000) + span.set_attribute("queue.length", 15) + span.set_attribute("processing.duration_ms", 127) +``` + ### Best Practices for Span Data When adding metrics as span data: @@ -44,7 +60,7 @@ When adding metrics as span data: For more detailed operations, tasks, or process tracking, you can create custom dedicated spans that focus on specific metrics or attributes that you want to track. This approach provides better discoverability and more precise span configurations, however it can also create more noise in your trace waterfall. -```python +```python {tabTitle:Transaction Mode (Default)} with sentry_sdk.start_span( op="db.metrics", name="Database Query Metrics" @@ -59,43 +75,27 @@ with sentry_sdk.start_span( pass ``` +```python {tabTitle:Stream Mode} +with sentry_sdk.traces.start_span( + name="Database Query Metrics", + attributes={"sentry.op": "db.metrics"}, +) as span: + # Set metrics after creating the span + span.set_attributes({ + "db.query_type": "SELECT", + "db.table": "users", + "db.execution_time_ms": 45, + "db.rows_returned": 100, + "db.connection_pool_size": 5, + }) + # Your database operation here + pass +``` + For detailed examples of how to implement span metrics in common scenarios, see our Span Metrics Examples guide. ## Adding Metrics to All Spans -To consistently add metrics across all spans in your application, you can use the `before_send_transaction` callback: - -```python -import sentry_sdk -from sentry_sdk.types import Event, Hint - -def before_send_transaction(event: Event, hint: Hint) -> Event | None: - # Add metrics to the root span - if "trace" in event.get("contexts", {}): - if "data" not in event["contexts"]["trace"]: - event["contexts"]["trace"]["data"] = {} - - event["contexts"]["trace"]["data"].update({ - "app.version": "1.2.3", - "environment.region": "us-west-2" - }) - - # Add metrics to all child spans - for span in event.get("spans", []): - if "data" not in span: - span["data"] = {} - - span["data"].update({ - "app.component_version": "2.0.0", - "app.deployment_stage": "production" - }) - - return event - -sentry_sdk.init( - # ... - before_send_transaction=before_send_transaction -) -``` + For detailed examples of how to implement span metrics in common scenarios, see our Span Metrics Examples guide. diff --git a/docs/platforms/python/tracing/troubleshooting/index.mdx b/docs/platforms/python/tracing/troubleshooting/index.mdx index b6c136095d57e..1e68ab304394f 100644 --- a/docs/platforms/python/tracing/troubleshooting/index.mdx +++ b/docs/platforms/python/tracing/troubleshooting/index.mdx @@ -91,3 +91,9 @@ with sentry_sdk.start_span(op="request", name="setup form") as span: # ... ``` + +## Traces Miss Spans, High Memory Usage, or Data Loss After Crashes + +If you're hitting the 1,000-span limit, experiencing high memory usage from long-running processes, or losing span data when your process crashes, consider enabling stream mode. Stream mode sends spans to Sentry in batches as they finish rather than holding them in memory until the transaction ends. + +See New Spans for more information. diff --git a/platform-includes/distributed-tracing/custom-instrumentation/python.mdx b/platform-includes/distributed-tracing/custom-instrumentation/python.mdx index 8ac6dcfdea6e6..bf511e4998175 100644 --- a/platform-includes/distributed-tracing/custom-instrumentation/python.mdx +++ b/platform-includes/distributed-tracing/custom-instrumentation/python.mdx @@ -2,6 +2,12 @@ Distributed tracing works out of the box for [supported frameworks](/platforms/p This page describes how to manually propagate trace information into and out of your Python application. All you have to do is to make sure your application extracts incoming headers and to set those headers again when making an outgoing request within your application. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + ## 1. Extract Incoming Tracing Information Incoming tracing information has to be extracted and stored in memory for later use. Sentry provides the `continue_trace()` function to help you with this. Incoming tracing information can come from different places: @@ -10,9 +16,16 @@ Incoming tracing information has to be extracted and stored in memory for later - In a job queue, like Celery, it can be retrieved from meta or header variables. - You also can pick up tracing information from environment variables. + +In **transaction mode**, `sentry_sdk.continue_trace()` returns a transaction, but does not start it. To start the transaction, use `start_transaction()`. + +In **stream mode**, `sentry_sdk.traces.continue_trace()` replaces `sentry_sdk.continue_trace()` and is not a context manager. Instead, it sets the propagation context, and the next span created with `sentry_sdk.traces.start_span()` picks it up automatically. + + + Here's an example of how to extract and store incoming tracing information using `continue_trace()`: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk from my_project import get_incoming_headers_as_dict @@ -23,14 +36,43 @@ with sentry_sdk.start_transaction(transaction): ... ``` -In this example, `get_incoming_headers_as_dict()` returns a dictionary that contains tracing information from HTTP headers, environment variables, or any other mechanism your project uses to communicate with the outside world. +```python {tabTitle:Stream Mode} +import sentry_sdk +from my_project import get_incoming_headers_as_dict + +headers = get_incoming_headers_as_dict() -Sentry's `continue_trace()` function will extract the given headers, try to find the `sentry-trace` and `baggage` headers, and store them in memory for later use. +sentry_sdk.traces.continue_trace(headers) -`continue_trace()` returns a transaction, but does not start it. To start the transaction, use `start_transaction()`. +with sentry_sdk.traces.start_span(name="handle request"): + ... +``` +In these examples, `get_incoming_headers_as_dict()` returns a dictionary that contains tracing information from HTTP headers, environment variables, or any other mechanism your project uses to communicate with the outside world. + +## 2. Start a New Trace + + + This step only applies to stream mode and is not available in transaction + mode. + + +In stream mode, if you need to start a completely new trace unconnected to the current one, use `sentry_sdk.traces.new_trace()`. This is useful for background jobs or scheduled tasks where you want a clean trace boundary: + +```python +import sentry_sdk + +with sentry_sdk.traces.start_span(name="span in trace 1"): + ... + +sentry_sdk.traces.new_trace() + +with sentry_sdk.traces.start_span(name="span in trace 2"): + # This span is the root of a new, separate trace + ... +``` -## 2. Inject Tracing Information to Outgoing Requests +## 3. Inject Tracing Information to Outgoing Requests For distributed tracing to work, the two headers `sentry-trace` and `baggage`, must be added to outgoing requests. If you pregenerate HTML on the server-side, you might want to take a look at [Inject Tracing Information into Rendered HTML](#inject-tracing-information-into-rendered-html), which describes how to pass on tracing information through HTML meta tags. diff --git a/platform-includes/tracing/span-metrics/add-all-spans/python.mdx b/platform-includes/tracing/span-metrics/add-all-spans/python.mdx new file mode 100644 index 0000000000000..bcf9676d3e4d7 --- /dev/null +++ b/platform-includes/tracing/span-metrics/add-all-spans/python.mdx @@ -0,0 +1,58 @@ +How you add data to every span depends on your tracing mode: + +- Transaction mode: `before_send_transaction` runs after the entire transaction finishes and has access to all data set during the span's lifetime. +- Stream mode: `before_send_span` (set under `_experiments`) can only work with attributes set at span start time. Attributes added later during the span's lifetime are **not available**. + +```python {tabTitle:Transaction Mode (Default)} +import sentry_sdk +from sentry_sdk.types import Event, Hint + +def before_send_transaction(event: Event, hint: Hint) -> Event | None: + # Add attributes to the root span (transaction) + if "trace" in event.get("contexts", {}): + if "data" not in event["contexts"]["trace"]: + event["contexts"]["trace"]["data"] = {} + + event["contexts"]["trace"]["data"].update({ + "app_version": "1.2.3", + "environment_region": "us-west-2" + }) + + # Add attributes to all child spans + for span in event.get("spans", []): + if "data" not in span: + span["data"] = {} + + span["data"].update({ + "component_version": "2.0.0", + "deployment_stage": "production" + }) + + return event + +sentry_sdk.init( + # ... + before_send_transaction=before_send_transaction +) +``` + +```python {tabTitle:Stream Mode} +import sentry_sdk + +def before_send_span(span, hint): + # Runs once per span, as it's flushed. + # Only attributes set at span-creation time are available here. + span["attributes"].update({ + "component_version": "2.0.0", + "deployment_stage": "production", + }) + return span + +sentry_sdk.init( + # ... + _experiments={ + "trace_lifecycle": "stream", + "before_send_span": before_send_span, + }, +) +``` From 54a6df8e0751c4c1e5f4eb2a7864725aa17e04ce Mon Sep 17 00:00:00 2001 From: inventarSarah Date: Mon, 22 Jun 2026 14:49:07 +0200 Subject: [PATCH 2/5] tracing docs update stream span draft --- .../tracing/configure-sampling/index.mdx | 330 ++++++++++++++++-- .../automatic-instrumentation.mdx | 2 +- .../traces-sampler-as-sampler/python.mdx | 36 +- 3 files changed, 345 insertions(+), 23 deletions(-) diff --git a/docs/platforms/python/tracing/configure-sampling/index.mdx b/docs/platforms/python/tracing/configure-sampling/index.mdx index d2e363d691bc0..9e94b0b0ba4fd 100644 --- a/docs/platforms/python/tracing/configure-sampling/index.mdx +++ b/docs/platforms/python/tracing/configure-sampling/index.mdx @@ -18,7 +18,7 @@ This page covers both transaction mode (default) and stream mode. See stream mode) will be sampled. This works the same way in both transaction mode and stream mode: @@ -44,14 +44,11 @@ In distributed systems, implementing inheritance logic when trace information is -
-Traces Sampler Examples - -#### Traces Sampler Examples + 1. Prioritizing Critical User Flows -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk from sentry_sdk.types import SamplingContext @@ -79,14 +76,48 @@ def traces_sampler(sampling_context: SamplingContext) -> float: return 0.1 sentry_sdk.init( - dsn="your-dsn", + # ... + traces_sampler=traces_sampler, +) +``` + +```python {tabTitle:Stream Mode} +import sentry_sdk +from sentry_sdk.types import SamplingContext + +def traces_sampler(sampling_context: SamplingContext) -> float: + # Use the parent sampling decision if we have an incoming trace. + # Note: we strongly recommend respecting the parent sampling decision, + # as this ensures your traces will be complete! + parent_sampling_decision = sampling_context["span_context"]["parent_sampled"] + if parent_sampling_decision is not None: + return float(parent_sampling_decision) + + span_ctx = sampling_context["span_context"] + name = span_ctx["name"] + op = span_ctx["attributes"].get("sentry.op") + + # Sample all checkout spans + if name and ('/checkout' in name or op == 'checkout'): + return 1.0 + + # Sample 50% of login spans + if name and ('/login' in name or op == 'login'): + return 0.5 + + # Sample 10% of everything else + return 0.1 + +sentry_sdk.init( + # ... traces_sampler=traces_sampler, + _experiments={"trace_lifecycle": "stream"}, ) ``` 2. Handling Different Environments and Error Rates -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk from sentry_sdk.types import SamplingContext @@ -121,7 +152,7 @@ def traces_sampler(sampling_context: SamplingContext) -> float: # Initialize the SDK with the sampling function sentry_sdk.init( - dsn="your-dsn", + # ... traces_sampler=traces_sampler, ) @@ -136,9 +167,61 @@ with sentry_sdk.start_transaction( # your code here ``` +```python {tabTitle:Stream Mode} +import sentry_sdk +from sentry_sdk.types import SamplingContext + +def traces_sampler(sampling_context: SamplingContext) -> float: + # Use the parent sampling decision if we have an incoming trace. + # Note: we strongly recommend respecting the parent sampling decision, + # as this ensures your traces will be complete! + parent_sampling_decision = sampling_context["span_context"]["parent_sampled"] + if parent_sampling_decision is not None: + return float(parent_sampling_decision) + + custom_sampling_ctx = sampling_context["custom_sampling_context"] + environment = os.environ.get("ENVIRONMENT", "development") + + # Sample all spans in development + if environment == "development": + return 1.0 + + # Sample more spans if there are recent errors + # Note: hasRecentErrors is a custom attribute that needs to be set + if custom_sampling_ctx.get("hasRecentErrors") is True: + return 0.8 + + # Sample based on environment + if environment == "production": + return 0.05 # 5% in production + elif environment == "staging": + return 0.2 # 20% in staging + + # Default sampling rate + return 0.1 + +# Initialize the SDK with the sampling function +sentry_sdk.init( + # ... + traces_sampler=traces_sampler, + _experiments={"trace_lifecycle": "stream"}, +) + +# Custom attributes need to be set on the scope, after continue_trace +# (which resets the propagation context) and before start_span +# (which is when sampling happens), in order to be available +# in the traces_sampler +sentry_sdk.Scope.set_custom_sampling_context({"hasRecentErrors": True}) +with sentry_sdk.traces.start_span( + name="GET /api/users", + attributes={"sentry.op": "http.request"}, +) as span: + # your code here +``` + 3. Controlling Sampling Based on User and Transaction Properties -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk from sentry_sdk.types import SamplingContext @@ -172,7 +255,7 @@ def traces_sampler(sampling_context: SamplingContext) -> float: # Initialize the SDK with the sampling function sentry_sdk.init( - dsn="your-dsn", + # ... traces_sampler=traces_sampler, ) @@ -185,9 +268,59 @@ with sentry_sdk.start_transaction( # Your code here ``` +```python {tabTitle:Stream Mode} +import sentry_sdk +from sentry_sdk.types import SamplingContext + +def traces_sampler(sampling_context: SamplingContext) -> float: + # Use the parent sampling decision if we have an incoming trace. + # Note: we strongly recommend respecting the parent sampling decision, + # as this ensures your traces will be complete! + parent_sampling_decision = sampling_context["span_context"]["parent_sampled"] + if parent_sampling_decision is not None: + return float(parent_sampling_decision) + + custom_sampling_ctx = sampling_context["custom_sampling_context"] + + # Always sample for premium users + # Note: user.tier is a custom attribute that needs to be set + if custom_sampling_ctx.get("user", {}).get("tier") == "premium": + return 1.0 + + # Sample more spans for users experiencing errors + # Note: hasRecentErrors is a custom attribute + if custom_sampling_ctx.get("hasRecentErrors") is True: + return 0.8 + + # Sample less for high-volume, low-value paths + name = sampling_context["span_context"]["name"] + if name and name.startswith("/api/metrics"): + return 0.01 + + # Default sampling rate + return 0.2 + +# Initialize the SDK with the sampling function +sentry_sdk.init( + # ... + traces_sampler=traces_sampler, + _experiments={"trace_lifecycle": "stream"}, +) + +# To set custom attributes for this example: +sentry_sdk.Scope.set_custom_sampling_context( + {"user": {"tier": "premium"}, "hasRecentErrors": True} +) +with sentry_sdk.traces.start_span( + name="GET /api/users", + attributes={"sentry.op": "http.request"}, +) as span: + # Your code here +``` + 4. Complex Business Logic Sampling -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk from sentry_sdk.types import SamplingContext @@ -232,7 +365,7 @@ def traces_sampler(sampling_context: SamplingContext) -> float: # Initialize the SDK with the sampling function sentry_sdk.init( - dsn="your-dsn", + # ... traces_sampler=traces_sampler, ) @@ -243,12 +376,72 @@ with sentry_sdk.start_transaction( custom_sampling_context={"user": {"segment": "enterprise"}, "transaction": {"value": 1500}, "service": {"error_rate": 0.03}}, ) as transaction: # Your code here +``` +```python {tabTitle:Stream Mode} +import sentry_sdk +from sentry_sdk.types import SamplingContext + +def traces_sampler(sampling_context: SamplingContext) -> float: + # Use the parent sampling decision if we have an incoming trace. + # Note: we strongly recommend respecting the parent sampling decision, + # as this ensures your traces will be complete! + parent_sampling_decision = sampling_context["span_context"]["parent_sampled"] + if parent_sampling_decision is not None: + return float(parent_sampling_decision) + + # Always sample critical business operations + # Note: sentry.op is an SDK-provided attribute + span_ctx = sampling_context["span_context"] + if span_ctx["attributes"].get("sentry.op") in ["payment.process", "order.create", "user.verify"]: + return 1.0 + + custom_sampling_context = sampling_context["custom_sampling_context"] + + # Sample based on user segment + # Note: user.segment is a custom attribute + user_segment = custom_sampling_context.get("user", {}).get("segment") + if user_segment == "enterprise": + return 0.8 + elif user_segment == "premium": + return 0.5 + + # Sample based on transaction value + # Note: transaction.value is a custom attribute + transaction_value = custom_sampling_context.get("transaction", {}).get("value") + if transaction_value is not None and transaction_value > 1000: # High-value transactions + return 0.7 + + # Sample based on error rate in the service + # Note: service.error_rate is a custom attribute + error_rate = custom_sampling_context.get("service", {}).get("error_rate") + if error_rate is not None and error_rate > 0.05: # Error rate above 5% + return 0.9 + + # Default sampling rate + return 0.1 + +# Initialize the SDK with the sampling function +sentry_sdk.init( + # ... + traces_sampler=traces_sampler, + _experiments={"trace_lifecycle": "stream"}, +) + +# To set custom attributes for this example: +sentry_sdk.Scope.set_custom_sampling_context( + {"user": {"segment": "enterprise"}, "transaction": {"value": 1500}, "service": {"error_rate": 0.03}} +) +with sentry_sdk.traces.start_span( + name="Process Payment", + attributes={"sentry.op": "payment.process"}, +) as span: + # Your code here ``` 5. Performance-Based Sampling -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk from sentry_sdk.types import SamplingContext @@ -285,7 +478,7 @@ def traces_sampler(sampling_context: SamplingContext) -> float: # Initialize the SDK with the sampling function sentry_sdk.init( - dsn="your-dsn", + # ... traces_sampler=traces_sampler, ) @@ -298,12 +491,67 @@ with sentry_sdk.start_transaction( # Your code here ``` -
+```python {tabTitle:Stream Mode} +import sentry_sdk +from sentry_sdk.types import SamplingContext + +def traces_sampler(sampling_context: SamplingContext) -> float: + # Use the parent sampling decision if we have an incoming trace. + # Note: we strongly recommend respecting the parent sampling decision, + # as this ensures your traces will be complete! + parent_sampling_decision = sampling_context["span_context"]["parent_sampled"] + if parent_sampling_decision is not None: + return float(parent_sampling_decision) + + custom_sampling_ctx = sampling_context["custom_sampling_context"] + + # Sample more spans with high memory usage + # Note: memory_usage_mb is a custom attribute + memory_usage = custom_sampling_ctx.get("memory_usage_mb") + if memory_usage is not None and memory_usage > 500: + return 0.8 + + # Sample more spans with high CPU usage + # Note: cpu_percent is a custom attribute + cpu_percent = custom_sampling_ctx.get("cpu_percent") + if cpu_percent is not None and cpu_percent > 80: + return 0.8 + + # Sample more spans with high database load + # Note: db_connections is a custom attribute + db_connections = custom_sampling_ctx.get("db_connections") + if db_connections is not None and db_connections > 100: + return 0.7 + + # Default sampling rate + return 0.1 + +# Initialize the SDK with the sampling function +sentry_sdk.init( + # ... + traces_sampler=traces_sampler, + _experiments={"trace_lifecycle": "stream"}, +) + +# To set custom attributes for this example: +sentry_sdk.Scope.set_custom_sampling_context( + {"memory_usage_mb": 600, "cpu_percent": 85, "db_connections": 120} +) +with sentry_sdk.traces.start_span( + name="Process Data", + attributes={"sentry.op": "data.process"}, +) as span: + # Your code here +``` + + ## The Sampling Context Object When the `traces_sampler` function is called, the Sentry SDK passes a `sampling_context` object with information from the relevant span to help make sampling decisions: +### Transaction Mode (Default) + ```python { "transaction_context": { @@ -317,10 +565,28 @@ When the `traces_sampler` function is called, the Sentry SDK passes a `sampling_ } ``` +### Stream Mode + +```python +{ + "span_context": { + "name": str, # span title at creation time (SDK-provided) + "trace_id": str, # the trace ID (SDK-provided) + "parent_span_id": Optional[str], # the parent span's ID, if any (SDK-provided) + "parent_sampled": Optional[bool], # whether the parent span was sampled (SDK-provided) + "parent_sample_rate": Optional[float], # the sample rate used by the parent (SDK-provided) + "attributes": dict[str, Any] # attributes set at span-creation time (SDK- and user-provided) + }, + "custom_sampling_context": Optional[dict[str, Any]] # additional custom data for sampling +} +``` + ### SDK-Provided vs. Custom Attributes The sampling context contains both SDK-provided attributes and custom attributes: +#### Transaction Mode (Default) + **SDK-Provided Attributes:** - `transaction_context.name`: The name of the transaction @@ -332,15 +598,37 @@ The sampling context contains both SDK-provided attributes and custom attributes - Any data you add to the `custom_sampling_context` parameter in `start_transaction`. Use this for data that you want to use for sampling decisions but don't want to include in the transaction data that gets sent to Sentry. Read more about sampling context [here](/platforms/python/configuration/sampling/#sampling-context). +#### Stream Mode + +**SDK-Provided Attributes:** + +- `span_context.name`: The name of the span +- `span_context.attributes`: Attributes set on the span at creation time, including `sentry.op` +- `span_context.parent_sampled`: Whether the parent span was sampled +- `span_context.parent_sample_rate`: The sample rate used by the parent + +**Custom Attributes:** + +- Any data you add via `sentry_sdk.Scope.set_custom_sampling_context()`, called after `continue_trace` and before `start_span`. Use this for data that you want to use for sampling decisions but don't want to include in the span data that gets sent to Sentry. Read more about sampling context [here](/platforms/python/configuration/sampling/#sampling-context). + ## Sampling Decision Precedence When multiple sampling mechanisms could apply, Sentry follows this order of precedence: -1. If a sampling decision is passed to `start_transaction`, that decision is used -2. If `traces_sampler` is defined, its decision is used. Although the `traces_sampler` can override the parent sampling decision, most users will want to ensure their `traces_sampler` respects the parent sampling decision -3. If no `traces_sampler` is defined, but there is a parent sampling decision from an incoming distributed trace, we use the parent sampling decision -4. If neither of the above, `traces_sample_rate` is used -5. If none of the above are set, no transactions are sampled. This is equivalent to setting `traces_sample_rate=0.0` +### Transaction Mode (Default) + +1. If a sampling decision is passed to `start_transaction`, that decision is used. +2. If `traces_sampler` is defined, its decision is used. Although the `traces_sampler` can override the parent sampling decision, most users will want to ensure their `traces_sampler` respects the parent sampling decision. +3. If no `traces_sampler` is defined, but there is a parent sampling decision from an incoming distributed trace, we use the parent sampling decision. +4. If neither of the above, `traces_sample_rate` is used. +5. If none of the above are set, no transactions are sampled. This is equivalent to setting `traces_sample_rate=0.0`. + +### Stream Mode + +1. If `traces_sampler` is defined, its decision is used. Although the `traces_sampler` can override the parent sampling decision, most users will want to ensure their `traces_sampler` respects the parent sampling decision. +2. If no `traces_sampler` is defined, but there is a parent sampling decision from an incoming distributed trace, we use the parent sampling decision. +3. If neither of the above, `traces_sample_rate` is used +4. If none of the above are set, no spans are sampled. This is equivalent to setting `traces_sample_rate=0.0`. ## How Sampling Propagates in Distributed Traces diff --git a/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx b/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx index 4d4387c3425d5..e079f89270ef3 100644 --- a/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx +++ b/docs/platforms/python/tracing/instrumentation/automatic-instrumentation.mdx @@ -35,4 +35,4 @@ Spans are instrumented for the following operations within a transaction/service In transaction mode, spans are only created within an existing transaction. If you're not using any of the supported frameworks, you'll need to create transactions manually. -In stream mode, any span without a parent is a service span (equivalent to transactions). You can force any span to be a service by removing its parent. See Custom Instrumentation to learn more. +Stream mode removes this limitation. Since there are no transactions, any span started without a parent is automatically promoted to a service span (the equivalent of a transaction). You can also force any span to become a service span when starting it by removing its parent. See Custom Instrumentation to learn more. diff --git a/platform-includes/performance/traces-sampler-as-sampler/python.mdx b/platform-includes/performance/traces-sampler-as-sampler/python.mdx index a8c2a6f44cca2..40f49bb0d7fbe 100644 --- a/platform-includes/performance/traces-sampler-as-sampler/python.mdx +++ b/platform-includes/performance/traces-sampler-as-sampler/python.mdx @@ -1,4 +1,4 @@ -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk from sentry_sdk.types import SamplingContext @@ -30,3 +30,37 @@ sentry_sdk.init( traces_sampler=traces_sampler, ) ``` + +```python {tabTitle:Stream Mode} +import sentry_sdk +from sentry_sdk.types import SamplingContext + +def traces_sampler(sampling_context: SamplingContext) -> float: + # Use the parent sampling decision if we have an incoming trace. + # Note: we strongly recommend respecting the parent sampling decision, + # as this ensures your traces will be complete! + parent_sampling_decision = sampling_context["span_context"]["parent_sampled"] + if parent_sampling_decision is not None: + return float(parent_sampling_decision) + + # Examine provided sampling context along with anything in the + # global namespace to compute the sample rate for this span + if "...": + # These are important - take a big sample + return 0.5 + elif "...": + # These are less important - only take 1% + return 0.01 + elif "...": + # These aren't worth tracking - drop these spans + return 0 + + # Default sample rate + return 0.1 + +sentry_sdk.init( + # ... + traces_sampler=traces_sampler, + _experiments={"trace_lifecycle": "stream"}, +) +``` From 63cf4af64be505637b33c31797ad4bab249ffe37 Mon Sep 17 00:00:00 2001 From: inventarSarah Date: Tue, 23 Jun 2026 11:24:02 +0200 Subject: [PATCH 3/5] draft tracing span first --- .../custom-instrumentation/index.mdx | 297 ++++++++++++++---- .../python/tracing/span-lifecycle/index.mdx | 38 +-- .../python/tracing/span-metrics/index.mdx | 5 +- .../performance/dropping-spans/python.mdx | 37 +++ 4 files changed, 271 insertions(+), 106 deletions(-) create mode 100644 platform-includes/performance/dropping-spans/python.mdx diff --git a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx index 24790f07d18de..c6874f4037a32 100644 --- a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx +++ b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/index.mdx @@ -11,24 +11,25 @@ This page covers both transaction mode (default) and stream mode. See -## Add a Transaction + + +The parameter `name` in `start_span()` used to be called `description`. In version 2.15.0 `description` was deprecated and from 2.15.0 on, only `name` should be used. `description` will be removed in `3.0.0`. - - Transactions are only available in transaction mode. In stream mode, - `start_transaction()` isn't available and you create spans instead. -Adding transactions will allow you to instrument and capture certain regions of your code. +## Add a Transaction/Service Span + +Adding transactions or service spans (in stream mode) will allow you to instrument and capture certain regions of your code. -If you're using one of Sentry's SDK integrations, transactions will be created for you automatically. +If you're using one of Sentry's SDK integrations, transactions/service spans will be created for you automatically. -The following example creates a transaction for an expensive operation (in this case, `eat_pizza`), and then sends the result to Sentry: +The following example creates a transaction/service span for an expensive operation (in this case, `eat_pizza`), and then sends the result to Sentry: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk def eat_slice(slice): @@ -40,27 +41,44 @@ def eat_pizza(pizza): eat_slice(pizza.slices.pop()) ``` -The [API reference](https://getsentry.github.io/sentry-python/api.html#sentry_sdk.api.start_transaction) documents `start_transaction` and all its parameters. +```python {tabTitle:Stream Mode} +import sentry_sdk + +def eat_slice(slice): + ... + +def eat_pizza(pizza): + with sentry_sdk.traces.start_span( + name="Eat Pizza", + attributes={"sentry.op": "task"}, + # remove the parent span to create service span + parent_span=None, + ): + while pizza.slices > 0: + eat_slice(pizza.slices.pop()) +``` + +The [API reference](https://getsentry.github.io/sentry-python/api.html#performance-monitoring) documents `start_transaction` and `start_span`, along with their parameters. -Note that `sentry_sdk.start_transaction()` is meant be used as a context manager. This ensures that the transaction will be properly set as active and any spans created within will be attached to it. +Note that `sentry_sdk.start_transaction()` (transaction mode) and `sentry_sdk.traces.start_span()` (stream mode) are meant be used as context managers. This ensures that the transaction/service span will be properly set as active and any spans created within will be attached to it. -## Add Spans to a Transaction +## Add Spans to a Transaction/Service Span -If you want to have more fine-grained performance monitoring, you can add child spans to your transaction, which can be done by either: +If you want to have more fine-grained performance monitoring, you can add child spans to your transaction/service span, which can be done by either: - Using a context manager - Using a decorator (this works on sync and async functions) - Manually starting and finishing a span -Calling `sentry_sdk.start_span()` will find the current active transaction and attach the span to it. +Calling `sentry_sdk.start_span()` in transaction mode or `sentry_sdk.traces.start_span()` in stream mode will find the current active transaction/span and attach the new span to it. ### Using a Context Manager -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk def eat_slice(slice): @@ -73,15 +91,26 @@ def eat_pizza(pizza): eat_slice(pizza.slices.pop()) ``` - +```python {tabTitle:Stream Mode} +import sentry_sdk -The parameter `name` in `start_span()` used to be called `description`. In version 2.15.0 `description` was deprecated and from 2.15.0 on, only `name` should be used. `description` will be removed in `3.0.0`. +def eat_slice(slice): + ... - +def eat_pizza(pizza): + with sentry_sdk.traces.start_span( + name="Eat Pizza", + attributes={"sentry.op": "task"}, + parent_span=None, + ): + while pizza.slices > 0: + with sentry_sdk.traces.start_span(name="Eat Slice"): + eat_slice(pizza.slices.pop()) +``` ### Using a Decorator -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk @sentry_sdk.trace @@ -94,17 +123,34 @@ def eat_pizza(pizza): eat_slice(pizza.slices.pop()) ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +@sentry_sdk.traces.trace +def eat_slice(slice): + ... + +def eat_pizza(pizza): + with sentry_sdk.traces.start_span( + name="Eat Pizza", + attributes={"sentry.op": "task"}, + parent_span=None, + ): + while pizza.slices > 0: + eat_slice(pizza.slices.pop()) +``` + See the [@sentry_sdk.trace decoration section](#sentry_sdktrace-decorator) below for more details. - + -When tracing a static or class method, you **must** add the `@sentry_sdk.trace` decorator **after** the `@staticmethod` or `@classmethod` decorator (i.e., **closer** to the function definition). Otherwise, your function will break! +When tracing a static or class method, you **must** add the `@sentry_sdk.trace` decorator **after** the `@staticmethod` or `@classmethod` decorator (i.e., **closer** to the function definition). Otherwise, your function will break! This applies in both transaction mode and stream mode. ### Manually Starting and Finishing a Span -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk def eat_slice(slice): @@ -118,11 +164,23 @@ def eat_pizza(pizza): span.finish() ``` - +```python {tabTitle:Stream Mode} +import sentry_sdk -The parameter `name` in `start_span()` used to be called `description`. In version 2.15.0 `description` was deprecated and from 2.15.0 on, only `name` should be used. `description` will be removed in `3.0.0`. +def eat_slice(slice): + ... - +def eat_pizza(pizza): + with sentry_sdk.traces.start_span( + name="Eat Pizza", + attributes={"sentry.op": "task"}, + parent_span=None, + ): + while pizza.slices > 0: + span = sentry_sdk.traces.start_span(name="Eat Slice") + eat_slice(pizza.slices.pop()) + span.finish() +``` When you create your span manually, make sure to call `span.finish()` after the block of code you want to wrap in a span to finish the span. If you do not finish the span it will not be sent to Sentry. @@ -132,7 +190,7 @@ Spans can be nested to form a span tree. If you'd like to learn more, read our [ ### Using a Context Manager -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk def chew(): @@ -144,15 +202,21 @@ def eat_slice(slice): chew() ``` - +```python {tabTitle:Stream Mode} +import sentry_sdk -The parameter `name` in `start_span()` used to be called `description`. In version 2.15.0 `description` was deprecated and from 2.15.0 on, only `name` should be used. `description` will be removed in `3.0.0`. +def chew(): + ... - +def eat_slice(slice): + with sentry_sdk.traces.start_span(name="Eat Slice"): + with sentry_sdk.traces.start_span(name="Chew"): + chew() +``` ### Using a Decorator -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk @sentry_sdk.trace @@ -164,11 +228,23 @@ def eat_slice(slice): chew() ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +@sentry_sdk.traces.trace +def chew(): + ... + +@sentry_sdk.traces.trace +def eat_slice(slice): + chew() +``` + See the [@sentry_sdk.trace decoration section](#sentry_sdktrace-decorator) below for more details. -### Manually +### Manually Starting and Finishing -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk def chew(): @@ -184,32 +260,72 @@ def eat_slice(slice): parent_span.finish() ``` - +```python {tabTitle:Stream Mode} +import sentry_sdk -The parameter `name` in `start_span()` used to be called `description`. In version 2.15.0 `description` was deprecated and from 2.15.0 on, only `name` should be used. `description` will be removed in `3.0.0`. +def chew(): + ... - +def eat_slice(slice): + parent_span = sentry_sdk.traces.start_span(name="Eat Slice") + + # pass the parent span via `parent_span` to create a child span + child_span = sentry_sdk.traces.start_span(name="Chew", parent_span=parent_span) + chew() + child_span.finish() + + parent_span.finish() +``` -The parameters of `start_span()` and `start_child()` are the same. See the [API reference](https://getsentry.github.io/sentry-python/api.html#sentry_sdk.api.start_span) for more details. +In transaction mode, the parameters of `start_span()` and `start_child()` are the same. See the [API reference](https://getsentry.github.io/sentry-python/api.html#sentry_sdk.api.start_span) for more details. +In stream mode, however, there's no separate `start_child()` method. Instead, pass the parent span explicitly via `parent_span` when starting the child (as shown in the example above). When you create your span manually, make sure to call `span.finish()` after the block of code you want to wrap in a span to finish the span. If you do not finish the span it will not be sent to Sentry. +### Sibling Spans + + + +Only available in stream mode. + + + +In stream mode, a span normally attaches to whatever span is currently active. To control parentage explicitly, for example to make a span a sibling instead of a child, pass `parent_span` directly: + +```python +import sentry_sdk + +def eat_slice(slice): + ... + +def chew(): + ... + +def eat_pizza(pizza): + with sentry_sdk.traces.start_span(name="Eat Pizza") as pizza_span: + with sentry_sdk.traces.start_span(name="Eat Slice"): + with sentry_sdk.traces.start_span(name="Chew", parent_span=pizza_span): + # "Chew" is a sibling of "Eat Slice", not its child + chew() + eat_slice(pizza.slices.pop()) +``` + ## @sentry_sdk.trace decorator -You can set `op`, `name` and `attributes` parameters in the `@sentry_sdk.trace` decorator to customize your spans. Attribute values can only be primitive types (like `int`, `float`, `bool`, `str`) or a list of those types without mixing types. +In transaction mode, you can set `op`, `name` and `attributes` parameters in the `@sentry_sdk.trace` decorator to customize your spans. Attribute values can only be primitive types (like `int`, `float`, `bool`, `str`) or a list of those types without mixing types. -When tracing a static or class method, you **must** add the `@sentry_sdk.trace` decorator **after** the `@staticmethod` or `@classmethod` decorator (i.e., **closer** to the function definition). Otherwise, your function will break. +In stream mode, the decorator is `@sentry_sdk.traces.trace` and accepts `name`, `attributes`, and `active`. Note that there's no `op` parameter; set the operation as the `sentry.op` key in `attributes` instead. Attribute values can only be primitive types (like `int`, `float`, `bool`, `str`) or a list of those types without mixing types. - + -The parameters `op`, `name` and `attributes` were added to the `@sentry_sdk.trace` decorator in version 2.35.0. +When tracing a static or class method, you **must** add the decorator **after** the `@staticmethod` or `@classmethod` decorator (i.e., **closer** to the function definition). Otherwise, your function will break. -```python {diff} +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk -+@sentry_sdk.trace(op="my_op", name="Paul", attributes={"x": True}) + @sentry_sdk.trace(op="my_op", name="Paul", attributes={"x": True}) def my_function(i): ... @@ -221,6 +337,21 @@ The parameters `op`, `name` and `attributes` were added to the `@sentry_sdk.trac root_function() ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + ++@sentry_sdk.traces.trace(name="Paul", attributes={"sentry.op": "my_op", "x": True}) + def my_function(i): + ... + + @sentry_sdk.traces.trace + def root_function(): + for i in range(3): + my_function(i) + + root_function() +``` + The code above will customize the `my_function` spans like this: ```mermaid @@ -236,6 +367,12 @@ gantt ### Span Templates + + +The `template` parameter is only available in transaction mode. + + + In the `@sentry_sdk.trace` decorator you can also specify a `template`. This helps create spans that follow a certain template. Currently this is only available for spans that are created for the [AI Agents instrumentation](/platforms/python/tracing/instrumentation/custom-instrumentation/ai-agents-module/#spans) of Sentry. Available templates are `AI_AGENT`, `AI_TOOL`, and `AI_CHAT`. @@ -308,9 +445,15 @@ To enable performance monitoring for the functions specified in `functions_to_tr ## Accessing the Current Transaction + + +Only available in transaction mode. + + + The `sentry_sdk.get_current_scope().transaction` property returns the active transaction or `None` if no transaction is active. You can use this property to modify data on the transaction. -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk def eat_pizza(pizza): @@ -325,11 +468,11 @@ def eat_pizza(pizza): ## Accessing the Current Span -To change data in the current span, use ` sentry_sdk.get_current_span()`. This function will return a span if there's one running, otherwise it will return `None`. +To change data in the current span, use ` sentry_sdk.get_current_span()` (transaction mode) or `sentry_sdk.traces.get_current_span()` (stream mode). This function will return a span if there's one running, otherwise it will return `None`. -In this example, we'll set a tag in the span created by the `@sentry_sdk.trace` decorator. +In this example, we'll set custom data in the span created by the `@sentry_sdk.trace` decorator. -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk @sentry_sdk.trace @@ -340,11 +483,22 @@ def eat_slice(slice): span.set_tag("slice_id", slice.id) ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +@sentry_sdk.trace +def eat_slice(slice): + span = sentry_sdk.traces.get_current_span() + + if span is not None: + span.set_attribute("slice_id", slice.id) +``` + ## Improving Data on Transactions and Spans -You can add data attributes to your transactions. This data is visible in the trace explorer in Sentry. Data attributes can be of type `string`, `number` or `boolean`, as well as (non-mixed) arrays of these types: +In transaction mode, you can add data attributes to your transactions. This data is visible in the trace explorer in Sentry. Data attributes can be of type `string`, `number` or `boolean`, as well as (non-mixed) arrays of these types: -```python +```python {tabTitle:Transaction Mode (Default)} with sentry_sdk.start_transaction(name="my-transaction") as transaction: transaction.set_data("my-data-attribute-1", "value1") transaction.set_data("my-data-attribute-2", 42) @@ -355,9 +509,9 @@ with sentry_sdk.start_transaction(name="my-transaction") as transaction: transaction.set_data("my-data-attribute-6", [True, False, True]) ``` -You can add data attributes to your spans the same way, with the same type restrictions as described above. +You can add data attributes to any span the same way, with the same type restrictions as described above. -```python +```python {tabTitle:Transaction Mode (Default)} with sentry_sdk.start_span(name="my-span") as span: span.set_data("my-data-attribute-1", "value1") span.set_data("my-data-attribute-2", 42) @@ -368,32 +522,41 @@ with sentry_sdk.start_span(name="my-span") as span: span.set_data("my-data-attribute-6", [True, False, True]) ``` -To attach data attributes to the transaction and all its spans, you can use `before_send_transaction`: +```python {tabTitle:Stream Mode} +with sentry_sdk.traces.start_span(name="my-span") as span: + span.set_attribute("my-data-attribute-1", "value1") + span.set_attribute("my-data-attribute-2", 42) + span.set_attribute("my-data-attribute-3", True) + + # or use set_attributes to set multiple attributes at once + span.set_attributes({ + "my-data-attribute-4": ["value1", "value2", "value3"], + "my-data-attribute-5": [42, 43, 44], + "my-data-attribute-6": [True, False, True] + }) +``` -```python -import sentry_sdk -from sentry_sdk.types import Event, Hint + -def before_send_transaction(event: Event, hint: Hint) -> Event | None: - # Set the data attribute "foo" to "bar" on every span belonging to this - # transaction event - for span in event["spans"]: - span["data"]["foo"] = "bar" + - # Set the data on the transaction itself, too - event["contexts"]["trace"]["data"]["foo"] = "bar" +`before_send_span` can only modify span data and you cannot use it to drop spans. See [Dropping Spans](#dropping-spans) below. - return event + -sentry_sdk.init( - traces_sample_rate=1.0, - before_send_transaction=before_send_transaction, -) -``` +## Dropping Spans + + ## Update Current Span Shortcut -You can update the data of the currently running span using the `sentry_sdk.update_current_span()` function. You can set `op`, `name` and `attributes` to update your span. Attribute values can only be primitive types (like `int`, `float`, `bool`, `str`) or a list of those types without mixing types. + + +`sentry_sdk.update_current_span()` isn't available in stream mode. Retrieve the current span with `sentry_sdk.traces.get_current_span()` and update its attributes using `set_attribute()` or `set_attributes()`. + + + +In transaction mode, you can update the data of the currently running span using the `sentry_sdk.update_current_span()` function. You can set `op`, `name` and `attributes` to update your span. Attribute values can only be primitive types (like `int`, `float`, `bool`, `str`) or a list of those types without mixing types. ```python {diff} import sentry_sdk diff --git a/docs/platforms/python/tracing/span-lifecycle/index.mdx b/docs/platforms/python/tracing/span-lifecycle/index.mdx index 48116b4ac34c7..aa0e8eb6a5de3 100644 --- a/docs/platforms/python/tracing/span-lifecycle/index.mdx +++ b/docs/platforms/python/tracing/span-lifecycle/index.mdx @@ -409,40 +409,4 @@ with sentry_sdk.traces.start_span( ## Dropping Spans -`ignore_span` is only available in stream mode. - -In stream mode, you can prevent specific spans from being created using `ignore_spans`, set under `_experiments`. It evaluates rules at span start, which can be strings, compiled regexes, or dictionaries with `name` and/or `attributes` conditions. If the dropped span is a service span, its children are dropped too. If it's a child span, only that span is dropped and its children are reparented to the nearest ancestor. - -```python -import re -import sentry_sdk - -sentry_sdk.init( - _experiments={ - "trace_lifecycle": "stream", - "ignore_spans": [ - # String match against span name - "/health", - - # Regex match against span name - re.compile(r"/flow/.*"), - - # Match by attributes (all must match) - { - "attributes": { - "service.id": "15def9a", - "flow.pipeline": "legacy", - } - }, - - # Match by name and attributes - { - "name": re.compile(r"/flow/.*"), - "attributes": { - "service.id": re.compile(r".*\.facade"), - }, - }, - ], - }, -) -``` + diff --git a/docs/platforms/python/tracing/span-metrics/index.mdx b/docs/platforms/python/tracing/span-metrics/index.mdx index 341840f5d45fe..a280d813c6c60 100644 --- a/docs/platforms/python/tracing/span-metrics/index.mdx +++ b/docs/platforms/python/tracing/span-metrics/index.mdx @@ -18,8 +18,9 @@ Span metrics allow you to extend the default metrics that are collected by traci 2. [Creating dedicated spans with custom metrics](#creating-dedicated-metric-spans) - This page covers both transaction mode (default) and stream mode. See{" "} - New Spans to learn more. + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + ## Adding Metrics to Existing Spans diff --git a/platform-includes/performance/dropping-spans/python.mdx b/platform-includes/performance/dropping-spans/python.mdx new file mode 100644 index 0000000000000..71dea6d207cdc --- /dev/null +++ b/platform-includes/performance/dropping-spans/python.mdx @@ -0,0 +1,37 @@ +`ignore_span` is only available in stream mode. + +In stream mode, you can prevent specific spans from being created using `ignore_spans`, set under `_experiments`. It evaluates rules at span start, which can be strings, compiled regexes, or dictionaries with `name` and/or `attributes` conditions. If the dropped span is a service span, its children are dropped too. If it's a child span, only that span is dropped and its children are reparented to the nearest ancestor. + +```python +import re +import sentry_sdk + +sentry_sdk.init( + _experiments={ + "trace_lifecycle": "stream", + "ignore_spans": [ + # String match against span name + "/health", + + # Regex match against span name + re.compile(r"/flow/.*"), + + # Match by attributes (all must match) + { + "attributes": { + "service.id": "15def9a", + "flow.pipeline": "legacy", + } + }, + + # Match by name and attributes + { + "name": re.compile(r"/flow/.*"), + "attributes": { + "service.id": re.compile(r".*\.facade"), + }, + }, + ], + }, +) +``` From 7fb063c4f56904bedbb644d275598af67a1cd12c Mon Sep 17 00:00:00 2001 From: inventarSarah Date: Tue, 23 Jun 2026 13:23:44 +0200 Subject: [PATCH 4/5] fix punctuation --- .../platforms/python/tracing/configure-sampling/index.mdx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/platforms/python/tracing/configure-sampling/index.mdx b/docs/platforms/python/tracing/configure-sampling/index.mdx index 9e94b0b0ba4fd..6356a331280ca 100644 --- a/docs/platforms/python/tracing/configure-sampling/index.mdx +++ b/docs/platforms/python/tracing/configure-sampling/index.mdx @@ -630,6 +630,12 @@ When multiple sampling mechanisms could apply, Sentry follows this order of prec 3. If neither of the above, `traces_sample_rate` is used 4. If none of the above are set, no spans are sampled. This is equivalent to setting `traces_sample_rate=0.0`. + + +Sampling decisions are made when spans are created. Child spans inherit the sampling decision of their parent span unless filtered by `ignore_spans`. + + + ## How Sampling Propagates in Distributed Traces Sentry uses a "head-based" sampling approach: @@ -642,4 +648,4 @@ The two key headers are: - `sentry-trace`: Contains trace ID, span ID, and sampling decision - `baggage`: Contains additional trace metadata including sample rate -The Sentry Python SDK automatically attaches these headers to outgoing HTTP requests when using auto-instrumentation with libraries like `requests`, `urllib3`, or `httpx`. For other communication channels, you can manually propagate trace information. Learn more about customizing tracing in [custom trace propagation](/platforms/python/tracing/distributed-tracing/custom-trace-propagation/) +The Sentry Python SDK automatically attaches these headers to outgoing HTTP requests when using auto-instrumentation with libraries like `requests`, `urllib3`, or `httpx`. For other communication channels, you can manually propagate trace information. Learn more about customizing tracing in [custom trace propagation](/platforms/python/tracing/distributed-tracing/custom-trace-propagation/). From a5bb204fbc2dbc06a9048cd0a58763f2269aa37c Mon Sep 17 00:00:00 2001 From: inventarSarah Date: Wed, 24 Jun 2026 08:05:12 +0200 Subject: [PATCH 5/5] update pages under custom instrumentation for stream mode --- .../ai-agents-module.mdx | 153 +++++++++++++++++- .../custom-instrumentation/caches-module.mdx | 75 ++++++++- .../custom-instrumentation/mcp-module.mdx | 136 +++++++++++++++- .../custom-instrumentation/queues-module.mdx | 146 ++++++++++++++--- .../requests-module.mdx | 36 ++++- .../ai-agents-module/ai-client-span.mdx | 34 ++-- .../ai-agents-module/execute-tool-span.mdx | 6 +- .../tracing/ai-agents-module/handoff-span.mdx | 2 +- .../ai-agents-module/invoke-agent-span.mdx | 8 +- .../mcp-module/prompt-retrieval-span.mdx | 4 +- .../tracing/mcp-module/resource-read-span.mdx | 4 +- .../mcp-module/tool-execution-span.mdx | 4 +- 12 files changed, 539 insertions(+), 69 deletions(-) diff --git a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/ai-agents-module.mdx b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/ai-agents-module.mdx index 566e9c354f64b..8b307cdb12ab8 100644 --- a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/ai-agents-module.mdx +++ b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/ai-agents-module.mdx @@ -10,6 +10,13 @@ With Sentry AI Agent Monitoringset up tracing. Once this is done, the Python SDK will automatically instrument AI agents created with supported libraries. If that doesn't fit your use case, you can use custom instrumentation described below. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + + ## Automatic Instrumentation The Python SDK supports automatic instrumentation for some AI libraries. We recommend adding their integrations to your Sentry configuration to automatically capture spans for AI agents. @@ -27,7 +34,8 @@ The Python SDK supports automatic instrumentation for some AI libraries. We reco For your AI agents data to show up in the [AI Agents Dashboards](https://sentry.io/orgredirect/organizations/:orgslug/dashboards/?filter=onlyPrebuilt&query=agents&sort=mostPopular), at least one of the AI spans needs to be created and have well-defined names and data attributes. See below. -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create these spans. +In transaction mode, you can also use the [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator to create these spans, using its `template` parameter. +In stream mode, these spans need to be created directly with `sentry_sdk.traces.start_span()`, as shown below. ## Span Hierarchy @@ -55,7 +63,7 @@ This span represents a request to an LLM model or service that generates a respo #### Example AI Request span -```python +```python {tabTitle:Transaction Mode (Default)} import json import sentry_sdk @@ -78,11 +86,41 @@ with sentry_sdk.start_span(op="gen_ai.chat", name="chat o3-mini") as span: span.set_data("gen_ai.usage.output_tokens", result.usage.completion_tokens) ``` +```python {tabTitle:Stream Mode} +import json +import sentry_sdk + +messages = [{"role": "user", "parts": [{"type": "text", "content": "Tell me a joke"}]}] + +with sentry_sdk.traces.start_span( + name="chat o3-mini", + attributes={"sentry.op": "gen_ai.chat"}, +) as span: + span.set_attributes({ + "gen_ai.operation.name": "chat", + "gen_ai.request.model": "o3-mini", + "gen_ai.provider.name": "openai", + "gen_ai.input.messages": json.dumps(messages), + }) + + result = client.chat.completions.create(model="o3-mini", messages=messages) + + span.set_attributes({ + "gen_ai.response.model": result.model, + "gen_ai.output.messages": json.dumps([ + {"role": "assistant", "parts": [{"type": "text", "content": result.choices[0].message.content}]} + ]), + "gen_ai.response.finish_reasons": json.dumps([result.choices[0].finish_reason]), + "gen_ai.usage.input_tokens": result.usage.prompt_tokens, + "gen_ai.usage.output_tokens": result.usage.completion_tokens, + }) +``` + #### Thinking / reasoning messages Models with extended thinking (such as Anthropic's `thinking` blocks, Gemini's `thought`, or DeepSeek's `reasoning_content`) produce internal reasoning that isn't part of the user-visible reply. Represent this content as a `reasoning` part inside the assistant message, alongside the user-facing `text` part. Sentry surfaces reasoning parts separately and filters them out of the user-facing Conversations view, so don't fold thinking into a `text` part. -```python +```python {tabTitle:Transaction Mode (Default)} import json import sentry_sdk @@ -111,6 +149,45 @@ with sentry_sdk.start_span(op="gen_ai.chat", name="chat o3-mini") as span: span.set_data("gen_ai.usage.output_tokens.reasoning", result.usage.completion_tokens_details.reasoning_tokens) ``` +```python {tabTitle:Stream Mode} +import json +import sentry_sdk + +messages = [{"role": "user", "parts": [{"type": "text", "content": "What is 6 * 7?"}]}] + +with sentry_sdk.traces.start_span( + name="chat o3-mini", + attributes={"sentry.op": "gen_ai.chat"}, +) as span: + span.set_attributes({ + "gen_ai.operation.name": "chat", + "gen_ai.request.model": "o3-mini", + "gen_ai.provider.name": "openai", + "gen_ai.input.messages": json.dumps(messages), + }) + + result = client.chat.completions.create(model="o3-mini", messages=messages) + + span.set_attributes({ + "gen_ai.response.model": result.model, + "gen_ai.output.messages": json.dumps([ + { + "role": "assistant", + "parts": [ + {"type": "reasoning", "content": "6 times 7 is 42."}, + {"type": "text", "content": "The answer is 42."}, + ], + } + ]), + "gen_ai.usage.output_tokens": result.usage.completion_tokens, + }) + # If the provider reports reasoning tokens, record them as a subset of output tokens + span.set_attribute( + "gen_ai.usage.output_tokens.reasoning", + result.usage.completion_tokens_details.reasoning_tokens, + ) +``` + When previous thinking is fed back into a multi-turn request, include the same `reasoning` parts in the assistant messages within `gen_ai.input.messages`. ### Invoke Agent Span @@ -129,7 +206,7 @@ For a complete guide on naming agents across all supported frameworks, see [Nami #### Example of an Invoke Agent Span: -```python +```python {tabTitle:Transaction Mode (Default)} import json import sentry_sdk @@ -147,6 +224,31 @@ with sentry_sdk.start_span(op="gen_ai.invoke_agent", name="invoke_agent Weather span.set_data("gen_ai.usage.output_tokens", final_output.usage.output_tokens) ``` +```python {tabTitle:Stream Mode} +import json +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="invoke_agent Weather Agent", + attributes={"sentry.op": "gen_ai.invoke_agent"}, +) as span: + span.set_attributes({ + "gen_ai.operation.name": "invoke_agent", + "gen_ai.request.model": "o3-mini", + "gen_ai.agent.name": "Weather Agent", + }) + + final_output = my_agent.run() + + span.set_attributes({ + "gen_ai.output.messages": json.dumps([ + {"role": "assistant", "parts": [{"type": "text", "content": str(final_output)}]} + ]), + "gen_ai.usage.input_tokens": final_output.usage.input_tokens, + "gen_ai.usage.output_tokens": final_output.usage.output_tokens, + }) +``` + ### Execute Tool Span This span represents the execution of a tool or function that was requested by an AI model, including the input arguments and resulting output. @@ -157,7 +259,7 @@ This span represents the execution of a tool or function that was requested by a #### Example Execute Tool span -```python +```python {tabTitle:Transaction Mode (Default)} import json import sentry_sdk @@ -171,6 +273,25 @@ with sentry_sdk.start_span(op="gen_ai.execute_tool", name="execute_tool get_weat span.set_data("gen_ai.tool.call.result", json.dumps(result)) ``` +```python {tabTitle:Stream Mode} +import json +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="execute_tool get_weather", + attributes={"sentry.op": "gen_ai.execute_tool"}, +) as span: + span.set_attributes({ + "gen_ai.operation.name": "execute_tool", + "gen_ai.tool.name": "get_weather", + "gen_ai.tool.call.arguments": json.dumps({"location": "Paris"}), + }) + + result = get_weather(location="Paris") + + span.set_attribute("gen_ai.tool.call.result", json.dumps(result)) +``` + ### Handoff Span This span marks the transition of control from one agent to another, typically when the current agent determines another agent is better suited to handle the task. @@ -181,7 +302,7 @@ This span marks the transition of control from one agent to another, typically w #### Example of a Handoff Span -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk with sentry_sdk.start_span(op="gen_ai.handoff", name="handoff from Weather Agent to Travel Agent"): @@ -192,10 +313,28 @@ with sentry_sdk.start_span(op="gen_ai.invoke_agent", name="invoke_agent Travel A pass ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +with sentry_sdk.traces.start_span( + name="handoff from Weather Agent to Travel Agent", + attributes={"sentry.op": "gen_ai.handoff"}, +): + pass # Handoff span just marks the transition + +with sentry_sdk.traces.start_span( + name="invoke_agent Travel Agent", + attributes={"sentry.op": "gen_ai.invoke_agent"}, +): + # Run the target agent here + pass +``` + ## Tracking Conversations - Tracking Conversations has **beta** stability. Configuration options and behavior may change. + Tracking Conversations has **beta** stability. Configuration options and + behavior may change. For AI applications that involve multi-turn conversations, you can use `sentry_sdk.ai.set_conversation_id()` to associate all AI spans from the same conversation. This enables you to track and analyze complete conversation flows within Sentry. diff --git a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/caches-module.mdx b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/caches-module.mdx index 2ae6ef48f87b8..05a707a3b1327 100644 --- a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/caches-module.mdx +++ b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/caches-module.mdx @@ -3,15 +3,22 @@ title: Instrument Caches sidebar_order: 1000 description: "Learn how to manually instrument your code to use Sentry's Caches module. " --- + A cache can be used to speed up data retrieval, thereby improving application performance. Because instead of getting data from a potentially slow data layer, your application will be getting data from memory (in a best case scenario). Caching can speed up read-heavy workloads for applications like Q&A portals, gaming, media sharing, and social networking. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + Sentry offers a [cache-monitoring dashboard](https://sentry.io/orgredirect/organizations/:orgslug/dashboards/) that can be auto-instrumented for popular Python caching setups (like Django and Redis). If you're using a custom caching solution that doesn't have auto instrumentation, you can manually instrument it and use Sentry to get a look into how your caching solution is performing by following the setup instructions below. To make it possible for Sentry to give you an overview of your cache performance, you'll need to create two spans - one indicating that something is being put into the cache, and a second one indicating that something is being fetched from the cache. -Make sure that there's a transaction running when you create the spans. If you're using a web framework those transactions will be created for you automatically. See Tracing for more information. +In transaction mode, make sure that there's a transaction running when you create the spans. If you're using a web framework those transactions will be created for you automatically. See Tracing for more information. For detailed information about which data can be set, see the [Cache Module Developer Specification](https://develop.sentry.dev/sdk/performance/modules/caches/). @@ -21,16 +28,16 @@ If you're using anything other than Dja ### Add Span When Putting Data Into the Cache -If the cache you’re using isn’t supported by auto instrumentation mentioned above, you can use the custom instrumentation instructions below to emit cache spans: +If the cache you're using isn't supported by auto instrumentation mentioned above, you can use the custom instrumentation instructions below to emit cache spans: 1. Set the cache value with whatever cache library you happen to be using. -2. Wrap the part of your application that uses the cached value with `with sentry_sdk.start_span(...)` +2. Wrap the part of your application that uses the cached value with `with sentry_sdk.start_span(...)` (transaction mode) or `with sentry_sdk.traces.start_span(...)` (stream mode). 3. Set `op` to `cache.put`. 4. Set `cache.item_size` to an integer representing the size of the cached item. (The steps described above are documented in the snippet.) -```python +```python {tabTitle:Transaction Mode (Default)} import my_caching import sentry_sdk @@ -52,20 +59,43 @@ with sentry_sdk.start_span(op="cache.put") as span: span.set_data("cache.item_size", len(value)) # Warning: if value is very big this could use lots of memory ``` +```python {tabTitle:Stream Mode} +import my_caching +import sentry_sdk + +key = "myCacheKey123" +value = "The value I want to cache." + +with sentry_sdk.traces.start_span(attributes={"sentry.op": "cache.put"}) as span: + # Set a key in your caching using your custom caching solution + my_caching.set(key, value) + + # Describe the cache server you are accessing + span.set_attributes({ + "network.peer.address": "cache.example.com/supercache", + "network.peer.port": 9000, + + # Add the key(s) you want to set + "cache.key": [key], + + # Add the size of the value you stored in the cache + "cache.item_size": len(value), # Warning: if value is very big this could use lots of memory + }) +``` ### Add Span When Retrieving Data From the Cache -If the cache you’re using isn’t supported by auto instrumentation mentioned above, you can use the custom instrumentation instructions below to emit cache spans: +If the cache you're using isn't supported by auto instrumentation mentioned above, you can use the custom instrumentation instructions below to emit cache spans: 1. Fetch the cached value from whatever cache library you happen to be using. -2. Wrap the part of your application that uses the cached value with `with sentry_sdk.start_span(...)` +2. Wrap the part of your application that uses the cached value with `with sentry_sdk.start_span(...)` (transaction mode) or `with sentry_sdk.traces.start_span(...)` (stream mode). 3. Set `op` to `cache.get`. 4. Set `cache.hit` to a boolean value representing whether the value was successfully fetched from the cache or not. 5. Set `cache.item_size` to an integer representing the size of the cached item. (The steps described above are documented in the snippet.) -```python +```python {tabTitle:Transaction Mode (Default)} import my_caching import sentry_sdk @@ -94,4 +124,35 @@ with sentry_sdk.start_span(op="cache.get") as span: span.set_data("cache.hit", False) ``` +```python {tabTitle:Stream Mode} +import my_caching +import sentry_sdk + +key = "myCacheKey123" +value = None + +with sentry_sdk.traces.start_span(attributes={"sentry.op": "cache.get"}) as span: + # Get a key from your caching solution + value = my_caching.get(key) + + # Describe the cache server you are accessing + span.set_attributes({ + "network.peer.address": "cache.example.com/supercache", + "network.peer.port": 9000, + + # Add the key(s) you just retrieved from the cache + "cache.key": [key], + }) + + if value is not None: + # If you retrieved a value, the cache was hit + span.set_attribute("cache.hit", True) + + # Optionally also add the size of the value you retrieved + span.set_attribute("cache.item_size", len(value)) + else: + # If you could not retrieve a value, it was a miss + span.set_attribute("cache.hit", False) +``` + You should now have the right spans in place. Head over to the [Cache dashboard](https://sentry.io/orgredirect/organizations/:orgslug/dashboards/) to see how your cache is performing. diff --git a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/mcp-module.mdx b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/mcp-module.mdx index 74b8af8cac941..5c951a785c2cb 100644 --- a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/mcp-module.mdx +++ b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/mcp-module.mdx @@ -8,6 +8,12 @@ With Sentry's [MCP monitoring](/ai/monitoring/mcp/), you can track and debug MCP As a prerequisite to setting up MCP monitoring with Python, you'll need to first set up tracing. Once this is done, the Python SDK will automatically instrument MCP servers created with supported libraries. If that doesn't fit your use case, you can use custom instrumentation described below. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + ## Automatic Instrumentation The Python SDK supports automatic instrumentation for MCP servers. We recommend adding the MCP integration to your Sentry configuration to automatically capture spans for MCP operations. @@ -18,7 +24,8 @@ The Python SDK supports automatic instrumentation for MCP servers. We recommend For your MCP data to show up in Sentry, spans must be created with well-defined names and data attributes. See below for the different types of MCP operations you can instrument. -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create these spans. +In transaction mode, you can also use the [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator to create these spans, using its `template` parameter. +In stream mode, these spans need to be created directly with `sentry_sdk.traces.start_span()`, as shown below. ## Spans @@ -28,7 +35,7 @@ The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instr #### Example Tool Execution Span: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk import json @@ -72,13 +79,59 @@ with sentry_sdk.start_span( raise ``` +```python {tabTitle:Stream Mode} +import sentry_sdk +import json + +sentry_sdk.init(...) + +# Example tool execution +tool_name = "get_weather" +tool_arguments = {"city": "San Francisco"} + +with sentry_sdk.traces.start_span( + name=f"tools/call {tool_name}", + attributes={"sentry.op": "mcp.server"}, +) as span: + # Set MCP-specific attributes + span.set_attributes({ + "mcp.tool.name": tool_name, + "mcp.method.name": "tools/call", + + # Set request metadata + "mcp.request.id": "req_123abc", + "mcp.session.id": "session_xyz789", + "mcp.transport": "stdio", # or "http", "sse" for HTTP/WebSocket/SSE + "network.transport": "pipe", # or "tcp" for HTTP/SSE + }) + + # Set tool arguments (optional, if send_default_pii=True) + for key, value in tool_arguments.items(): + span.set_attribute(f"mcp.request.argument.{key}", value) + + # Execute the tool + try: + result = execute_tool(tool_name, tool_arguments) + + # Set result data + span.set_attribute("mcp.tool.result.content", json.dumps(result)) + span.set_attribute("mcp.tool.result.is_error", False) + + # Set result content count if applicable + if isinstance(result, (list, dict)): + span.set_attribute("mcp.tool.result.content_count", len(result)) + except Exception as e: + span.set_attribute("mcp.tool.result.is_error", True) + raise +``` + ### Prompt Retrieval Span #### Example Prompt Retrieval Span: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk import json @@ -123,13 +176,60 @@ with sentry_sdk.start_span( ) ``` +```python {tabTitle:Stream Mode} +import sentry_sdk +import json + +sentry_sdk.init(...) + +# Example prompt retrieval +prompt_name = "code_review" +prompt_arguments = {"language": "python"} + +with sentry_sdk.traces.start_span( + name=f"prompts/get {prompt_name}", + attributes={"sentry.op": "mcp.server"}, +) as span: + # Set MCP-specific attributes + span.set_attributes({ + "mcp.prompt.name": prompt_name, + "mcp.method.name": "prompts/get", + + # Set request metadata + "mcp.request.id": "req_456def", + "mcp.session.id": "session_xyz789", + "mcp.transport": "http", + "network.transport": "tcp", + }) + + # Set prompt arguments (optional, if send_default_pii=True) + for key, value in prompt_arguments.items(): + span.set_attribute(f"mcp.request.argument.{key}", value) + + # Retrieve the prompt + prompt_result = get_prompt(prompt_name, prompt_arguments) + + # Set result data + messages = prompt_result.get("messages", []) + span.set_attribute("mcp.prompt.result.message_count", len(messages)) + + # For single-message prompts, set role and content + if len(messages) == 1: + span.set_attribute("mcp.prompt.result.message_role", messages[0].get("role")) + # Content is PII, only set if send_default_pii=True + span.set_attribute( + "mcp.prompt.result.message_content", + json.dumps(messages[0].get("content")) + ) +``` + ### Resource Read Span #### Example Resource Read Span: -```python +```python {tabTitle:Transaction Mode (Default)} import sentry_sdk sentry_sdk.init(...) @@ -155,6 +255,34 @@ with sentry_sdk.start_span( resource_data = read_resource(resource_uri) ``` +```python {tabTitle:Stream Mode} +import sentry_sdk + +sentry_sdk.init(...) + +# Example resource access +resource_uri = "file:///path/to/resource.txt" + +with sentry_sdk.traces.start_span( + name=f"resources/read {resource_uri}", + attributes={"sentry.op": "mcp.server"}, +) as span: + # Set MCP-specific attributes + span.set_attributes({ + "mcp.resource.uri": resource_uri, + "mcp.method.name": "resources/read", + + # Set request metadata + "mcp.request.id": "req_789ghi", + "mcp.session.id": "session_xyz789", + "mcp.transport": "http", + "network.transport": "tcp", + }) + + # Access the resource + resource_data = read_resource(resource_uri) +``` + ## Common Span Attributes diff --git a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/queues-module.mdx b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/queues-module.mdx index 5741c398a766b..4f5212baa3ff6 100644 --- a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/queues-module.mdx +++ b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/queues-module.mdx @@ -3,26 +3,34 @@ title: Instrument Queues sidebar_order: 3000 description: "Learn how to manually instrument your code to use Sentry's Queues module. " --- + Sentry comes with a [queue-monitoring dashboard](https://sentry.io/orgredirect/organizations/:orgslug/dashboards/) that can be auto-instrumented for popular Python queue setups (like Celery). In case yours isn't supported, you can still instrument custom spans and transactions around your queue producers and consumers to ensure that you have performance data about your messaging queues. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + ## Producer Instrumentation -To start capturing performance metrics, use the `sentry_sdk.start_span()` function to wrap your queue producer events. Your span `op` must be set to `queue.publish`. Include the following span data to enrich your producer spans with queue metrics: +To start capturing performance metrics, use the `start_span()` function to wrap your queue producer events. Your span `op` must be set to `queue.publish`. Include the following span data to enrich your producer spans with queue metrics: -| Data Attribute | Type | Description | -|:--|:--|:--| -| `messaging.message.id ` | string | The message identifier | -| `messaging.destination.name` | string | The queue or topic name | -| `messaging.message.body.size` | int | Size of the message body in bytes | +| Data Attribute | Type | Description | +| :---------------------------- | :----- | :-------------------------------- | +| `messaging.message.id ` | string | The message identifier | +| `messaging.destination.name` | string | The queue or topic name | +| `messaging.message.body.size` | int | Size of the message body in bytes | -Your `queue.publish` span must exist inside a transaction in order to be recognized as a producer span. If you are using a supported web framework, the transaction is created by the integration. If you use plain Python, you can start a new one using `sentry_sdk.start_transaction()`. +In transaction mode, your `queue.publish` span must exist inside a transaction in order to be recognized as a producer span. If you are using a supported web framework, the transaction is created by the integration. If you use plain Python, you can start a new one using `sentry_sdk.start_transaction()`. -You must also include trace headers (`sentry-trace` and `baggage`) in your message so that your consumers can continue your trace once your message is picked up. +In stream mode, there's no separate transaction to depend on. If your `queue.process` span has no parent, it's automatically promoted to a service span. If you'd rather group it under an explicit service span, start one with `sentry_sdk.traces.start_span(parent_span=None)` first. +You must also include trace headers (`sentry-trace` and `baggage`) in your message so that your consumers can continue your trace once your message is picked up. -```python +```python {tabTitle:Transaction Mode (Default)} from datetime import datetime, timezone import sentry_sdk @@ -68,25 +76,74 @@ with sentry_sdk.start_transaction( ) ``` +```python {tabTitle:Stream Mode} +from datetime import datetime, timezone + +import sentry_sdk +import my_custom_queue + +# Initialize Sentry +sentry_sdk.init(...) + +connection = my_custom_queue.connect() + +# The message you want to send to the queue +queue = "messages" +message = "Hello World!" +message_id = "abc123" + +# Create the service span +# If you are using a web framework, the framework integration +# will create this for you and you can omit this. +with sentry_sdk.traces.start_span( + name="queue_producer_transaction", + attributes={"sentry.op": "function"}, + parent_span=None, +): + # Create the span + with sentry_sdk.traces.start_span( + name="queue_producer", + attributes={"sentry.op": "queue.publish"}, + ) as span: + # Set span data + span.set_attributes({ + "messaging.message.id": message_id, + "messaging.destination.name": queue, + "messaging.message.body.size": len(message.encode("utf-8")), + }) + + # Publish the message to the queue (including trace information and current time stamp) + now = int(datetime.now(timezone.utc).timestamp()) + connection.publish( + queue=queue, + body=message, + timestamp=now, + headers={ + "sentry-trace": sentry_sdk.get_traceparent(), + "baggage": sentry_sdk.get_baggage(), + }, + ) +``` ## Consumer Instrumentation -To start capturing performance metrics, use the `sentry_sdk.start_span()` function to wrap your queue consumers. Your span `op` must be set to `queue.process`. Include the following span data to enrich your consumer spans with queue metrics: +To start capturing performance metrics, use the `start_span()` function to wrap your queue consumers. Your span `op` must be set to `queue.process`. Include the following span data to enrich your consumer spans with queue metrics: -| Data Attribute | Type | Description | -|:--|:--|:--| -| `messaging.message.id ` | string | The message identifier | -| `messaging.destination.name` | string | The queue or topic name | -| `messaging.message.body.size` | number | Size of the message body in bytes | -| `messaging.message.retry.count ` | number | The number of times a message was attempted to be processed | +| Data Attribute | Type | Description | +| :----------------------------------- | :----- | :------------------------------------------------------------------ | +| `messaging.message.id ` | string | The message identifier | +| `messaging.destination.name` | string | The queue or topic name | +| `messaging.message.body.size` | number | Size of the message body in bytes | +| `messaging.message.retry.count ` | number | The number of times a message was attempted to be processed | | `messaging.message.receive.latency ` | number | The time in milliseconds that a message awaited processing in queue | -Your `queue.process` span must exist inside a transaction in order to be recognized as a consumer span. If you are using a supported web framework, the transaction is created by the integration. If you use plain Python, you can start a new one using `sentry_sdk.start_transaction()`. +In transaction mode, your `queue.process` span must exist inside a transaction in order to be recognized as a consumer span. If you are using a supported web framework, the transaction is created by the integration. If you use plain Python, you can start a new one using `sentry_sdk.start_transaction()`. -Use `sentry_sdk.continue_trace()` to connect your consumer spans to their associated producer spans, and `transaction.set_status()` to mark the trace of your message as success or failed. +In stream mode, there's no separate transaction to depend on. If your `queue.process` span has no parent, it's automatically promoted to a service span. If you'd rather group it under an explicit service span, start one with `sentry_sdk.traces.start_span(parent_span=None)` first. +Use `continue_trace()` to connect your consumer spans to their associated producer spans. To mark the trace of your message as success or failed, use `transaction.set_status()` in transaction mode, or set `.status = "ok"/"error"` directly on the service span in stream mode. -```python +```python {tabTitle:Transaction Mode (Default)} from datetime import datetime, timezone import sentry_sdk @@ -134,3 +191,54 @@ with sentry_sdk.start_transaction(transaction): # In case of an error set the status to "internal_error" transaction.set_status("internal_error") ``` + +```python {tabTitle:Stream Mode} +from datetime import datetime, timezone + +import sentry_sdk +import my_custom_queue + +# Initialize Sentry +sentry_sdk.init(...) + +connection = my_custom_queue.connect() + +# Pick up message from queues +queue = "messages" +message = connection.consume(queue=queue) + +# Calculate latency (optional, but valuable) +now = datetime.now(timezone.utc) +message_time = datetime.fromtimestamp(message["timestamp"], timezone.utc) +latency = now - message_time + +# Continue the trace started in the producer +# If you are using a web framework, the framework integration +# will create this for you and you can omit this. +sentry_sdk.traces.continue_trace(message["headers"]) +with sentry_sdk.traces.start_span( + name="queue_consumer_transaction", + attributes={"sentry.op": "function"}, + parent_span=None, +) as service_span: + # Create the span + with sentry_sdk.traces.start_span( + name="queue_consumer", + attributes={"sentry.op": "queue.process"}, + ) as span: + # Set span data + span.set_attributes({ + "messaging.message.id": message["message_id"], + "messaging.destination.name": queue, + "messaging.message.body.size": message["body"], + "messaging.message.receive.latency": latency, + "messaging.message.retry.count": 0, + }) + + try: + # Process the message + process_message(message) + except Exception: + # In case of an error set the status to "error" + service_span.status = "error" +``` diff --git a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/requests-module.mdx b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/requests-module.mdx index d25a4355f9f80..89ec0a86567bc 100644 --- a/docs/platforms/python/tracing/instrumentation/custom-instrumentation/requests-module.mdx +++ b/docs/platforms/python/tracing/instrumentation/custom-instrumentation/requests-module.mdx @@ -3,8 +3,15 @@ title: Instrument HTTP Requests sidebar_order: 2000 description: "Learn how to manually instrument your code to use Sentry's Requests module." --- + As a prerequisite to setting up [Requests](/product/dashboards/sentry-dashboards/outbound-api-requests/), you’ll need to first set up tracing. Once this is done, the Python SDK will automatically instrument outgoing HTTP requests made via `HTTPConnection` and show the data in the [requests-monitoring dashboard](https://sentry.io/orgredirect/organizations/:orgslug/dashboards/). If that doesn't fit your use case, you can set up using custom instrumentation described below. + + +This page covers both transaction mode (default) and stream mode. See New Spans to learn more. + + + ## Custom Instrumentation For detailed information about which data can be set, see the [Requests Module developer specifications](https://develop.sentry.dev/sdk/performance/modules/requests/). @@ -15,9 +22,10 @@ NOTE: Refer to [HTTP Span Data Conventions](https://develop.sentry.dev/sdk/perfo Here is an example of an instrumented function that makes HTTP requests: -```python +```python {tabTitle:Transaction Mode (Default)} from urllib.parse import urlparse import requests +import sentry_sdk def make_request(method, url): span = sentry_sdk.start_span( @@ -42,3 +50,29 @@ def make_request(method, url): return response ``` + +```python {tabTitle:Stream Mode} +from urllib.parse import urlparse +import requests +import sentry_sdk + +def make_request(method, url): + span = sentry_sdk.traces.start_span( + name="%s %s" % (method, url), + attributes={"sentry.op": "http.client"}, + ) + parsed_url = urlparse(url) + span.set_attributes({ + "http.request.method": method, + "url": url, + "server.address": parsed_url.hostname, + "server.port": parsed_url.port, + }) + response = requests.request(method=method, url=url) + span.set_attributes({ + "http.response.status_code": response.status_code, + "http.response_content_length": response.headers["content-length"], + }) + span.finish() + return response +``` diff --git a/includes/tracing/ai-agents-module/ai-client-span.mdx b/includes/tracing/ai-agents-module/ai-client-span.mdx index f9f47eb95c305..36d0af9d815df 100644 --- a/includes/tracing/ai-agents-module/ai-client-span.mdx +++ b/includes/tracing/ai-agents-module/ai-client-span.mdx @@ -1,8 +1,10 @@ -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span. + +In transaction mode, the [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span using its `template` parameter. The stream-mode decorator doesn't have a `template` option. + -- The span `op` MUST be `"gen_ai.{gen_ai.operation.name}"`. (e.g. `"gen_ai.chat"`) +- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.{gen_ai.operation.name}"`. (e.g. `"gen_ai.chat"`) - The span `name` SHOULD be `"{gen_ai.operation.name} {gen_ai.request.model}"`. (e.g. `"chat o3-mini"`) - The `gen_ai.request.model` attribute MUST be the requested model. (e.g. `"o3-mini"`) - The `gen_ai.response.model` attribute MUST be the concrete model that responded. (e.g. `"gpt-4o-2024-08-06"`) @@ -12,9 +14,9 @@ The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instr ### Request Attributes -| Data Attribute | Type | Requirement Level | Description | Example | -| :--------------------------------- | :----- | :---------------- | :------------------------------------------------------------------------------------------------------------ | :-------------------------------------------------------------------- | -| `gen_ai.input.messages` | string | optional | List of message objects sent to the LLM. **[0]**, **[1]** | `'[{"role": "user", "parts": [{"type": "text", "content": "..."}]}]'` | +| Data Attribute | Type | Requirement Level | Description | Example | +| :--------------------------------- | :----- | :---------------- | :------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------- | +| `gen_ai.input.messages` | string | optional | List of message objects sent to the LLM. **[0]**, **[1]** | `'[{"role": "user", "parts": [{"type": "text", "content": "..."}]}]'` | | `gen_ai.tool.definitions` | string | optional | List of objects describing the available tools. **[0]** | `'[{"name": "random_number", "description": "..."}]'` | | `gen_ai.system_instructions` | string | optional | The system instructions passed to the model. | `"You are a helpful assistant."` | | `gen_ai.request.frequency_penalty` | float | optional | Model configuration parameter. | `0.5` | @@ -29,17 +31,17 @@ The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instr ### Response Attributes -| Data Attribute | Type | Requirement Level | Description | Example | -| :-------------------------------- | :------ | :---------------- | :------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------- | -| `gen_ai.response.model` | string | required | The concrete model that responded (may differ from `gen_ai.request.model`). | `"gpt-4o-2024-08-06"` | -| `gen_ai.output.messages` | string | optional | Stringified array of message objects representing the model's output. **[0]**, **[1]** | `'[{"role": "assistant", "parts": [{"type": "text", "content": "..."}]}]'` | -| `gen_ai.response.finish_reasons` | string | optional | Stringified array of reasons the model stopped generating. **[0]** | `'["stop"]'` | -| `gen_ai.response.id` | string | optional | Unique identifier for the completion. | `"chatcmpl-abc123"` | -| `gen_ai.response.streaming` | boolean | optional | Whether the response was streamed. | `true` | -| `gen_ai.response.time_to_first_token` | double | optional | Seconds until first response chunk in streaming. | `0.5` | -| `gen_ai.response.tokens_per_second` | double | optional | Output tokens per second throughput. | `50.0` | -| `gen_ai.response.text` | string | optional | **Deprecated.** Use `gen_ai.output.messages` instead. The text representation of the model's responses. | `"The weather in Paris is rainy"` | -| `gen_ai.response.tool_calls` | string | optional | **Deprecated.** Use `gen_ai.output.messages` instead. The tool calls in the model's response. **[0]** | `'[{"name": "random_number", "type": "function_call", "arguments": "..."}]'` | +| Data Attribute | Type | Requirement Level | Description | Example | +| :------------------------------------ | :------ | :---------------- | :------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------- | +| `gen_ai.response.model` | string | required | The concrete model that responded (may differ from `gen_ai.request.model`). | `"gpt-4o-2024-08-06"` | +| `gen_ai.output.messages` | string | optional | Stringified array of message objects representing the model's output. **[0]**, **[1]** | `'[{"role": "assistant", "parts": [{"type": "text", "content": "..."}]}]'` | +| `gen_ai.response.finish_reasons` | string | optional | Stringified array of reasons the model stopped generating. **[0]** | `'["stop"]'` | +| `gen_ai.response.id` | string | optional | Unique identifier for the completion. | `"chatcmpl-abc123"` | +| `gen_ai.response.streaming` | boolean | optional | Whether the response was streamed. | `true` | +| `gen_ai.response.time_to_first_token` | double | optional | Seconds until first response chunk in streaming. | `0.5` | +| `gen_ai.response.tokens_per_second` | double | optional | Output tokens per second throughput. | `50.0` | +| `gen_ai.response.text` | string | optional | **Deprecated.** Use `gen_ai.output.messages` instead. The text representation of the model's responses. | `"The weather in Paris is rainy"` | +| `gen_ai.response.tool_calls` | string | optional | **Deprecated.** Use `gen_ai.output.messages` instead. The tool calls in the model's response. **[0]** | `'[{"name": "random_number", "type": "function_call", "arguments": "..."}]'` | ### Token Usage diff --git a/includes/tracing/ai-agents-module/execute-tool-span.mdx b/includes/tracing/ai-agents-module/execute-tool-span.mdx index d39f2ebc372cb..a91cc02ca4270 100644 --- a/includes/tracing/ai-agents-module/execute-tool-span.mdx +++ b/includes/tracing/ai-agents-module/execute-tool-span.mdx @@ -1,10 +1,12 @@ Describes a tool execution. -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span. + +In transaction mode, the [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span using its `template` parameter. The stream-mode decorator doesn't have a `template` option. + -- The span `op` MUST be `"gen_ai.execute_tool"`. +- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.execute_tool"`. - The span `name` SHOULD be `"execute_tool {gen_ai.tool.name}"`. (e.g. `"execute_tool query_database"`) - The `gen_ai.tool.name` attribute SHOULD be set to the name of the tool. (e.g. `"query_database"`) - All [Common Span Attributes](#common-span-attributes) SHOULD be set (all `required` common attributes MUST be set). diff --git a/includes/tracing/ai-agents-module/handoff-span.mdx b/includes/tracing/ai-agents-module/handoff-span.mdx index b39be4999e137..2cc275a842dba 100644 --- a/includes/tracing/ai-agents-module/handoff-span.mdx +++ b/includes/tracing/ai-agents-module/handoff-span.mdx @@ -1,5 +1,5 @@ A span that describes the handoff from one agent to another. -- The spans `op` MUST be `"gen_ai.handoff"`. +- The spans `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.handoff"`. - The spans `name` SHOULD be `"handoff from {from_agent} to {to_agent}"`. - All [Common Span Attributes](#common-span-attributes) SHOULD be set. diff --git a/includes/tracing/ai-agents-module/invoke-agent-span.mdx b/includes/tracing/ai-agents-module/invoke-agent-span.mdx index 4e0729bf96988..c7e97a6714e4c 100644 --- a/includes/tracing/ai-agents-module/invoke-agent-span.mdx +++ b/includes/tracing/ai-agents-module/invoke-agent-span.mdx @@ -1,10 +1,12 @@ Describes AI agent invocation. -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span. + +In transaction mode, the [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span using its `template` parameter. The stream-mode decorator doesn't have a `template` option. + -- The span `op` MUST be `"gen_ai.invoke_agent"`. +- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"gen_ai.invoke_agent"`. - The span `name` SHOULD be `"invoke_agent {gen_ai.agent.name}"`. - The `gen_ai.operation.name` attribute MUST be `"invoke_agent"`. - The `gen_ai.agent.name` attribute SHOULD be set to the agent's name. (e.g. `"Weather Agent"`) @@ -20,7 +22,7 @@ Additional attributes on the span: | `gen_ai.input.messages` | string | optional | List of message objects given to the agent. **[0]**, **[1]** | `'[{"role": "user", "parts": [{"type": "text", "content": "..."}]}]'` | | `gen_ai.tool.definitions` | string | optional | List of objects describing the available tools. **[0]** | `'[{"name": "random_number", "description": "..."}]'` | | `gen_ai.system_instructions` | string | optional | The system instructions passed to the model. | `"You are a helpful assistant."` | -| `gen_ai.pipeline.name` | string | optional | The name of the AI workflow or pipeline the agent belongs to. | `"weather-pipeline"` | +| `gen_ai.pipeline.name` | string | optional | The name of the AI workflow or pipeline the agent belongs to. | `"weather-pipeline"` | | `gen_ai.request.messages` | string | optional | **Deprecated.** Use `gen_ai.input.messages` instead. List of message objects given to the agent. **[0]** | `'[{"role": "system", "content": "..."}]'` | | `gen_ai.request.available_tools` | string | optional | **Deprecated.** Use `gen_ai.tool.definitions` instead. List of objects describing the available tools. **[0]** | `'[{"name": "random_number", "description": "..."}]'` | diff --git a/includes/tracing/mcp-module/prompt-retrieval-span.mdx b/includes/tracing/mcp-module/prompt-retrieval-span.mdx index 5c68cec65f697..faccf4775c574 100644 --- a/includes/tracing/mcp-module/prompt-retrieval-span.mdx +++ b/includes/tracing/mcp-module/prompt-retrieval-span.mdx @@ -1,8 +1,6 @@ Describes MCP prompt retrieval. -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span. - -- The span's `op` MUST be `"mcp.server"`. +- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"mcp.server"`. - The span `name` SHOULD be `"prompts/get {mcp.prompt.name}"`. - The `mcp.prompt.name` attribute MUST be set to the prompt's name. (e.g. `"code_review"`) - The `mcp.method.name` attribute SHOULD be set to `"prompts/get"`. diff --git a/includes/tracing/mcp-module/resource-read-span.mdx b/includes/tracing/mcp-module/resource-read-span.mdx index 83d25e859345a..f670977d0635d 100644 --- a/includes/tracing/mcp-module/resource-read-span.mdx +++ b/includes/tracing/mcp-module/resource-read-span.mdx @@ -1,8 +1,6 @@ Describes MCP resource access. -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span. - -- The span's `op` MUST be `"mcp.server"`. +- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"mcp.server"`. - The span `name` SHOULD be `"resources/read {mcp.resource.uri}"`. - The `mcp.resource.uri` attribute MUST be set to the resource's URI. (e.g. `"file:///path/to/resource"`) - The `mcp.method.name` attribute SHOULD be set to `"resources/read"`. diff --git a/includes/tracing/mcp-module/tool-execution-span.mdx b/includes/tracing/mcp-module/tool-execution-span.mdx index 6af1a75028215..de435384e6a3c 100644 --- a/includes/tracing/mcp-module/tool-execution-span.mdx +++ b/includes/tracing/mcp-module/tool-execution-span.mdx @@ -1,8 +1,6 @@ Describes MCP tool execution. -The [@sentry_sdk.trace()](/platforms/python/tracing/instrumentation/custom-instrumentation/#span-templates) decorator can also be used to create this span. - -- The span's `op` MUST be `"mcp.server"`. +- The span `op` (transaction mode) or the span's `sentry.op` attribute (stream mode) MUST be `"mcp.server"`. - The span `name` SHOULD be `"tools/call {mcp.tool.name}"`. - The `mcp.tool.name` attribute MUST be set to the tool's name. (e.g. `"get_weather"`) - The `mcp.method.name` attribute SHOULD be set to `"tools/call"`.