diff --git a/docs/impulse/docs/config/configuration.md b/docs/impulse/docs/config/configuration.md index 531fd8b2..d8199953 100644 --- a/docs/impulse/docs/config/configuration.md +++ b/docs/impulse/docs/config/configuration.md @@ -170,6 +170,36 @@ Top-level fields on `SolverConfig`: the `project_id` column (after column-name mapping) of every table it reads that carries one — `container_tags` (if configured), `container_metrics`, and `channel_mapping` (if configured). Omit it if you don't need project-level scoping; the solver does not require it. +- `channel_time_unit` (`"s"` | `"ms"` | `"us"` | `"ns"`, optional) and `channel_time_origin` + (`"epoch"` (default) | `"container_start"`): the time frame of the channel timestamps + (`tstart`/`tend`, or `timestamp` when `data_type = "RAW"`). `"epoch"` means absolute epoch + numbers; `"container_start"` means time relative to the container's start + (`container_metrics.start_ts`, e.g. seconds since the recording started). Channel timestamps are never converted; they must already be + numbers in that frame. + +- `container_time_unit` (`"s"` | `"ms"` | `"us"` | `"ns"`, optional): the unit of **numeric** + `container_metrics.start_ts`/`stop_ts`, when it differs from the channels' unit. For example, + boundaries in epoch ms and channel samples in epoch µs need `container_time_unit = "ms"` and + `channel_time_unit = "us"`. Requires `channel_time_unit`; not allowed for `TIMESTAMP` + boundaries, which carry their own unit. Unset means the numeric boundaries are already in the + channels' unit. + + These settings are only used by `TimeWindowEvent`, whose windows must lie in the channel time + frame. It derives its window bounds from `container_metrics.start_ts`/`stop_ts`: + - origin `"epoch"`: `TIMESTAMP` boundaries as epoch numbers in `channel_time_unit`; numeric + boundaries converted from `container_time_unit` to `channel_time_unit` (as they are when + `container_time_unit` is unset); + - origin `"container_start"`: `0` to `stop_ts - start_ts`, converted the same way. + + `channel_time_unit` is required when the boundaries are `TIMESTAMP` columns; a report with a + `TimeWindowEvent` fails with a clear error until it is set. `TIMESTAMP_NTZ` and `DATE` boundaries + are not supported. Everything else sees the original `container_metrics.start_ts`/`stop_ts`: + `ContainerEvent`, `measurement_dimension`, container filters, and UDFs that request them via + `apply(..., container_metrics=[...])` (a `TIMESTAMP` arrives there as a `pd.Timestamp`). + + The channel time frame is part of the definition hash of every `TimeWindowEvent` and of the + aggregations scoped to it, so changing it recomputes them over all containers in incremental + mode instead of mixing time frames in the gold tables. Per-table sections (each a `TableConfig`): @@ -192,7 +222,7 @@ Internal column names that mappings can target: | `tstart`, `tend`| Sample interval start/end on the `channels` table (RLE) | | `timestamp` | Raw sample timestamp on the `channels` table (RAW mode; encoded into `tstart`/`tend`) | | `is_plausible` | Boolean plausibility flag on the `channels` table (RAW mode); consumed by `drop_implausible_data` | -| `start_ts`, `stop_ts` | Measurement start/stop epoch timestamps on the `container_metrics` table — referenced by `ContainerEvent` to derive event-fact start/end | +| `start_ts`, `stop_ts` | Measurement start/stop epoch timestamps on the `container_metrics` table — referenced by `ContainerEvent` and `TimeWindowEvent` to derive event-fact start/end. May be `TIMESTAMP` (see `channel_time_unit`) | | `value` | Sample value (or attribute value on the EAV tag table) | | `key` | Attribute key on the EAV `container_tags` table | | `priority` | Tie-breaker column on the `channel_mapping` table | diff --git a/docs/impulse/docs/data_model/silver_layer_schema.md b/docs/impulse/docs/data_model/silver_layer_schema.md index d5ce7508..36491670 100644 --- a/docs/impulse/docs/data_model/silver_layer_schema.md +++ b/docs/impulse/docs/data_model/silver_layer_schema.md @@ -180,6 +180,14 @@ for human-readable display, `start_ts`/`stop_ts` for the gold the epoch-typed pair). Populate whichever your queries and `measurement_dimensions` config need. +`start_ts`/`stop_ts` may also be `TIMESTAMP` columns. To use them with a `TimeWindowEvent`, set +[`solver_config.channel_time_unit`](../config/configuration.md#solver-column-mappings-and-filters) +to the unit of the channel sample timestamps (`tstart`/`tend`, or `timestamp` in the raw format), +plus `channel_time_origin="container_start"` if those are relative to the container start, so +the window boundaries share the samples' time base. Numeric `start_ts`/`stop_ts` may use a different +epoch unit than the channel samples (e.g. ms boundaries, µs samples); then also set +`solver_config.container_time_unit`. The columns themselves are not converted. + ::: --- diff --git a/docs/impulse/docs/references/api/impulse_query_engine/analyze/query/solvers/solver_config.md b/docs/impulse/docs/references/api/impulse_query_engine/analyze/query/solvers/solver_config.md index e34c236e..adb1aa08 100644 --- a/docs/impulse/docs/references/api/impulse_query_engine/analyze/query/solvers/solver_config.md +++ b/docs/impulse/docs/references/api/impulse_query_engine/analyze/query/solvers/solver_config.md @@ -126,6 +126,21 @@ so that solver code can always reference the same constants. override for the channel mapping (alias) table. - `channels` (`TableConfig`): Column mappings and filters for the channel data table. - `unit_conversion` (`TableConfig`): Column mappings and filters for the unit conversion table. +- `channel_time_unit` (`{"s", "ms", "us", "ns"} or None`): Time unit of the timestamps in the ``channels`` table (``tstart`` / ``tend``, or +``timestamp`` for RAW data). Only used to compute ``TimeWindowEvent`` windows in +that unit (see ``solvers.utils.window_bounds.with_window_bounds``); required when +``container_metrics`` ``start_ts`` / ``stop_ts`` are ``TIMESTAMP`` columns. Nothing +else is converted: channel timestamps, and the ``start_ts`` / ``stop_ts`` seen by +UDFs, ``ContainerEvent`` and ``measurement_dimension``, keep their original values. +- `channel_time_origin` (`{"epoch", "container_start"}`): Origin of the channel timestamps: absolute epoch (default), or relative to the +container's ``start_ts``. Like :attr:`channel_time_unit`, only used for the +``TimeWindowEvent`` windows. +- `container_time_unit` (`{"s", "ms", "us", "ns"} or None`): Unit of **numeric** ``container_metrics`` ``start_ts`` / ``stop_ts``, when it differs +from :attr:`channel_time_unit` (e.g. boundaries in epoch ms, channels in µs). Only +used to convert them into :attr:`channel_time_unit` for the ``TimeWindowEvent`` +windows; requires :attr:`channel_time_unit`. Unset means the numeric boundaries are +already in the channels' unit. Not allowed for ``TIMESTAMP`` boundaries, which carry +their own unit. #### from\_json @@ -215,6 +230,30 @@ def start_ts_col() -> str Internal column name for the measurement-start epoch timestamp on container_metrics. +#### window\_start\_col + +```python +def window_start_col() -> str +``` + +Internal column name for the container start in the channel time frame. + +Added by ``solvers.utils.window_bounds.with_window_bounds``; prefixed so it cannot +clash with a customer column. + + +#### window\_stop\_col + +```python +def window_stop_col() -> str +``` + +Internal column name for the container stop in the channel time frame. + +Added by ``solvers.utils.window_bounds.with_window_bounds``; prefixed so it cannot +clash with a customer column. + + #### stop\_ts\_col ```python @@ -457,3 +496,25 @@ def col_map() -> dict[str, str] Short-key → internal-column-name mapping for UDFs and caches. +#### reject\_implausible\_channels\_filter\_in\_raw + +```python +def reject_implausible_channels_filter_in_raw(is_raw: bool) -> None +``` + +Raise if an is_plausible channels filter is set in RAW mode. + +Such a filter runs before raw encoding and bridges intervals across dropped +samples instead of splitting them; use drop_implausible_data instead. No-op +when not raw. + + +#### validate\_container\_time\_unit\_requires\_channel\_time\_unit + +```python +def validate_container_time_unit_requires_channel_time_unit() +``` + +``container_time_unit`` converts into ``channel_time_unit``, so it needs one. + + diff --git a/docs/impulse/docs/references/api/impulse_reporting/events/container_event.md b/docs/impulse/docs/references/api/impulse_reporting/events/container_event.md index 463c6bbb..af2911e8 100644 --- a/docs/impulse/docs/references/api/impulse_reporting/events/container_event.md +++ b/docs/impulse/docs/references/api/impulse_reporting/events/container_event.md @@ -9,7 +9,7 @@ ContainerEvent — an event spanning the full measurement container. ## ContainerEvent ```python -class ContainerEvent(Event) +class ContainerEvent(ContainerBoundaryEvent) ``` Event that treats the full measurement container as a single event instance. @@ -33,18 +33,6 @@ Initialise a ContainerEvent. - `desc` (`str`): Human-readable description. - `attributes` (`dict`): Key-value metadata for the event. -#### get\_id - -```python -def get_id() -> int -``` - -Return a unique identifier derived from the event name. - -**Returns**: - -`int`: Positive 32-bit integer identifier. - #### get\_expression ```python @@ -86,30 +74,6 @@ so the name of the event is hashed. `int`: Hash value representing the computation definition. -#### as\_dict - -```python -def as_dict() -> dict -``` - -Return a dictionary representation of the event. - -**Returns**: - -`dict`: - -#### as\_spark\_row - -```python -def as_spark_row() -> Row -``` - -Return a Spark ``Row`` representation. - -**Returns**: - -`Row`: - #### determine\_events ```python @@ -142,21 +106,3 @@ produces one event instance per container. `DataFrame`: Spark DataFrame matching ``EVENT_INSTANCE_FACT_SCHEMA``. -#### determine\_metadata\_df - -```python -def determine_metadata_df(cls, spark: SparkSession, - events: list[ContainerEvent]) -> DataFrame -``` - -Create a Spark DataFrame containing event metadata. - -**Arguments**: - -- `spark` (`SparkSession`): Active Spark session. -- `events` (`list of ContainerEvent`): List of ContainerEvent objects. - -**Returns**: - -`DataFrame`: Spark DataFrame matching ``EVENT_DIMENSION_SCHEMA``. - diff --git a/docs/impulse/docs/references/api/impulse_reporting/events/time_window_event.md b/docs/impulse/docs/references/api/impulse_reporting/events/time_window_event.md new file mode 100644 index 00000000..07da0708 --- /dev/null +++ b/docs/impulse/docs/references/api/impulse_reporting/events/time_window_event.md @@ -0,0 +1,160 @@ +--- +sidebar_label: time_window_event +title: impulse_reporting.events.time_window_event +--- + +TimeWindowEvent — splits each container into consecutive fixed-duration windows. + + +## TimeWindowEvent + +```python +class TimeWindowEvent(ContainerBoundaryEvent) +``` + +Event that divides each measurement container into consecutive fixed windows. + +Unlike ``ContainerEvent`` (one instance per container), a ``TimeWindowEvent`` emits one +event instance per fixed-duration slice, tiling the container's ``start_ts`` / ``stop_ts`` +span with windows of length ``window_length``. The final slice is clamped to the +container end. + +The event fact is computed from ``container_metrics`` alone (via + + +#### \_\_init\_\_ + +```python +def __init__(name: str, + window_length: float, + desc: str = None, + required_channels: list[str] = None, + attributes: Mapping[str, str] = None, + max_windows_per_container: int = MAX_WINDOWS_PER_CONTAINER) +``` + +Initialize a TimeWindowEvent object. + +**Arguments**: + +- `name` (`str`): Name of the event. +- `window_length` (`float`): Fixed window length, in the same time unit as the underlying timestamps +(e.g. milliseconds-since-epoch). Must be strictly positive and finite. +- `desc` (`str`): Description of the event. +- `required_channels` (`list of str`): List of required channels for the event. Informational; stored in the event +dimension table. +- `attributes` (`Mapping[str, str]`): Key-value metadata for the event. ``window_length`` is surfaced here +automatically (without overriding a user-supplied key). +- `max_windows_per_container` (`int`): Maximum number of windows per container (default 1,000,000). A container +exceeding it fails the report with an error naming the limit, which usually +means ``window_length`` is in the wrong unit for the boundaries. Not part of +the definition hash. + +**Raises**: + +- `ValueError`: If ``window_length`` is not strictly positive and finite, or +``max_windows_per_container`` is not a positive integer. + +#### set\_channel\_time + +```python +def set_channel_time(unit: str | None, + origin: str = "epoch", + container_unit: str | None = None) -> None +``` + +Record the channel time frame the windows are computed in. + +Set by ``Report.add_event`` from the report's ``solver_config``. Stored on the +expression, whose string form feeds the definition hashes of this event and of the +aggregations scoped to it. + +**Arguments**: + +- `unit` (`str or None`): The report's ``solver_config.channel_time_unit``. +- `origin` (`str`): The report's ``solver_config.channel_time_origin`` (default ``"epoch"``). +- `container_unit` (`str or None`): The report's ``solver_config.container_time_unit``. + +#### get\_expression + +```python +def get_expression() -> TimeSeriesExpression | None +``` + +Get the time series expression associated with the event. + +**Returns**: + +`TimeSeriesExpression or None`: The time-window expression for the event. + +#### get\_event\_type\_str + +```python +def get_event_type_str() -> str +``` + +Get the event type string for TimeWindowEvent. + +**Returns**: + +`str`: Event type string. + +#### determine\_definition\_hash + +```python +def determine_definition_hash() -> int +``` + +Calculate definition hash for the time-window event. + +Only includes the expression string, which encodes the attributes that affect the +event results: ``window_length`` and the channel time frame (``channel_time_unit``, +``channel_time_origin``, ``container_time_unit``; omitted while unset / default). +Resizing the window or changing the time frame therefore forces a full recompute in +incremental mode. + +Excludes: name, description, required_channels, max_windows_per_container, +report_id + +**Returns**: + +`int`: Hash value representing the computation definition. + +#### determine\_events + +```python +def determine_events( + cls, + spark: SparkSession, + events: list[TimeWindowEvent], + *, + solved_df: DataFrame = None, + query: QueryBuilder = None, + solver: QuerySolver = None, + pre_filtered_containers_df: DataFrame = None) -> DataFrame +``` + +Extract the event fact table for the given list of TimeWindowEvent objects. + +Resolves the matching containers via the solver's filter pipeline (like +``ContainerEvent``) and computes each event's windows natively from the +containers' ``start_ts`` / ``stop_ts`` in the channel time frame +(``solvers.utils.window_bounds.with_window_bounds``), so every filtered container +gets windows. +Each window becomes one event instance (``start_ts < end_ts``) whose +``event_instance_id`` hashes its boundaries. The solve uses the same window function +for scoped aggregations (see :func:`window_intervals_udf`), so the ids match. + +**Arguments**: + +- `spark` (`SparkSession`): Spark session for data processing. +- `events` (`list of TimeWindowEvent`): List of TimeWindowEvent objects to process. +- `solved_df` (`DataFrame`): Not used by TimeWindowEvent (kept for interface compatibility). +- `query` (`QueryBuilder`): Query builder with filters applied. +- `solver` (`QuerySolver`): Solver whose filter pipeline is used for container resolution. +- `pre_filtered_containers_df` (`DataFrame`): Pre-filtered containers for incremental processing. + +**Returns**: + +`DataFrame`: Spark DataFrame containing event instance facts. + diff --git a/docs/impulse/docs/references/api/sidebar.json b/docs/impulse/docs/references/api/sidebar.json index f919f467..3f08df0f 100644 --- a/docs/impulse/docs/references/api/sidebar.json +++ b/docs/impulse/docs/references/api/sidebar.json @@ -106,7 +106,8 @@ "items": [ "references/api/impulse_reporting/events/basic_event", "references/api/impulse_reporting/events/container_event", - "references/api/impulse_reporting/events/sequence_of_events" + "references/api/impulse_reporting/events/sequence_of_events", + "references/api/impulse_reporting/events/time_window_event" ], "label": "impulse_reporting.events", "type": "category" diff --git a/docs/impulse/docs/references/report/event.md b/docs/impulse/docs/references/report/event.md index 46ac02e2..fa4671be 100644 --- a/docs/impulse/docs/references/report/event.md +++ b/docs/impulse/docs/references/report/event.md @@ -195,6 +195,99 @@ The expression **must** evaluate to a `PointsInTime`; otherwise construction rai --- +## TimeWindowEvent + +A `TimeWindowEvent` divides each matching container into **consecutive fixed-duration windows** -- +one event instance per slice. Unlike `ContainerEvent` (one instance for the whole container), it +produces repeated windows (e.g. one-minute, ten-minute, hourly, or daily segments) across every +matching container. No signal expression is needed: the window boundaries are derived from the +container's `start_ts` / `stop_ts` on the `container_metrics` table. + +```python +from impulse_reporting.events.time_window_event import TimeWindowEvent + +ten_minute_windows = TimeWindowEvent( + name="ten_minute_windows", + window_length=600_000, # in the same time unit as the underlying timestamps (see note) + desc="Ten-minute segments across each measurement", +) +my_report.add_event(ten_minute_windows) +``` + +### Parameters + +| Parameter | Type | Required | Description | +|---------------------|---------------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `name` | `str` | Yes | Unique event name. | +| `window_length` | `float` | Yes | Fixed window length, **in the same time unit as the underlying timestamps** (e.g. milliseconds-since-epoch). Must be strictly positive and finite; validated at construction. | +| `desc` | `str` | No | Human-readable description. | +| `required_channels` | `list[str]` | No | Channel names required for this event. Informational; stored in the event dimension table. | +| `attributes` | `Mapping[str, str]` | No | Free-form key-value metadata. `window_length` is surfaced here automatically (without overriding a user key). | +| `max_windows_per_container` | `int` | No | Upper bound on the windows per container (default `1_000_000`). A container that would exceed it fails the report with an error naming the limit, which usually means `window_length` is in the wrong unit for the timestamps. Raise it for very long containers with short windows. Not part of the definition hash. | + +:::note +`window_length` follows the same convention as `SequenceOfEvents.max_overlap`: it is expressed in +the same time unit as the channel timestamps (microseconds-since-epoch in the sample data), not +seconds or any derived unit. So 60 one-minute windows over millisecond timestamps use +`window_length=60_000`. + +The windows are computed in the time frame of the channel timestamps, set by +[`solver_config.channel_time_unit`, `channel_time_origin` and `container_time_unit`](../../config/configuration.md#solver-column-mappings-and-filters): + +- If `container_metrics.start_ts`/`stop_ts` are `TIMESTAMP` columns, set `channel_time_unit` to the + unit of the channel timestamps (`tstart`/`tend`, or `timestamp` for RAW data; e.g. `"s"`), and + `window_length` is expressed in it. Without it, the report fails with an error naming the + setting. +- If the channel timestamps are relative to the container start (e.g. seconds since the + recording started), also set `channel_time_origin="container_start"`. The windows then run from + `0` to the container's duration (`stop_ts - start_ts` of `container_metrics`). +- If numeric `container_metrics.start_ts`/`stop_ts` are in another unit than the channel + timestamps (e.g. epoch ms boundaries, µs samples), set `container_time_unit` to their unit + (e.g. `"ms"`) and `channel_time_unit` to the channels' (e.g. `"us"`). Otherwise no window + overlaps the samples. + +Only the windows use these settings: `ContainerEvent`, `measurement_dimension` and UDFs that read +`container_metrics.start_ts`/`stop_ts` keep seeing the original values. The channel time frame is part of the +event's definition (and of the aggregations scoped to it), so changing it recomputes them over all +containers in incremental mode. +::: + +### How it works + +1. The event resolves the matching containers through the report's container filters (like + `ContainerEvent`), reads `start_ts` and `stop_ts` from the `container_metrics` table, and + tiles that span, in the channel time frame, into consecutive windows of length + `window_length`. The window instances in `event_instance_fact` are in that frame too. +2. The **final window is clamped** to the container's `stop_ts` when the last full window would + overrun it; any zero-length trailing slice is dropped (every window instance in + `event_instance_fact` satisfies `start_ts < end_ts`). Containers whose + `container_metrics.start_ts` or `stop_ts` is null, NaN or infinite get no windows. +3. Each window becomes one **event instance**, written to the shared `event_instance_fact` table. + Its `event_instance_id` hashes the container, the event name and the window's start and end, + like for other interval events. +4. An aggregation scoped to the event (`StatsAggregator(..., event=time_window_event)`) computes + its statistic **once per window** and joins back to those instances. + +:::note +The windows are computed from `container_metrics` alone, so **every** container that matches the +report's filters gets windows, whether or not it has channel data and whether or not an +aggregation is scoped to the event. An aggregation scoped to the event uses the same window +function in the query engine, so its per-window rows carry the same `event_instance_id` values. +For the per-window values to be meaningful, the container boundaries (`container_metrics.start_ts` +/ `stop_ts`) must share the channel samples' time base, or be converted into it with the settings +above. +::: + +:::note +Window boundaries are stored in `event_instance_fact` as doubles (its `start_ts` / `end_ts` +columns), like every other event type. Epoch timestamps in nanoseconds exceed the range doubles +represent exactly, so their window boundaries are rounded to about 256 ns. The event and its +aggregations use the same window function, so the rounding is the same on both sides and their +`event_instance_id` values still match. If `window_length` is not a whole number in the channel +unit (e.g. `0.2` or `1.7` over timestamps in seconds), or not a multiple of 256 ns for nanosecond +epochs, rounding can add a final window only a few ulps long. +::: + ## Event output schema ### event_dimension @@ -205,7 +298,7 @@ Stores event definitions (one row per event per report). |---------------------|---------------------|-----------------------------------------------------------------------------| | `event_id` | `int` | Unique event identifier (CRC32 hash of name + expression). | | `report_id` | `int` | Report identifier. | -| `event_type` | `str` | `"BASIC_EVENT"`, `"CONTAINER_EVENT"`, `"SEQUENCE_OF_EVENTS"`, or `"POINTS_IN_TIME_EVENT"`. | +| `event_type` | `str` | `"BASIC_EVENT"`, `"CONTAINER_EVENT"`, `"SEQUENCE_OF_EVENTS"`, `"POINTS_IN_TIME_EVENT"`, or `"TIME_WINDOW_EVENT"`. | | `event_name` | `str` | Event name. | | `event_description` | `str` | Event description. | | `required_channels` | `array[str]` | Required channel names (null for `ContainerEvent`). | @@ -232,9 +325,9 @@ Interval events satisfy `start_ts < end_ts`; `PointsInTimeEvent` instances are z ## Choosing between event types -| Criterion | BasicEvent | ContainerEvent | SequenceOfEvents | PointsInTimeEvent | -|----------------------------------|---------------------------------------------------------|---------------------------------------------------|---------------------------------------------------------------------------|------------------------------------------------------------| -| Requires a TSAL expression | Yes (one) | No | Yes (ordered list) | Yes (one, must evaluate to `PointsInTime`) | -| Multiple instances per container | Yes (one per matching interval) | No (always one per container) | Yes (one per joined sequence) | Yes (one per instant) | -| Instance duration | Interval (`start_ts < end_ts`) | Full container window | Interval (`start_ts < end_ts`) | Zero (`start_ts == end_ts`) | -| Use case | Signal-based conditions, operating bands, distance bins | Full-run aggregations, container-level statistics | State transitions and multi-step patterns where consecutive states overlap | Edge/instant events, e.g. `rising_edges()` / `falling_edges()` | +| Criterion | BasicEvent | ContainerEvent | SequenceOfEvents | PointsInTimeEvent | TimeWindowEvent | +|----------------------------------|---------------------------------------------------------|---------------------------------------------------|---------------------------------------------------------------------------|------------------------------------------------------------|-------------------------------------------------------------| +| Requires a TSAL expression | Yes (one) | No | Yes (ordered list) | Yes (one, must evaluate to `PointsInTime`) | No (needs a `window_length`) | +| Multiple instances per container | Yes (one per matching interval) | No (always one per container) | Yes (one per joined sequence) | Yes (one per instant) | Yes (one per fixed window) | +| Instance duration | Interval (`start_ts < end_ts`) | Full container window | Interval (`start_ts < end_ts`) | Zero (`start_ts == end_ts`) | Fixed window (last clamped to container end) | +| Use case | Signal-based conditions, operating bands, distance bins | Full-run aggregations, container-level statistics | State transitions and multi-step patterns where consecutive states overlap | Edge/instant events, e.g. `rising_edges()` / `falling_edges()` | Repeated time segments (1-min / 10-min / hourly / daily) | diff --git a/docs/impulse/pydoc-markdown.yml b/docs/impulse/pydoc-markdown.yml index aa28ace6..f52dda2b 100644 --- a/docs/impulse/pydoc-markdown.yml +++ b/docs/impulse/pydoc-markdown.yml @@ -7,6 +7,7 @@ loaders: - impulse_reporting.events.basic_event - impulse_reporting.events.container_event - impulse_reporting.events.sequence_of_events + - impulse_reporting.events.time_window_event - impulse_reporting.aggregations.histogram - impulse_reporting.aggregations.histogram2d - impulse_reporting.aggregations.stats_aggregator diff --git a/skills/README.md b/skills/README.md index cb98eccf..bcae0a27 100644 --- a/skills/README.md +++ b/skills/README.md @@ -12,7 +12,7 @@ Each skill is a folder with a `SKILL.md` file that documents usage patterns. Sta | [`impulse-tsal`](./impulse-tsal/SKILL.md) | The Time Series Analytics Language DSL — selecting channels, deriving virtual signals, and the four result types (`SampleSeries`, `Intervals`, `PointsInTime`, `PointsInTimeSeries`). | | [`impulse-data-model`](./impulse-data-model/SKILL.md) | The silver-layer input tables Impulse reads, the gold-layer star schema it writes, landing your own data, and adapting an existing layout via column mappings. | | [`impulse-config`](./impulse-config/SKILL.md) | The `ImpulseConfig` schema — source tables, sink, container filters, solver options, incremental processing, and sinkless mode. | -| [`impulse-events`](./impulse-events/SKILL.md) | Defining event windows: `BasicEvent`, `ContainerEvent`, `SequenceOfEvents`, `PointsInTimeEvent`. | +| [`impulse-events`](./impulse-events/SKILL.md) | Defining event windows: `BasicEvent`, `ContainerEvent`, `SequenceOfEvents`, `PointsInTimeEvent`, `TimeWindowEvent`. | | [`impulse-aggregations`](./impulse-aggregations/SKILL.md)| Computing results over channels: 1D/2D histograms (duration/distance/custom-weight), `StatsAggregator`, `PointValueAggregator`, and pages. | | [`impulse-channels`](./impulse-channels/SKILL.md) | Calculated (derived) channels — materializing a new signal from existing channels via `CalculatedChannel` and `solve_calculated_channels`. | | [`impulse-reporting`](./impulse-reporting/SKILL.md) | The batch pipeline that persists events and aggregations to the gold-layer star schema with `Report` / `Page`, plus incremental runs. | diff --git a/skills/impulse-config/SKILL.md b/skills/impulse-config/SKILL.md index 6e2c9eed..81568e03 100644 --- a/skills/impulse-config/SKILL.md +++ b/skills/impulse-config/SKILL.md @@ -124,6 +124,18 @@ rejected (use `drop_implausible_data`), any other channels filter warns. Top-level `project_id` (str, optional) applies an equality filter on the `project_id` column of every table that has one (`container_tags`, `container_metrics`, `channel_mapping`). Omit if not needed. +Top-level `channel_time_unit` (`"s"` | `"ms"` | `"us"` | `"ns"`, optional) and `channel_time_origin` +(`"epoch"` default | `"container_start"`) describe the time frame of the `channels` timestamps +(`tstart`/`tend`, or `timestamp` with `data_type="RAW"`): absolute epoch numbers, or time relative +to the container's start (`container_metrics.start_ts`). Only `TimeWindowEvent` uses them, to compute its windows in that +frame from `container_metrics.start_ts`/`stop_ts` (origin `"container_start"`: from `0` to +`stop_ts - start_ts`). `channel_time_unit` is required when those are `TIMESTAMP` columns. If they +are numeric but in another unit than the channels (e.g. epoch ms boundaries, µs samples), also set +`container_time_unit` (e.g. `"ms"`; requires `channel_time_unit`, not allowed for `TIMESTAMP`). +Nothing is converted in place: the channel timestamps, and the `container_metrics.start_ts` / +`stop_ts` seen by `ContainerEvent`, `measurement_dimension` and UDFs (a `pd.Timestamp`), keep their +original values. + ```python "query_engine": { "solver": "DefaultSolver", @@ -133,7 +145,9 @@ table that has one (`container_tags`, `container_metrics`, `channel_mapping`). O "column_name_mapping": {"entity_id": "container_id"}, "filters": {"parent_id": "my_parent_id"} }, - "container_metrics": {"column_name_mapping": {"start_dt": "tstart", "stop_dt": "tend"}}, + "container_metrics": { + "column_name_mapping": {"measurement_start": "start_ts", "measurement_end": "stop_ts"} + }, "channel_mapping": {"filters": {"toolbox_id": "my_toolbox"}} } } diff --git a/skills/impulse-events/SKILL.md b/skills/impulse-events/SKILL.md index faa81740..542caddf 100644 --- a/skills/impulse-events/SKILL.md +++ b/skills/impulse-events/SKILL.md @@ -3,9 +3,10 @@ name: impulse-events description: > Define event windows in Impulse — the time spans that scope aggregations. Use when the user wants to "define an event", segment recordings into intervals (e.g. "engine RPM between 2000 and 5000"), - aggregate over the whole recording, capture state transitions/sequences, or mark instants like rising - edges. Covers BasicEvent, ContainerEvent, SequenceOfEvents, and PointsInTimeEvent — which TSAL result - type each requires, their constructor parameters, and the event fact/dimension output. + aggregate over the whole recording, split a recording into fixed time windows (one-minute, hourly, + daily segments), capture state transitions/sequences, or mark instants like rising edges. Covers + BasicEvent, ContainerEvent, SequenceOfEvents, PointsInTimeEvent, and TimeWindowEvent — which TSAL + result type each requires, their constructor parameters, and the event fact/dimension output. --- # Impulse — events @@ -29,6 +30,7 @@ Choose the type by what you need: | `ContainerEvent` | none | exactly one | full recording | | `SequenceOfEvents` | ordered list, each `Intervals` | one per joined sequence | interval (`start < end`) | | `PointsInTimeEvent` | one, must yield `PointsInTime` | one per instant | zero (`start == end`) | +| `TimeWindowEvent` | none (needs a `window_length`) | one per fixed window | fixed window (last clamped) | The TSAL result type is validated at construction — passing the wrong type raises `ValueError`. @@ -123,12 +125,54 @@ report.add_event(rpm_rising) Parameters: `name` (required), `expr` (required, must yield `PointsInTime`), `desc`, `required_channels`, `attributes`. +## TimeWindowEvent + +Splits each recording into consecutive fixed-duration windows — one instance per slice — from the +container's `start_ts`/`stop_ts` in `container_metrics` (no expression). The final window is clamped to +the recording end. Use it for repeated segments (one-minute, ten-minute, hourly, daily) that scope +aggregations per window. + +```python +from impulse_reporting.events.time_window_event import TimeWindowEvent + +ten_minute = TimeWindowEvent( + name="ten_minute_windows", + window_length=600_000, # same time unit as the timestamps (e.g. ms-since-epoch) + desc="Ten-minute segments", +) +report.add_event(ten_minute) +``` + +| Parameter | Type | Required | Description | +|---------------------|---------------------|----------|---------------------------------------------------------------------------------| +| `name` | `str` | Yes | Unique event name. | +| `window_length` | `float` | Yes | Fixed window length, **in the same time unit as the timestamps**. Must be finite and > 0. | +| `desc` | `str` | No | Description. | +| `required_channels` | `list[str]` | No | Informational. | +| `attributes` | `Mapping[str, str]` | No | Free-form metadata; `window_length` is added automatically. | +| `max_windows_per_container` | `int` | No | Windows-per-container limit (default 1,000,000); exceeding it fails the report, usually a `window_length` unit mismatch. | + +Windows are computed from `container_metrics` for every container matching the report's filters, with +or without channel data or a scoped aggregation. Pair it with an aggregation scoped to the event (e.g. +`StatsAggregator(..., event=...)`) to compute one statistic per window; those rows carry the same +`event_instance_id` values as the windows (both sides use the same window function). +Because the windows come from `container_metrics`, those boundaries must share the channel samples' +time base for the per-window values to be meaningful. If `container_metrics.start_ts`/`stop_ts` are +`TIMESTAMP` columns, set `query_engine.solver_config.channel_time_unit` to the unit of the channel +timestamps (`tstart`/`tend`, or `timestamp` for RAW), and `channel_time_origin="container_start"` if +they are relative to the container start (windows then run from `0`). Numeric `container_metrics` +boundaries in another unit than the channels (e.g. epoch ms vs. µs samples) need +`container_time_unit` as well. Only the windows use these settings; `container_metrics.start_ts` / +`stop_ts` keep their original values for `ContainerEvent` and UDFs. +Containers with null, NaN or infinite boundaries get no windows. + ## Output schema All event types share two gold tables. **event_dimension** (one row per event) — key columns: `event_id`, `report_id`, -`event_type` (`"BASIC_EVENT"`, `"CONTAINER_EVENT"`, `"SEQUENCE_OF_EVENTS"`, `"POINTS_IN_TIME_EVENT"`), +`event_type` (`"BASIC_EVENT"`, `"CONTAINER_EVENT"`, `"SEQUENCE_OF_EVENTS"`, `"POINTS_IN_TIME_EVENT"`, +`"TIME_WINDOW_EVENT"`), `event_name`, `event_description`, `required_channels`, `event_expression` (TSAL string, `"NA"` for `ContainerEvent`), `definition_hash`, `attributes`. diff --git a/src/impulse_query_engine/analyze/query/events/__init__.py b/src/impulse_query_engine/analyze/query/events/__init__.py index 50a1ce48..ad5c45a9 100644 --- a/src/impulse_query_engine/analyze/query/events/__init__.py +++ b/src/impulse_query_engine/analyze/query/events/__init__.py @@ -1 +1,4 @@ from .sequence_of_events_expression import SequenceOfEventsExpression +from .time_window_expression import TimeWindowExpression + +__all__ = ["SequenceOfEventsExpression", "TimeWindowExpression"] diff --git a/src/impulse_query_engine/analyze/query/events/time_window_expression.py b/src/impulse_query_engine/analyze/query/events/time_window_expression.py new file mode 100644 index 00000000..3c7072bd --- /dev/null +++ b/src/impulse_query_engine/analyze/query/events/time_window_expression.py @@ -0,0 +1,371 @@ +from __future__ import annotations + +import math +import numbers + +import numpy as np +import pandas as pd +import pyspark.sql.functions as F + +from impulse_query_engine.analyze.metadata.tag_expression import TagExpression +from impulse_query_engine.analyze.metadata.time_series_expression import ( + TimeSeriesExpression, + TimeSeriesSelector, +) +from impulse_query_engine.analyze.query.solvers.series_cache import SeriesCache +from impulse_query_engine.analyze.query.solvers.solver_config import SolverConfig +from impulse_query_engine.model.series.intervals import Intervals + +# Reuse SolverConfig's internal column names for the container bounds in the channel time +# frame (see solvers.utils.window_bounds.with_window_bounds) rather than re-declaring the +# literals here. These are the keys under which the solve exposes them via +# ``SeriesCache.container_metrics``. +# A default instance suffices since the names are config-invariant. +_SOLVER_CONFIG = SolverConfig() + +# Default upper bound on the windows per container. A window_length in the wrong unit for the +# boundaries (e.g. 60 meant as seconds over ns epochs) would otherwise yield billions of +# windows: numpy would allocate arrays of that size and the event fact explode as many rows. +MAX_WINDOWS_PER_CONTAINER = 1_000_000 + +_WINDOW_LIMIT_HINT = ( + "Check that window_length is in the unit of the channel timestamps " + "(solver_config.channel_time_unit) and, for numeric container boundaries in another " + "unit, that solver_config.container_time_unit is set, or raise the limit " + "(TimeWindowEvent max_windows_per_container, TimeWindowExpression max_windows)." +) + + +def validate_max_windows(max_windows: int, param_name: str = "max_windows") -> int: + """Return *max_windows* as an int, raising unless it is a positive integer. + + Parameters + ---------- + max_windows : int + Maximum number of windows per container. + param_name : str, optional + Name of the caller's parameter, used in the error message (default + ``"max_windows"``). + + Returns + ------- + int + The validated limit. + + Raises + ------ + ValueError + If *max_windows* is not a positive integer. + """ + if ( + isinstance(max_windows, bool) + or not isinstance(max_windows, numbers.Integral) + or max_windows <= 0 + ): + raise ValueError(f"{param_name} must be a positive integer, got {max_windows!r}.") + return int(max_windows) + + +def tile_windows( + start, stop, window_length: float, max_windows: int = MAX_WINDOWS_PER_CONTAINER +) -> tuple[np.ndarray, np.ndarray]: + """Tile ``[start, stop]`` into consecutive windows of length *window_length*. + + The one window implementation behind ``TimeWindowEvent``: the solve calls it through + :meth:`TimeWindowExpression.build` (scoped aggregations), the event fact through + :func:`window_intervals_udf`. ``event_instance_id`` hashes each window's boundaries, so + both sides must produce identical windows, which a single function guarantees as long as + both pass in the same values. Both read the same Spark-computed bounds + (``solvers.utils.window_bounds.with_window_bounds``), but pandas hands them over as + ``int64`` or ``float64`` (nulls force ``float64``), or as ``None`` / ``NaN``. The bounds are + therefore converted to ``float`` first: ``int64`` -> ``float64`` rounds to the nearest + double on either path, so the arithmetic below runs on identical doubles. + + Window ``i`` spans ``[start + i * W, min(start + (i + 1) * W, stop)]``, so the last one + is clamped to *stop*; windows with ``start_i >= end_i`` (possible only through rounding) + are dropped. + + Parameters + ---------- + start, stop : float, int, None + Container bounds in the channel time frame. + window_length : float + Fixed window length, in the same unit as the bounds. Strictly positive. + max_windows : int, optional + Maximum number of windows (default :data:`MAX_WINDOWS_PER_CONTAINER`). + + Returns + ------- + tuple of numpy.ndarray + ``(starts, ends)`` as float64 arrays; empty when a bound is null, NaN or infinite, + or the span is not strictly positive. + + Raises + ------ + ValueError + If the span would produce more than *max_windows* windows. + """ + empty = (np.empty(0), np.empty(0)) + if pd.isna(start) or pd.isna(stop): + return empty + start, stop = float(start), float(stop) + # NaN / infinite bounds (e.g. an unfinished recording) yield no windows, like nulls. + if not (math.isfinite(start) and math.isfinite(stop) and stop > start): + return empty + + # Compared before the int conversion, since an overflowing span gives an infinite count. + window_count = np.ceil((stop - start) / window_length) + if window_count > max_windows: + raise ValueError( + f"TimeWindowExpression: {window_count:.0f} windows of length {window_length} " + f"over a container span of {stop - start} exceed " + f"max_windows={max_windows}. {_WINDOW_LIMIT_HINT}" + ) + indices = np.arange(int(window_count)) + starts = start + indices * window_length + ends = np.minimum(start + (indices + 1) * window_length, stop) + keep = starts < ends + return starts[keep], ends[keep] + + +def window_intervals_udf(window_length: float, max_windows: int = MAX_WINDOWS_PER_CONTAINER): + """Scalar pandas UDF giving each container's windows via :func:`tile_windows`. + + Used by the reporting ``TimeWindowEvent`` for its event fact (one row per container), so + the event fact and the solve share one window implementation. + + Parameters + ---------- + window_length : float + Fixed window length, in the same unit as the bounds. Strictly positive. + max_windows : int, optional + Maximum number of windows per container (default + :data:`MAX_WINDOWS_PER_CONTAINER`). A container exceeding it fails the query with + an error naming the limit. + + Returns + ------- + callable + A pandas UDF ``(start, stop) -> struct, ends: array>`` + with the window starts and ends, in order; empty when a bound is null, NaN or + infinite, or the span is not strictly positive. + """ + window_length = float(window_length) + max_windows = validate_max_windows(max_windows) + + @F.pandas_udf("struct, ends: array>") + def windows(start: pd.Series, stop: pd.Series) -> pd.DataFrame: + pairs = [ + tile_windows(s, e, window_length, max_windows) + for s, e in zip(start, stop, strict=True) + ] + # object dtype keeps one array per row, also when all rows have equal window counts. + return pd.DataFrame( + { + "starts": pd.Series([p[0] for p in pairs], dtype=object), + "ends": pd.Series([p[1] for p in pairs], dtype=object), + } + ) + + return windows + + +class TimeWindowExpression(TimeSeriesExpression): + """Produce consecutive fixed-duration windows spanning a measurement container. + + The windows are derived purely from the container's ``start_ts`` / ``stop_ts`` metadata + (no channel data), so the expression declares no selectors and instead requests the + container bounds in the channel time frame via :meth:`required_container_metrics` + (computed by ``solvers.utils.window_bounds.with_window_bounds``). Windows tile those + bounds with a fixed length ``window_length`` (expressed in the same time unit as the + channel timestamps); the final window is clamped to the stop bound when the last full + window would overrun it. + + Visual timeline (window_length = W):: + + time ---> + container: | ------------------------------- | + windows: | --W-- | --W-- | --W-- | -rest- | + + This is the query-engine counterpart of the reporting ``TimeWindowEvent``. It evaluates + to :class:`Intervals`, so it can scope a ``StatsAggregator`` (one statistic per window). + The windows come from :func:`tile_windows`, which the reporting event fact also uses (via + :func:`window_intervals_udf`), so both produce the same windows in the same order. + + Attributes + ---------- + channel_time_unit : str or None + ``solver_config.channel_time_unit``, set by the reporting ``TimeWindowEvent``. + channel_time_origin : str + ``solver_config.channel_time_origin`` (default ``"epoch"``), set the same way. + container_time_unit : str or None + ``solver_config.container_time_unit``, set the same way. + + Both are descriptive only: :meth:`build` does not convert (the solver computes the + bounds). They are part of the string form, so the definition hashes of the event and of + every aggregation scoped to it change with the channel time frame. + """ + + def __init__(self, window_length: float, max_windows: int = MAX_WINDOWS_PER_CONTAINER): + """ + Initialize a TimeWindowExpression. + + Parameters + ---------- + window_length : float + Fixed window length, in the same time unit as the underlying timestamps + (e.g. milliseconds-since-epoch). Must be strictly positive and finite. + max_windows : int, optional + Maximum number of windows per container (default + :data:`MAX_WINDOWS_PER_CONTAINER`); :meth:`build` raises beyond it. Not part + of the string form, since it only decides between an error and a result. + + Raises + ------ + ValueError + If ``window_length`` is not strictly positive and finite, or ``max_windows`` + is not a positive integer. + """ + # inf / NaN must be rejected too: inf gives a zero window count and NaN an undefined + # one in tile_windows. + if window_length is None or not math.isfinite(window_length) or window_length <= 0: + raise ValueError( + f"TimeWindowExpression requires a strictly positive, finite window_length, " + f"got {window_length!r}." + ) + # Store as float so the string form (and thus the event definition hash) is stable + # regardless of whether an int or float was passed: 10 and 10.0 are the same window + # and must not trigger a spurious full recompute in incremental mode. + self.window_length = float(window_length) + self.max_windows = validate_max_windows(max_windows) + self.channel_time_unit: str | None = None + self.channel_time_origin: str = "epoch" + self.container_time_unit: str | None = None + TimeSeriesExpression.__init__(self, is_single_signal=False) + + def __str__(self) -> str: + """ + Return a string representation of the TimeWindowExpression. + + The ``window_length`` and the channel time frame are included so they flow into the + definition hashes of the event and of the aggregations scoped to it. Unset units + and the default ``"epoch"`` origin are omitted, keeping the default string + unchanged. + + Returns + ------- + str + String representation of the object. + """ + frame = "" + if self.channel_time_unit is not None: + frame += f", channel_time_unit={self.channel_time_unit}" + if self.channel_time_origin != "epoch": + frame += f", channel_time_origin={self.channel_time_origin}" + if self.container_time_unit is not None: + frame += f", container_time_unit={self.container_time_unit}" + return f"TimeWindowExpression" + + def dtype(self): + """ + Return the Spark data type of the result. + + Returns + ------- + pyspark.sql.types.ArrayType + Same dtype as Intervals: ArrayType(ArrayType(DoubleType())). + """ + return Intervals.empty().dtype() + + def get_required_tag_exprs(self) -> set[TagExpression]: + """ + Return required tag expressions (none: windows use container metrics only). + + Returns + ------- + set of TagExpression + """ + return set() + + def required_tags(self) -> set[str]: + """ + Return required tags (none). + + Returns + ------- + set of str + """ + return set() + + def required_container_tags(self) -> set[str]: + """ + Return required container tags (none). + + Returns + ------- + set of str + """ + return set() + + def required_container_metrics(self) -> set[str]: + """ + Return the container-metric columns needed to bound the windows. + + Returns + ------- + set of str + The container start/stop in the channel time frame, which the solver derives + from ``start_ts`` / ``stop_ts`` (``solvers.utils.window_bounds.with_window_bounds``). + """ + return {_SOLVER_CONFIG.window_start_col, _SOLVER_CONFIG.window_stop_col} + + def get_selectors(self) -> list[TimeSeriesSelector]: + """ + Return channel selectors (none: windows depend on no channel data). + + Returns + ------- + list of TimeSeriesSelector + """ + return [] + + def get_selector_expr(self): + """ + Return the combined selector expression (none). + + Returns + ------- + None + """ + return None + + def build(self, cache: SeriesCache) -> Intervals: + """ + Build the fixed-duration windows spanning the container. + + Parameters + ---------- + cache : SeriesCache + Cache exposing the requested container metrics via ``container_metrics``. + + Returns + ------- + Intervals + Consecutive fixed-length windows over the container bounds, with the final + window clamped to the stop bound. Empty when the bounds are absent (e.g. the + empty cache used for type validation), NaN or infinite, or non-positive in span. + + Raises + ------ + ValueError + If the container would produce more than ``max_windows`` windows. + """ + starts, ends = tile_windows( + cache.container_metrics.get(_SOLVER_CONFIG.window_start_col), + cache.container_metrics.get(_SOLVER_CONFIG.window_stop_col), + self.window_length, + self.max_windows, + ) + if len(starts) == 0: + return Intervals.empty() + return Intervals(starts, ends, del_last_empty=True) diff --git a/src/impulse_query_engine/analyze/query/solvers/default_solver.py b/src/impulse_query_engine/analyze/query/solvers/default_solver.py index d8992443..f9e6d7b0 100644 --- a/src/impulse_query_engine/analyze/query/solvers/default_solver.py +++ b/src/impulse_query_engine/analyze/query/solvers/default_solver.py @@ -24,6 +24,7 @@ from .solver_config import RawEncoder, SolverConfig from .utils.interval_encoder import IntervalEncoder from .utils.rle_encoder import RleEncoder +from .utils.window_bounds import with_window_bounds if TYPE_CHECKING: from impulse_query_engine.measurement_db import MeasurementDB @@ -1305,6 +1306,12 @@ def _build_container_metadata_df( if metric_cols: metrics = self.scoped_container_metrics(self.spark, query, pre_filtered_containers_df) + window_bounds = {self.config.window_start_col, self.config.window_stop_col} + if window_bounds & set(metric_cols): + # TimeWindowExpression reads the container bounds in the channel time + # frame, computed exactly like the TimeWindowEvent fact does. The raw + # start_ts/stop_ts stay unchanged for any other expression (e.g. UDFs). + metrics = with_window_bounds(metrics, self.config) missing = [c for c in metric_cols if c not in metrics.columns] if missing: raise ValueError( diff --git a/src/impulse_query_engine/analyze/query/solvers/solver_config.py b/src/impulse_query_engine/analyze/query/solvers/solver_config.py index 813f6c44..559289ba 100644 --- a/src/impulse_query_engine/analyze/query/solvers/solver_config.py +++ b/src/impulse_query_engine/analyze/query/solvers/solver_config.py @@ -16,8 +16,9 @@ import json from enum import StrEnum +from typing import Literal -from pydantic import BaseModel +from pydantic import BaseModel, model_validator class RawEncoder(StrEnum): @@ -133,9 +134,30 @@ class SolverConfig(BaseModel): Column mappings and filters for the channel data table. unit_conversion : TableConfig Column mappings and filters for the unit conversion table. + channel_time_unit : {"s", "ms", "us", "ns"} or None + Time unit of the timestamps in the ``channels`` table (``tstart`` / ``tend``, or + ``timestamp`` for RAW data). Only used to compute ``TimeWindowEvent`` windows in + that unit (see ``solvers.utils.window_bounds.with_window_bounds``); required when + ``container_metrics`` ``start_ts`` / ``stop_ts`` are ``TIMESTAMP`` columns. Nothing + else is converted: channel timestamps, and the ``start_ts`` / ``stop_ts`` seen by + UDFs, ``ContainerEvent`` and ``measurement_dimension``, keep their original values. + channel_time_origin : {"epoch", "container_start"} + Origin of the channel timestamps: absolute epoch (default), or relative to the + container's ``start_ts``. Like :attr:`channel_time_unit`, only used for the + ``TimeWindowEvent`` windows. + container_time_unit : {"s", "ms", "us", "ns"} or None + Unit of **numeric** ``container_metrics`` ``start_ts`` / ``stop_ts``, when it differs + from :attr:`channel_time_unit` (e.g. boundaries in epoch ms, channels in µs). Only + used to convert them into :attr:`channel_time_unit` for the ``TimeWindowEvent`` + windows; requires :attr:`channel_time_unit`. Unset means the numeric boundaries are + already in the channels' unit. Not allowed for ``TIMESTAMP`` boundaries, which carry + their own unit. """ project_id: str | None = None + channel_time_unit: Literal["s", "ms", "us", "ns"] | None = None + channel_time_origin: Literal["epoch", "container_start"] = "epoch" + container_time_unit: Literal["s", "ms", "us", "ns"] | None = None container_tags: TableConfig = TableConfig() container_metrics: TableConfig = TableConfig() @@ -222,6 +244,24 @@ def start_ts_col(self) -> str: """Internal column name for the measurement-start epoch timestamp on container_metrics.""" return "start_ts" + @property + def window_start_col(self) -> str: + """Internal column name for the container start in the channel time frame. + + Added by ``solvers.utils.window_bounds.with_window_bounds``; prefixed so it cannot + clash with a customer column. + """ + return "__window_start" + + @property + def window_stop_col(self) -> str: + """Internal column name for the container stop in the channel time frame. + + Added by ``solvers.utils.window_bounds.with_window_bounds``; prefixed so it cannot + clash with a customer column. + """ + return "__window_stop" + @property def stop_ts_col(self) -> str: """Internal column name for the measurement-stop epoch timestamp on container_metrics.""" @@ -417,3 +457,14 @@ def reject_implausible_channels_filter_in_raw(self, is_raw: bool) -> None: "samples. Use drop_implausible_data=True instead -- it drops " "implausible points inside the encoder with correct interval boundaries." ) + + @model_validator(mode="after") + def validate_container_time_unit_requires_channel_time_unit(self): + """``container_time_unit`` converts into ``channel_time_unit``, so it needs one.""" + if self.container_time_unit is not None and self.channel_time_unit is None: + raise ValueError( + "container_time_unit requires channel_time_unit: numeric container boundaries " + "are converted from container_time_unit into the unit of the channel " + "timestamps." + ) + return self diff --git a/src/impulse_query_engine/analyze/query/solvers/utils/window_bounds.py b/src/impulse_query_engine/analyze/query/solvers/utils/window_bounds.py new file mode 100644 index 00000000..068d2024 --- /dev/null +++ b/src/impulse_query_engine/analyze/query/solvers/utils/window_bounds.py @@ -0,0 +1,153 @@ +"""Container bounds in the channel time frame, for ``TimeWindowEvent`` windows. + +Both the ``TimeWindowEvent`` event fact and the solve (``TimeWindowExpression`` via the +container metadata) derive their window bounds here, from the raw ``container_metrics`` +``start_ts`` / ``stop_ts`` and the ``SolverConfig`` channel time settings +(``channel_time_unit``, ``channel_time_origin``, ``container_time_unit``). +""" + +import pyspark.sql.functions as F +import pyspark.sql.types as T +from pyspark.sql import Column, DataFrame + +from impulse_query_engine.analyze.query.solvers.solver_config import SolverConfig + +# Nanoseconds per time unit, for converting container boundaries into the channel unit. +_NANOS_PER_UNIT = {"s": 10**9, "ms": 10**6, "us": 10**3, "ns": 1} + + +def with_window_bounds(df: DataFrame, config: SolverConfig) -> DataFrame: + """Add the container start/stop in the channel time frame, for ``TimeWindowEvent``. + + A ``TimeWindowEvent`` tiles each container into windows that must be in the same time + frame as the channel timestamps (``config.channel_time_unit``, + ``config.channel_time_origin``). This adds ``config.window_start_col`` / + ``config.window_stop_col``, derived from the raw ``start_ts`` / ``stop_ts``, which stay + unchanged for UDFs, ``ContainerEvent`` and ``measurement_dimension``: + + - origin ``"epoch"``: ``TIMESTAMP`` boundaries as epoch numbers in + ``channel_time_unit``; numeric boundaries converted from ``container_time_unit`` to + ``channel_time_unit`` (as they are when unset); + - origin ``"container_start"``: ``0`` and ``stop_ts - start_ts``, converted the same way + (the difference is taken first, in the boundaries' own unit). + + ``TIMESTAMP`` values are converted via ``unix_micros``, which is exact and independent of + the session time zone; ``"s"`` / ``"ms"`` give doubles, ``"us"`` / ``"ns"`` longs. + Numeric boundaries converted to a finer unit are multiplied by an integer (exact, + keeping longs), to a coarser unit divided (doubles). The event fact and the solve both + call this, so their windows use the same bounds. The types are checked on the schema, + so a missing setting fails before any Spark job runs. + + Parameters + ---------- + df : pyspark.sql.DataFrame + Column-mapped ``container_metrics`` frame (or a projection of it) with ``start_ts`` + and ``stop_ts``. + config : SolverConfig + Solver configuration holding the channel time settings and column names. + + Returns + ------- + pyspark.sql.DataFrame + *df* with the two window-bound columns added. + + Raises + ------ + ValueError + If ``start_ts`` / ``stop_ts`` are missing, are ``TIMESTAMP_NTZ`` or ``DATE``, mix + ``TIMESTAMP`` and numeric types, or are ``TIMESTAMP`` while ``channel_time_unit`` is + unset or ``container_time_unit`` is set. + """ + types = {field.name: field.dataType for field in _boundary_fields(df, config)} + missing = [c for c in (config.start_ts_col, config.stop_ts_col) if c not in types] + if missing: + raise ValueError( + f"TimeWindowEvent needs the container_metrics columns {missing} to compute " + f"its windows. Available columns: {df.columns}. If they have other physical " + "names, map them via " + "query_engine.solver_config.container_metrics.column_name_mapping." + ) + for name, dtype in types.items(): + if isinstance(dtype, (T.TimestampNTZType, T.DateType)): + raise ValueError( + f"container_metrics column '{name}' has type {dtype.simpleString()}, " + "which cannot be converted to an epoch unambiguously (it carries no time " + "zone). Use a TIMESTAMP or epoch-number column." + ) + is_timestamp = {isinstance(dtype, T.TimestampType) for dtype in types.values()} + if len(is_timestamp) > 1: + raise ValueError( + f"container_metrics columns '{config.start_ts_col}' and '{config.stop_ts_col}' " + "must both be TIMESTAMP or both be numeric to compute TimeWindowEvent windows." + ) + timestamps = is_timestamp.pop() + if timestamps and config.channel_time_unit is None: + raise ValueError( + f"TimeWindowEvent needs its windows in the channel time frame, but " + f"container_metrics '{config.start_ts_col}' / '{config.stop_ts_col}' are " + "TIMESTAMP columns. Set query_engine.solver_config.channel_time_unit to the " + "unit of the channel timestamps (one of 's', 'ms', 'us', 'ns'), and " + "channel_time_origin to 'container_start' if they are relative to the " + "container start." + ) + if timestamps and config.container_time_unit is not None: + raise ValueError( + f"container_time_unit only applies to numeric container_metrics " + f"'{config.start_ts_col}' / '{config.stop_ts_col}', but they are TIMESTAMP " + "columns, which carry their own unit. Remove container_time_unit." + ) + + start, stop = F.col(config.start_ts_col), F.col(config.stop_ts_col) + unit = config.channel_time_unit + if config.channel_time_origin == "container_start": + window_start = F.lit(0) + # Subtract exactly in microseconds before scaling to the channel unit. + window_stop = ( + _micros_in_unit(F.unix_micros(stop) - F.unix_micros(start), unit) + if timestamps + else _container_to_channel_unit(stop - start, config) + ) + elif timestamps: + window_start = _micros_in_unit(F.unix_micros(start), unit) + window_stop = _micros_in_unit(F.unix_micros(stop), unit) + else: + window_start = _container_to_channel_unit(start, config) + window_stop = _container_to_channel_unit(stop, config) + return df.withColumn(config.window_start_col, window_start).withColumn( + config.window_stop_col, window_stop + ) + + +def _boundary_fields(df: DataFrame, config: SolverConfig) -> list[T.StructField]: + """Return the container start/stop timestamp fields present on *df*.""" + names = {config.start_ts_col, config.stop_ts_col} + return [field for field in df.schema.fields if field.name in names] + + +def _container_to_channel_unit(col: Column, config: SolverConfig) -> Column: + """Numeric boundary *col* converted from ``container_time_unit`` to ``channel_time_unit`` + (unchanged when unset or equal).""" + if ( + config.container_time_unit is None + or config.container_time_unit == config.channel_time_unit + ): + return col + source = _NANOS_PER_UNIT[config.container_time_unit] + target = _NANOS_PER_UNIT[config.channel_time_unit] + if source > target: + # Finer target unit: an integer factor keeps long boundaries exact. A long literal + # widens INT boundaries to long (int * int would stay int and overflow, e.g. epoch + # seconds * 1000); doubles and decimals keep their type. + return col * F.lit(source // target).cast(T.LongType()) + return col / F.lit(float(target // source)) + + +def _micros_in_unit(micros: Column, unit: str | None) -> Column: + """Microseconds converted to *unit*.""" + if unit == "s": + return micros / F.lit(1e6) + if unit == "ms": + return micros / F.lit(1e3) + if unit == "ns": + return micros * F.lit(1000) + return micros diff --git a/src/impulse_reporting/core/report.py b/src/impulse_reporting/core/report.py index 131d3731..82955e8a 100644 --- a/src/impulse_reporting/core/report.py +++ b/src/impulse_reporting/core/report.py @@ -43,9 +43,11 @@ split_by_hash_change, validate_full_recalculation_scope, ) +from impulse_reporting.events.container_boundary_event import ContainerBoundaryEvent from impulse_reporting.events.container_event import ContainerEvent from impulse_reporting.events.event import Event from impulse_reporting.events.event_types import EventType +from impulse_reporting.events.time_window_event import TimeWindowEvent from impulse_reporting.incremental.container_detector import ContainerUpsertDetector from impulse_reporting.incremental.definition_hash_comparator import ( DefinitionHashComparator, @@ -414,6 +416,15 @@ def add_event(self, event: Event): ) self.events.append(event) event.set_report_id(self.report_id) + if isinstance(event, TimeWindowEvent): + # The windows are computed in the channel time frame, so it is part of the + # event's (and its scoped aggregations') definition. + solver_config = self.solver.config + event.set_channel_time( + solver_config.channel_time_unit, + solver_config.channel_time_origin, + solver_config.container_time_unit, + ) def get_events(self) -> list[Event]: """ @@ -1137,12 +1148,13 @@ def determine_report(self, is_incremental: bool = None): ) ) - # Collect all solvable expressions (exclude ContainerEvent) + # Collect all solvable expressions. Container-boundary events (ContainerEvent, + # TimeWindowEvent) resolve from container_metrics, not the channel solve. all_changed_expressions = collect_solvable_expressions( - changed_events_by_type, EventType, exclude_cls=ContainerEvent + changed_events_by_type, EventType, exclude_cls=ContainerBoundaryEvent ) + collect_solvable_expressions(changed_aggs_by_type, AggregationType) all_unchanged_expressions = collect_solvable_expressions( - unchanged_events_by_type, EventType, exclude_cls=ContainerEvent + unchanged_events_by_type, EventType, exclude_cls=ContainerBoundaryEvent ) + collect_solvable_expressions(unchanged_aggs_by_type, AggregationType) # Centralized solve @@ -1163,7 +1175,7 @@ def determine_report(self, is_incremental: bool = None): self.query, self.solver, changed_pre_filtered_containers_df, - ContainerEvent, + ContainerBoundaryEvent, ) unchanged_event_dfs = dispatch_events( self.spark, @@ -1173,7 +1185,7 @@ def determine_report(self, is_incremental: bool = None): self.query, self.solver, pre_filtered_containers_df, - ContainerEvent, + ContainerBoundaryEvent, ) # Merge event results into {type: {"changed": df, "unchanged": df}} and diff --git a/src/impulse_reporting/core/report_utils.py b/src/impulse_reporting/core/report_utils.py index 1a540e4b..cde1c9f5 100644 --- a/src/impulse_reporting/core/report_utils.py +++ b/src/impulse_reporting/core/report_utils.py @@ -316,12 +316,12 @@ def dispatch_events( query: QueryBuilder, solver: QuerySolver, pre_filtered_containers_df: DataFrame | None, - container_event_cls: type, + boundary_event_cls: type, ) -> dict: """Dispatch ``determine_events`` calls per type. - Solvable event types receive ``solved_df``; ``ContainerEvent`` receives - ``query``/``solver``. + Solvable event types receive ``solved_df``; container-boundary events + (``ContainerEvent``, ``TimeWindowEvent``) receive ``query``/``solver``. Parameters ---------- @@ -332,8 +332,9 @@ def dispatch_events( query : QueryBuilder solver : QuerySolver pre_filtered_containers_df : DataFrame | None - container_event_cls : type - The ``ContainerEvent`` class. + boundary_event_cls : type + Base class of the container-boundary events (``ContainerBoundaryEvent``); + subclasses are resolved via the filter pipeline instead of ``solved_df``. Returns ------- @@ -347,8 +348,8 @@ def dispatch_events( continue cls = type_enum[type_name].value - if issubclass(cls, container_event_cls): - # ContainerEvent uses filter pipeline, not solved_df + if issubclass(cls, boundary_event_cls): + # Container-boundary events use the filter pipeline, not solved_df event_dfs[type_name] = cls.determine_events( spark, events, diff --git a/src/impulse_reporting/events/container_boundary_event.py b/src/impulse_reporting/events/container_boundary_event.py new file mode 100644 index 00000000..e92ab580 --- /dev/null +++ b/src/impulse_reporting/events/container_boundary_event.py @@ -0,0 +1,134 @@ +"""ContainerBoundaryEvent — base for events derived from container boundaries.""" + +from __future__ import annotations + +import hashlib +import zlib +from collections.abc import Mapping + +from pyspark.sql import DataFrame, Row, SparkSession + +from impulse_query_engine.analyze.query.query_builder import QueryBuilder +from impulse_query_engine.analyze.query.solvers.query_solver import QuerySolver +from impulse_reporting.events.event import Event +from impulse_reporting.persist.dimension_schema import EVENT_DIMENSION_SCHEMA + + +class ContainerBoundaryEvent(Event): + """Base class for events whose instances are derived from container boundaries. + + The instances are resolved from ``container_metrics`` (``start_ts`` / ``stop_ts``) + via the solver's filter pipeline instead of the centralized channel solve, so every + filtered container yields instances regardless of its channel data. The report + therefore excludes these event types from the solvable expressions and dispatches + them with ``query`` / ``solver`` rather than ``solved_df``. + + Subclasses set ``description`` and ``attributes`` (via :meth:`_normalize_attributes`), + and ``required_channels`` when they have any; :meth:`as_dict` writes them to + ``event_dimension``. + """ + + required_channels: list[str] | None = None + + @staticmethod + def _normalize_attributes(attributes: Mapping[str, str] | None) -> dict[str, str]: + """Return *attributes* with string keys and values (empty when ``None``).""" + return {str(k): str(v) for k, v in (attributes or {}).items()} + + @staticmethod + def _sha256_long(text: str) -> int: + """SHA-256 of *text*, truncated to a signed 64-bit int (the ``definition_hash`` + type).""" + hash_bytes = hashlib.sha256(text.encode()).digest() + return int.from_bytes(hash_bytes[:8], byteorder="big", signed=True) + + def as_dict(self) -> dict: + """Return the event's ``event_dimension`` row as a dictionary. + + Returns + ------- + dict + Event metadata keyed by the ``event_dimension`` column names. + """ + return { + "event_id": self.get_id(), + "report_id": self.report_id, + "event_type": self.get_event_type_str(), + "event_name": self.name, + "event_description": self.description, + "required_channels": self.required_channels, + "event_expression": self.get_expression_str(), + "definition_hash": self.determine_definition_hash(), + "attributes": self.attributes, + } + + def get_id(self) -> int: + """Return a unique identifier derived from the event name. + + Returns + ------- + int + Positive 32-bit integer identifier. + """ + return zlib.crc32(self.name.encode()) & 0x7FFFFFFF + + def as_spark_row(self) -> Row: + """Return a Spark ``Row`` representation of :meth:`as_dict`. + + Returns + ------- + Row + """ + return Row(**self.as_dict()) + + @classmethod + def determine_metadata_df( + cls, spark: SparkSession, events: list[ContainerBoundaryEvent] + ) -> DataFrame: + """Create a Spark DataFrame containing event metadata. + + Parameters + ---------- + spark : SparkSession + Active Spark session. + events : list of ContainerBoundaryEvent + Events of one container-boundary type. + + Returns + ------- + DataFrame + Spark DataFrame matching ``EVENT_DIMENSION_SCHEMA``. + """ + rows = [event.as_spark_row() for event in events] + return spark.createDataFrame(rows, schema=EVENT_DIMENSION_SCHEMA) + + @staticmethod + def resolve_container_metrics( + spark: SparkSession, + query: QueryBuilder, + solver: QuerySolver, + pre_filtered_containers_df: DataFrame = None, + ) -> DataFrame: + """Resolve the filtered containers' metrics via the solver filter pipeline. + + Parameters + ---------- + spark : SparkSession + Active Spark session. + query : QueryBuilder + Query builder with filters applied. + solver : QuerySolver + Solver whose filter pipeline is used for container resolution. + pre_filtered_containers_df : DataFrame, optional + Pre-filtered containers for incremental processing. + + Returns + ------- + DataFrame + Column-mapped ``container_metrics`` rows of the matching containers, with the + original ``start_ts`` / ``stop_ts``. + """ + container_tags_df = solver.filter_container_tags(spark, query) + return solver.filter_container_metrics( + spark, query, container_tags_df, pre_filtered_containers_df + ) diff --git a/src/impulse_reporting/events/container_event.py b/src/impulse_reporting/events/container_event.py index 7a6ea264..66ad7cda 100644 --- a/src/impulse_reporting/events/container_event.py +++ b/src/impulse_reporting/events/container_event.py @@ -2,17 +2,14 @@ from __future__ import annotations -import hashlib from typing import TYPE_CHECKING import pyspark.sql.functions as f -import zlib -from pyspark.sql import DataFrame, Row, SparkSession +from pyspark.sql import DataFrame, SparkSession from impulse_query_engine.analyze.query.query_builder import QueryBuilder from impulse_query_engine.analyze.query.solvers.query_solver import QuerySolver -from impulse_reporting.events.event import Event -from impulse_reporting.persist.dimension_schema import EVENT_DIMENSION_SCHEMA +from impulse_reporting.events.container_boundary_event import ContainerBoundaryEvent from impulse_reporting.persist.fact_schema import EVENT_INSTANCE_FACT_SCHEMA from impulse_reporting.util.event_instance_util import generate_event_instance_id_column from impulse_reporting.util.report_entity_util import ReportEntityUtil @@ -23,7 +20,7 @@ ) -class ContainerEvent(Event): +class ContainerEvent(ContainerBoundaryEvent): """Event that treats the full measurement container as a single event instance. Unlike ``BasicEvent``, no time-series expression is needed — the event @@ -46,25 +43,12 @@ def __init__(self, name: str, desc: str = None, attributes: dict[str, str] = Non """ super().__init__(name) self.description = desc - normalized_attributes: dict[str, str] = {} - if attributes is not None: - normalized_attributes = {str(k): str(v) for k, v in attributes.items()} - self.attributes = normalized_attributes + self.attributes = self._normalize_attributes(attributes) # ------------------------------------------------------------------ # Instance methods # ------------------------------------------------------------------ - def get_id(self) -> int: - """Return a unique identifier derived from the event name. - - Returns - ------- - int - Positive 32-bit integer identifier. - """ - return zlib.crc32(self.name.encode()) & 0x7FFFFFFF - def get_expression(self) -> TimeSeriesExpression | None: """ContainerEvent has no time-series expression. @@ -97,37 +81,7 @@ def determine_definition_hash(self) -> int: int Hash value representing the computation definition. """ - hash_input = self.name - hash_bytes = hashlib.sha256(hash_input.encode()).digest() - return int.from_bytes(hash_bytes[:8], byteorder="big", signed=True) - - def as_dict(self) -> dict: - """Return a dictionary representation of the event. - - Returns - ------- - dict - """ - return { - "event_id": self.get_id(), - "report_id": self.report_id, - "event_type": self.get_event_type_str(), - "event_name": self.name, - "event_description": self.description, - "required_channels": None, - "event_expression": self.get_expression_str(), - "definition_hash": self.determine_definition_hash(), - "attributes": self.attributes, - } - - def as_spark_row(self) -> Row: - """Return a Spark ``Row`` representation. - - Returns - ------- - Row - """ - return Row(**self.as_dict()) + return self._sha256_long(self.name) # ------------------------------------------------------------------ # Class methods @@ -170,9 +124,8 @@ def determine_events( Spark DataFrame matching ``EVENT_INSTANCE_FACT_SCHEMA``. """ # Resolve containers via solver filter pipeline - container_tags_df = solver.filter_container_tags(spark, query) - container_metrics_df = solver.filter_container_metrics( - spark, query, container_tags_df, pre_filtered_containers_df + container_metrics_df = cls.resolve_container_metrics( + spark, query, solver, pre_filtered_containers_df ) # Rename silver columns to gold event fact column names and cast @@ -206,22 +159,3 @@ def determine_events( # Select only the columns defined in the fact schema return df.select(EVENT_INSTANCE_FACT_SCHEMA.fieldNames()) - - @classmethod - def determine_metadata_df(cls, spark: SparkSession, events: list[ContainerEvent]) -> DataFrame: - """Create a Spark DataFrame containing event metadata. - - Parameters - ---------- - spark : SparkSession - Active Spark session. - events : list of ContainerEvent - List of ContainerEvent objects. - - Returns - ------- - DataFrame - Spark DataFrame matching ``EVENT_DIMENSION_SCHEMA``. - """ - rows = [event.as_spark_row() for event in events] - return spark.createDataFrame(rows, schema=EVENT_DIMENSION_SCHEMA) diff --git a/src/impulse_reporting/events/event.py b/src/impulse_reporting/events/event.py index 9d590c77..a45ea243 100644 --- a/src/impulse_reporting/events/event.py +++ b/src/impulse_reporting/events/event.py @@ -171,9 +171,9 @@ def determine_events( solved_df : DataFrame, optional Pre-solved wide DataFrame from centralized batch solve. query : QueryBuilder, optional - Query builder for constructing event queries (ContainerEvent path). + Query builder for constructing event queries (container-boundary event path). solver : QuerySolver, optional - Query solver for executing queries (ContainerEvent path). + Query solver for executing queries (container-boundary event path). pre_filtered_containers_df : DataFrame, optional Pre-filtered containers for incremental processing. diff --git a/src/impulse_reporting/events/event_types.py b/src/impulse_reporting/events/event_types.py index 5b877096..608c350e 100644 --- a/src/impulse_reporting/events/event_types.py +++ b/src/impulse_reporting/events/event_types.py @@ -6,6 +6,7 @@ from impulse_reporting.events.container_event import ContainerEvent from impulse_reporting.events.points_in_time_event import PointsInTimeEvent from impulse_reporting.events.sequence_of_events import SequenceOfEvents +from impulse_reporting.events.time_window_event import TimeWindowEvent from impulse_reporting.persist.dimension_schema import EVENT_DIMENSION_SCHEMA from impulse_reporting.persist.fact_schema import EVENT_INSTANCE_FACT_SCHEMA @@ -25,6 +26,8 @@ class EventType(Enum): Container event type spanning the full measurement container. SEQUENCE_OF_EVENTS : SequenceOfEvents Sequence-of-events type for ordered interval sequence detection. + TIME_WINDOW_EVENT : TimeWindowEvent + Fixed-duration time-window type; one instance per window across each container. """ @@ -32,6 +35,7 @@ class EventType(Enum): CONTAINER_EVENT = ContainerEvent SEQUENCE_OF_EVENTS = SequenceOfEvents POINTS_IN_TIME_EVENT = PointsInTimeEvent + TIME_WINDOW_EVENT = TimeWindowEvent def get_fact_table_name(self) -> str: """ @@ -53,6 +57,7 @@ def get_fact_table_name(self) -> str: | EventType.CONTAINER_EVENT | EventType.SEQUENCE_OF_EVENTS | EventType.POINTS_IN_TIME_EVENT + | EventType.TIME_WINDOW_EVENT ): return "event_instance_fact" case _: @@ -78,6 +83,7 @@ def get_fact_schema(self) -> StructType: | EventType.CONTAINER_EVENT | EventType.SEQUENCE_OF_EVENTS | EventType.POINTS_IN_TIME_EVENT + | EventType.TIME_WINDOW_EVENT ): return EVENT_INSTANCE_FACT_SCHEMA case _: @@ -103,6 +109,7 @@ def get_dimension_table_name(self) -> str: | EventType.CONTAINER_EVENT | EventType.SEQUENCE_OF_EVENTS | EventType.POINTS_IN_TIME_EVENT + | EventType.TIME_WINDOW_EVENT ): return "event_dimension" case _: @@ -126,6 +133,7 @@ def get_dimension_schema(self) -> StructType: | EventType.CONTAINER_EVENT | EventType.SEQUENCE_OF_EVENTS | EventType.POINTS_IN_TIME_EVENT + | EventType.TIME_WINDOW_EVENT ): return EVENT_DIMENSION_SCHEMA case _: diff --git a/src/impulse_reporting/events/time_window_event.py b/src/impulse_reporting/events/time_window_event.py new file mode 100644 index 00000000..38367081 --- /dev/null +++ b/src/impulse_reporting/events/time_window_event.py @@ -0,0 +1,258 @@ +"""TimeWindowEvent — splits each container into consecutive fixed-duration windows.""" + +from __future__ import annotations + +from collections.abc import Mapping + +import pyspark.sql.functions as f +from pyspark.sql import DataFrame, SparkSession + +from impulse_query_engine.analyze.metadata.time_series_expression import ( + TimeSeriesExpression, +) +from impulse_query_engine.analyze.query.events.time_window_expression import ( + MAX_WINDOWS_PER_CONTAINER, + TimeWindowExpression, + validate_max_windows, + window_intervals_udf, +) +from impulse_query_engine.analyze.query.query_builder import QueryBuilder +from impulse_query_engine.analyze.query.solvers.query_solver import QuerySolver +from impulse_query_engine.analyze.query.solvers.utils.window_bounds import with_window_bounds +from impulse_reporting.events.container_boundary_event import ContainerBoundaryEvent +from impulse_reporting.persist.fact_schema import EVENT_INSTANCE_FACT_SCHEMA +from impulse_reporting.util.event_instance_util import generate_event_instance_id_column +from impulse_reporting.util.report_entity_util import ReportEntityUtil + + +class TimeWindowEvent(ContainerBoundaryEvent): + """Event that divides each measurement container into consecutive fixed windows. + + Unlike ``ContainerEvent`` (one instance per container), a ``TimeWindowEvent`` emits one + event instance per fixed-duration slice, tiling the container's ``start_ts`` / ``stop_ts`` + span with windows of length ``window_length``. The final slice is clamped to the + container end. + + The event fact is computed from ``container_metrics`` alone (via + :func:`window_intervals_udf`), so every filtered container gets windows regardless of + its channel data. Aggregations scoped to this event evaluate the + :class:`TimeWindowExpression` in the solve. Both use the same window function + (``tile_windows``), so they produce identical windows, and the timestamp-based + ``event_instance_id`` (like for other interval events) matches on both sides. + """ + + def __init__( + self, + name: str, + window_length: float, + desc: str = None, + required_channels: list[str] = None, + attributes: Mapping[str, str] = None, + max_windows_per_container: int = MAX_WINDOWS_PER_CONTAINER, + ): + """ + Initialize a TimeWindowEvent object. + + Parameters + ---------- + name : str + Name of the event. + window_length : float + Fixed window length, in the same time unit as the underlying timestamps + (e.g. milliseconds-since-epoch). Must be strictly positive and finite. + desc : str, optional + Description of the event. + required_channels : list of str, optional + List of required channels for the event. Informational; stored in the event + dimension table. + attributes : Mapping[str, str], optional + Key-value metadata for the event. ``window_length`` is surfaced here + automatically (without overriding a user-supplied key). + max_windows_per_container : int, optional + Maximum number of windows per container (default 1,000,000). A container + exceeding it fails the report with an error naming the limit, which usually + means ``window_length`` is in the wrong unit for the boundaries. Not part of + the definition hash. + + Raises + ------ + ValueError + If ``window_length`` is not strictly positive and finite, or + ``max_windows_per_container`` is not a positive integer. + """ + ContainerBoundaryEvent.__init__(self, name) + # window_length is validated by TimeWindowExpression. max_windows_per_container is + # validated here so the error names this event's parameter, not the expression's. + max_windows_per_container = validate_max_windows( + max_windows_per_container, param_name="max_windows_per_container" + ) + self.expression = TimeWindowExpression( + window_length, max_windows=max_windows_per_container + ).alias(name) + # Use the expression's normalized (float) length everywhere, so the event fact, + # the solve and event_dimension all see the same value for 10 and 10.0. + self.window_length = self.expression.window_length + self.max_windows_per_container = self.expression.max_windows + self.description = desc + self.required_channels = required_channels + self.attributes = self._normalize_attributes(attributes) + # Surface the window length for traceability in event_dimension, without + # clobbering an explicit user-supplied attribute of the same key. + self.attributes.setdefault("window_length", str(self.window_length)) + + def set_channel_time( + self, unit: str | None, origin: str = "epoch", container_unit: str | None = None + ) -> None: + """Record the channel time frame the windows are computed in. + + Set by ``Report.add_event`` from the report's ``solver_config``. Stored on the + expression, whose string form feeds the definition hashes of this event and of the + aggregations scoped to it. + + Parameters + ---------- + unit : str or None + The report's ``solver_config.channel_time_unit``. + origin : str, optional + The report's ``solver_config.channel_time_origin`` (default ``"epoch"``). + container_unit : str or None, optional + The report's ``solver_config.container_time_unit``. + """ + self.expression.channel_time_unit = unit + self.expression.channel_time_origin = origin + self.expression.container_time_unit = container_unit + + def get_expression(self) -> TimeSeriesExpression | None: + """ + Get the time series expression associated with the event. + + Returns + ------- + TimeSeriesExpression or None + The time-window expression for the event. + """ + return self.expression + + def get_event_type_str(self) -> str: + """Get the event type string for TimeWindowEvent. + + Returns + ------- + str + Event type string. + """ + return "TIME_WINDOW_EVENT" + + def determine_definition_hash(self) -> int: + """ + Calculate definition hash for the time-window event. + + Only includes the expression string, which encodes the attributes that affect the + event results: ``window_length`` and the channel time frame (``channel_time_unit``, + ``channel_time_origin``, ``container_time_unit``; omitted while unset / default). + Resizing the window or changing the time frame therefore forces a full recompute in + incremental mode. + + Excludes: name, description, required_channels, max_windows_per_container, + report_id + + Returns + ------- + int + Hash value representing the computation definition. + """ + return self._sha256_long(self.get_expression_str()) + + @classmethod + def determine_events( + cls, + spark: SparkSession, + events: list[TimeWindowEvent], + *, + solved_df: DataFrame = None, + query: QueryBuilder = None, + solver: QuerySolver = None, + pre_filtered_containers_df: DataFrame = None, + ) -> DataFrame: + """ + Extract the event fact table for the given list of TimeWindowEvent objects. + + Resolves the matching containers via the solver's filter pipeline (like + ``ContainerEvent``) and computes each event's windows natively from the + containers' ``start_ts`` / ``stop_ts`` in the channel time frame + (``solvers.utils.window_bounds.with_window_bounds``), so every filtered container + gets windows. + Each window becomes one event instance (``start_ts < end_ts``) whose + ``event_instance_id`` hashes its boundaries. The solve uses the same window function + for scoped aggregations (see :func:`window_intervals_udf`), so the ids match. + + Parameters + ---------- + spark : SparkSession + Spark session for data processing. + events : list of TimeWindowEvent + List of TimeWindowEvent objects to process. + solved_df : DataFrame, optional + Not used by TimeWindowEvent (kept for interface compatibility). + query : QueryBuilder, optional + Query builder with filters applied. + solver : QuerySolver, optional + Solver whose filter pipeline is used for container resolution. + pre_filtered_containers_df : DataFrame, optional + Pre-filtered containers for incremental processing. + + Returns + ------- + DataFrame + Spark DataFrame containing event instance facts. + """ + container_metrics_df = cls.resolve_container_metrics( + spark, query, solver, pre_filtered_containers_df + ) + # The windows are computed in the channel time frame, from the same bounds the solve + # uses for scoped aggregations (fails fast on the schema, e.g. when TIMESTAMP + # boundaries lack solver_config.channel_time_unit). + container_metrics_df = with_window_bounds(container_metrics_df, solver.config) + start_ts = f.col(solver.config.window_start_col) + stop_ts = f.col(solver.config.window_stop_col) + + # One (event_name, windows) struct per event, exploded in a single pass over the + # containers. + per_event = f.array( + *[ + f.struct( + f.lit(event.get_name()).alias("event_name"), + window_intervals_udf( + event.window_length, max_windows=event.max_windows_per_container + )(start_ts, stop_ts).alias("windows"), + ) + for event in events + ] + ) + + df = ( + container_metrics_df.select( + f.col(solver.config.container_id_col).alias("container_id"), + f.explode(per_event).alias("event"), + ) + .select( + "container_id", + f.col("event.event_name").alias("event_name"), + f.inline( + f.arrays_zip( + f.col("event.windows.starts").alias("start_ts"), + f.col("event.windows.ends").alias("end_ts"), + ) + ), + ) + .withColumn( + "event_instance_id", + generate_event_instance_id_column(event_type=TimeWindowEvent), + ) + .withColumn( + "event_id", + ReportEntityUtil.get_event_id_column(elements=events, element_name="event_name"), + ) + .select(EVENT_INSTANCE_FACT_SCHEMA.fieldNames()) + ) + return df diff --git a/tests/impulse_query_engine/unit/analyze/query/events/time_window_expression_test.py b/tests/impulse_query_engine/unit/analyze/query/events/time_window_expression_test.py new file mode 100644 index 00000000..3e946121 --- /dev/null +++ b/tests/impulse_query_engine/unit/analyze/query/events/time_window_expression_test.py @@ -0,0 +1,374 @@ +from __future__ import annotations + +import random +from unittest.mock import MagicMock + +import numpy as np +import pandas as pd +import pyspark.sql.functions as F +import pytest + +from impulse_query_engine.analyze.query.aggregations.stats_aggregator import StatsAggregator +from impulse_query_engine.analyze.query.events import TimeWindowExpression +from impulse_query_engine.analyze.query.events.time_window_expression import ( + MAX_WINDOWS_PER_CONTAINER, + tile_windows, + window_intervals_udf, +) +from impulse_query_engine.analyze.query.solvers.empty_cache import EmptyTimeSeriesCache +from impulse_query_engine.model.series.intervals import Intervals +from impulse_query_engine.model.series.sample_series import SampleSeries +from tests.conftest import spark # noqa: F401 (pytest fixture) + + +class _FakeCache: + """Minimal SeriesCache stand-in exposing container metrics for ``build``.""" + + def __init__(self, container_metrics: dict): + self._container_metrics = container_metrics + + @property + def container_metrics(self) -> dict: + return self._container_metrics + + @property + def container_tags(self) -> dict: + return {} + + +# The container bounds in the channel time frame, as SolverConfig.with_window_bounds adds them. +_WINDOW_START, _WINDOW_STOP = "__window_start", "__window_stop" + + +def _build(start_ts, stop_ts, window_length, **kwargs) -> Intervals: + expr = TimeWindowExpression(window_length, **kwargs) + return expr.build(_FakeCache({_WINDOW_START: start_ts, _WINDOW_STOP: stop_ts})) + + +def test_exact_multiple_windows_last_ends_at_stop(): + """D=100, W=10 -> 10 contiguous windows, final window ends exactly at stop_ts.""" + iv = _build(0, 100, 10) + assert len(iv) == 10 + assert iv.tstarts.tolist() == [0, 10, 20, 30, 40, 50, 60, 70, 80, 90] + assert iv.tends.tolist() == [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] + # Contiguity: each window's end equals the next window's start. + assert iv.tstarts[1:].tolist() == iv.tends[:-1].tolist() + assert iv.tends[-1] == 100 + + +def test_non_multiple_has_short_final_window_clamped_to_stop(): + """D=105, W=10 -> 11 windows; the last is a short slice clamped to stop_ts.""" + iv = _build(0, 105, 10) + assert len(iv) == 11 + assert iv.tstarts[-1] == 100 + assert iv.tends[-1] == 105 # clamped, not 110 + # The final window is shorter than the fixed length. + assert (iv.tends[-1] - iv.tstarts[-1]) < 10 + assert iv.tstarts[1:].tolist() == iv.tends[:-1].tolist() + + +def test_span_equal_to_window_yields_single_window(): + iv = _build(1000, 1010, 10) + assert len(iv) == 1 + assert iv.tstarts.tolist() == [1000] + assert iv.tends.tolist() == [1010] + + +def test_span_smaller_than_window_yields_single_clamped_window(): + iv = _build(1000, 1001, 10) + assert len(iv) == 1 + assert iv.tstarts.tolist() == [1000] + assert iv.tends.tolist() == [1001] + + +def test_epoch_millisecond_boundaries(): + """Realistic epoch-ms container with a 10s (10000 ms) window.""" + start, stop = 1751528502708, 1751528610253 # ~107.545 s span + iv = _build(start, stop, 10000) + assert len(iv) == int(np.ceil((stop - start) / 10000)) # 11 + assert iv.tstarts[0] == start + assert iv.tends[-1] == stop + assert iv.tstarts[1:].tolist() == iv.tends[:-1].tolist() + assert bool(np.all(iv.tstarts < iv.tends)) + + +def test_degenerate_container_yields_no_windows(): + assert len(_build(50, 50, 10)) == 0 # stop == start + assert len(_build(60, 50, 10)) == 0 # stop < start + + +def test_missing_container_metrics_yields_empty(): + expr = TimeWindowExpression(10) + assert len(expr.build(_FakeCache({}))) == 0 + # The empty cache used by evaluation_type() has no container metrics. + assert len(expr.build(EmptyTimeSeriesCache())) == 0 + + +def test_evaluation_type_is_intervals(): + assert TimeWindowExpression(10).evaluation_type() is Intervals + + +def test_no_selectors_and_requests_container_metrics(): + expr = TimeWindowExpression(10) + assert expr.get_selectors() == [] + assert expr.get_selector_expr() is None + # The bounds in the channel time frame, not the raw start_ts / stop_ts (which UDFs + # keep reading unconverted). + assert expr.required_container_metrics() == {_WINDOW_START, _WINDOW_STOP} + assert expr.required_tags() == set() + + +def test_str_includes_window_length(): + assert "window_length=10" in str(TimeWindowExpression(10)) + + +def test_str_stable_across_int_and_float_window_length(): + # int 10 and float 10.0 are the same window; the string form (which feeds the event + # definition hash) must not differ between them. + assert str(TimeWindowExpression(10)) == str(TimeWindowExpression(10.0)) + + +def test_str_includes_channel_time_frame_only_when_set(): + # The string feeds the definition hashes of the event and its scoped aggregations, so + # the channel time frame must move them, while the defaults keep the plain form. + expr = TimeWindowExpression(10) + assert str(expr) == "TimeWindowExpression" + expr.channel_time_unit = "ms" + assert str(expr) == "TimeWindowExpression" + expr.channel_time_origin = "container_start" + assert str(expr) == ( + "TimeWindowExpression" + ) + expr.container_time_unit = "s" + assert str(expr).endswith(", container_time_unit=s>") + + +def test_max_windows_not_part_of_str(): + # The cap only decides between an error and a result, so it must not force a recompute. + assert MAX_WINDOWS_PER_CONTAINER == 1_000_000 + assert TimeWindowExpression(10).max_windows == MAX_WINDOWS_PER_CONTAINER + assert str(TimeWindowExpression(10, max_windows=5)) == str(TimeWindowExpression(10)) + + +@pytest.mark.parametrize("bad", [0, -1, 1.5, True, None, "10"]) +def test_invalid_max_windows_raises(bad): + with pytest.raises(ValueError, match="max_windows must be a positive integer"): + TimeWindowExpression(10, max_windows=bad) + + +def test_build_raises_beyond_max_windows(): + # 10 windows are fine at max_windows=10, not at 9. + assert len(_build(0, 100, 10, max_windows=10)) == 10 + with pytest.raises(ValueError, match="10 windows of length 10.0 .* exceed max_windows=9"): + _build(0, 100, 10, max_windows=9) + + +def test_build_unit_mismatch_hits_default_cap(): + # window_length=60 meant as seconds over a 1 h ns-epoch span: 6e10 windows. + start = 1_700_000_000_000_000_000 + with pytest.raises(ValueError, match="unit of the channel timestamps"): + _build(np.int64(start), np.int64(start + 3_600_000_000_000), 60) + + +@pytest.mark.parametrize("bad", [0, -1, -10.5, None, float("inf"), float("-inf"), float("nan")]) +def test_non_positive_window_length_raises(bad): + with pytest.raises(ValueError, match="strictly positive"): + TimeWindowExpression(bad) + + +def test_int64_and_float64_inputs_build_identical_windows(): + # A long container metric reaches pandas as int64 or float64 depending on the group + # (nulls force float64); both must produce the same windows. + start, stop = 1_700_000_000_000_000_123, 1_700_000_007_000_000_049 + a = _build(np.int64(start), np.int64(stop), 1_000_000_007) + b = _build(np.float64(start), np.float64(stop), 1_000_000_007) + assert a.get_data() == b.get_data() + + +def test_nan_container_metrics_yield_empty(): + # A null start/stop arrives as NaN in a float64 column. + assert len(_build(np.nan, 100.0, 10)) == 0 + assert len(_build(0.0, np.nan, 10)) == 0 + + +def test_infinite_container_metrics_yield_empty(): + # Used to overflow in int(np.ceil(inf)). + assert len(_build(0.0, np.inf, 10)) == 0 + assert len(_build(-np.inf, 100.0, 10)) == 0 + + +def test_tile_windows_none_and_na_yield_empty(): + for start, stop in ((None, 100), (0, None), (pd.NA, 100), (0, np.nan)): + starts, ends = tile_windows(start, stop, 10) + assert len(starts) == len(ends) == 0 + + +# --------------------------------------------------------------------------- +# window_intervals_udf: tile_windows on the event fact side (one row per container) +# --------------------------------------------------------------------------- +def _udf_windows(spark, rows, window_length, ts_type="long", **kwargs): # noqa: F811 + # One partition, so all rows reach the UDF in one Arrow batch. + df = spark.createDataFrame(rows, f"k int, start_ts {ts_type}, stop_ts {ts_type}").coalesce(1) + windows = window_intervals_udf(window_length, **kwargs) + out = df.select("k", windows(F.col("start_ts"), F.col("stop_ts")).alias("w")) + return out, { + r.k: [[s, e] for s, e in zip(r.w.starts, r.w.ends, strict=True)] for r in out.collect() + } + + +def test_window_intervals_udf_edge_cases(spark): # noqa: F811 + rows = [ + (0, 0, 100), # exact multiple -> 10 windows + (1, 0, 105), # short final window clamped to stop + (2, 1000, 1010), # span == W -> 1 window + (3, 1000, 1001), # span < W -> 1 clamped window + (4, 50, 50), # stop == start -> none + (5, 60, 50), # stop < start -> none + (6, None, 50), # null bound -> none + (7, 0, None), + ] + out, w = _udf_windows(spark, rows, 10) + + assert out.schema["w"].dataType.simpleString() == ( + "struct,ends:array>" + ) + assert w[0] == [[float(s), float(s + 10)] for s in range(0, 100, 10)] + assert w[1][-1] == [100.0, 105.0] and len(w[1]) == 11 + assert w[2] == [[1000.0, 1010.0]] + assert w[3] == [[1000.0, 1001.0]] + assert w[4] == w[5] == w[6] == w[7] == [] + + +def test_window_intervals_udf_non_finite_bounds_yield_no_windows(spark): # noqa: F811 + nan, inf = float("nan"), float("inf") + rows = [(0, 0.0, nan), (1, nan, 10.0), (2, nan, nan), (3, 0.0, inf), (4, -inf, 10.0)] + _, w = _udf_windows(spark, rows, 4, ts_type="double") + assert all(w[k] == [] for k, _, _ in rows), w + + +def test_window_intervals_udf_raises_beyond_max_windows(spark): # noqa: F811 + _, ok = _udf_windows(spark, [(0, 0, 100)], 10, max_windows=10) + assert len(ok[0]) == 10 + with pytest.raises(Exception, match="10 windows of length 10.0 .* exceed max_windows=9"): + _udf_windows(spark, [(0, 0, 100)], 10, max_windows=9) + + +def test_window_intervals_udf_invalid_max_windows_raises(): + with pytest.raises(ValueError, match="max_windows must be a positive integer"): + window_intervals_udf(10, max_windows=0) + + +def test_window_intervals_udf_uniform_and_empty_batches(spark): # noqa: F811 + # Equal window counts across a batch still give one array per row. + _, w = _udf_windows(spark, [(k, 100 * k, 100 * k + 30) for k in range(4)], 10) + assert w == {k: [[100.0 * k + s, 100.0 * k + s + 10] for s in (0, 10, 20)] for k in range(4)} + # A batch without any windows. + _, w = _udf_windows(spark, [(0, None, None), (1, 50, 50)], 10) + assert w == {0: [], 1: []} + + +def _as_list(windows) -> list[tuple[float, float]]: + """Windows as ordered (start, end) pairs.""" + return [(float(s), float(e)) for s, e in windows] + + +def _count_mismatch_case(window_length: float) -> tuple[int, int]: + """Find ns-epoch (start, stop) whose int64 and double spans yield different counts. + + This is exactly the case where exact int64 subtraction and double subtraction disagree + on the number of windows, so tile_windows must convert to float first on every path. + """ + base = 1_700_000_000_000_000_000 + for start in range(base, base + 512): + for k in (1, 3, 5, 7): + for d in range(-300, 1): + stop = start + int(k * window_length) + d + exact = int(np.ceil((stop - start) / window_length)) + rounded = int(np.ceil((float(stop) - float(start)) / window_length)) + if exact != rounded: + return start, stop + raise AssertionError("no int64/double count-mismatch case found") + + +def _batch_dtype_udf(): + """Pandas UDF reporting the dtype the bounds arrive in, per row of the batch (created + lazily: defining a pandas UDF needs an active Spark session).""" + + @F.pandas_udf("string") + def batch_dtype(start: pd.Series) -> pd.Series: + return pd.Series([str(start.dtype)] * len(start)) + + return batch_dtype + + +def test_event_fact_and_solve_windows_identical_across_input_dtypes(spark): # noqa: F811 + """Both sides call tile_windows, but pandas hands the bounds over differently: the event + fact UDF gets int64 for a batch without nulls and float64 once a null is in the batch, + the solve gets float64 (its container metrics are nulled on most rows). For ns epochs + beyond 2^53, including a span where int64 and double subtraction disagree on the count, + all paths must produce the same windows in the same order.""" + rnd = random.Random(7) + window_length = 1_000_000_007.0 + cases = [] + for _ in range(60): + start = rnd.randint(1_600_000_000_000_000_000, 1_800_000_000_000_000_000) + cases.append( + (start, start + int(rnd.randint(1, 50) * window_length) + rnd.randint(-600, 600)) + ) + cases.append(_count_mismatch_case(window_length)) + rows = [(k, s, e) for k, (s, e) in enumerate(cases)] + windows = window_intervals_udf(window_length) + batch_dtype = _batch_dtype_udf() + + def _event_fact(batch_rows) -> tuple[dict, set]: + # One partition, so one Arrow batch: a null anywhere turns the whole batch float64. + df = spark.createDataFrame(batch_rows, "k int, start_ts long, stop_ts long").coalesce(1) + out = df.select( + "k", + windows(F.col("start_ts"), F.col("stop_ts")).alias("w"), + batch_dtype(F.col("start_ts")).alias("dtype"), + ).collect() + per_container = { + r.k: _as_list(zip(r.w.starts, r.w.ends, strict=True)) for r in out if r.k >= 0 + } + return per_container, {r.dtype for r in out} + + without_nulls, dtypes_int = _event_fact(rows) + with_null, dtypes_float = _event_fact([*rows, (-1, None, None)]) + assert dtypes_int == {"int64"} and dtypes_float == {"float64"} + + mismatches = [] + for k, (start, stop) in enumerate(cases): + solve = _as_list(_build(np.float64(start), np.float64(stop), window_length).get_data()) + assert solve, f"case {k} produced no windows" + if not (without_nulls[k] == with_null[k] == solve): + mismatches.append((k, start, stop)) + assert not mismatches, f"event fact / solve window mismatch: {mismatches[:5]}" + + +def test_stats_aggregator_windows_equal_helper_windows(spark): # noqa: F811 + """A StatsAggregator scoped to a TimeWindowExpression emits exactly the helper's windows + (no merging of touching windows, no extra drops).""" + start, stop, w = 1_700_000_000_000_000_123, 1_700_000_007_000_000_049, 1_000_000_007.0 + expected = list(zip(*tile_windows(start, stop, w), strict=True)) + + # One channel sampled across the whole container, so every window has data. + ts = np.linspace(float(start), float(stop), 50) + series = SampleSeries(tstarts=ts[:-1], tends=ts[1:], values=np.arange(49, dtype=float)) + + channel = MagicMock() + channel.build.return_value = series + + agg = StatsAggregator( + input_expressions=[channel], + event_expression=TimeWindowExpression(w), + statistics=["mean"], + ) + cache = _FakeCache({_WINDOW_START: np.float64(start), _WINDOW_STOP: np.float64(stop)}) + event_timestamps, numeric_values, _, _ = agg.build(cache) + + assert len(expected) == 7 + # The same windows, in the same order. + assert _as_list(event_timestamps) == _as_list(expected) + assert len(numeric_values[0]) == len(event_timestamps) diff --git a/tests/impulse_query_engine/unit/analyze/query/solvers/default_solver_container_metadata_test.py b/tests/impulse_query_engine/unit/analyze/query/solvers/default_solver_container_metadata_test.py index b0293fe3..c87b52b8 100644 --- a/tests/impulse_query_engine/unit/analyze/query/solvers/default_solver_container_metadata_test.py +++ b/tests/impulse_query_engine/unit/analyze/query/solvers/default_solver_container_metadata_test.py @@ -14,11 +14,14 @@ """ import pandas as pd +import pyspark.sql.functions as F import pytest from pyspark.sql import SparkSession import impulse_query_engine.schema as S +from impulse_query_engine.analyze.query.aggregations.stats_aggregator import StatsAggregator from impulse_query_engine.analyze.query.channels.calculated_channel import CalculatedChannel +from impulse_query_engine.analyze.query.events import TimeWindowExpression from impulse_query_engine.analyze.query.solvers.default_solver import ( DefaultSolver, TimeSeriesCache, @@ -329,3 +332,96 @@ def test_cache_reads_container_meta_from_surviving_row_only(): ) assert cache.container_tags == {"brand": "BMW"} assert cache.container_metrics["num_channels"] == 11 + + +def _timestamp_boundaries_db(basic_narrow_db: MeasurementDB) -> MeasurementDB: + """Clone of basic_narrow_db with start_ts / stop_ts (epoch ms) recast to TIMESTAMP.""" + tables = dict(basic_narrow_db.config.debug_tables) + tables["container_metrics"] = ( + tables["container_metrics"] + .withColumn("start_ts", F.timestamp_millis("start_ts")) + .withColumn("stop_ts", F.timestamp_millis("stop_ts")) + ) + return MeasurementDB(MeasurementDBConfig.for_debug(tables), ws=basic_narrow_db.ws) + + +def _grab_start_ts(ts, container_metrics): + value = container_metrics["start_ts"] + if value is None: # type-inference pass on the empty cache + return 0.0 + # Like a customer UDF rebuilding absolute time: a pd.Timestamp (naive, in the session + # time zone, here UTC) becomes epoch microseconds (exact); anything else is flagged -1. + if not isinstance(value, pd.Timestamp): + return -1.0 + return float((value - pd.Timestamp("1970-01-01")) // pd.Timedelta(microseconds=1)) + + +@pytest.mark.parametrize( + "config", + [ + SolverConfig(), + SolverConfig(channel_time_unit="ms"), + SolverConfig(channel_time_unit="ms", channel_time_origin="container_start"), + ], +) +def test_udf_gets_raw_timestamp_start_ts_regardless_of_channel_time( + spark: SparkSession, basic_narrow_db: MeasurementDB, config: SolverConfig +): + """A UDF reading a TIMESTAMP start_ts always gets the absolute pd.Timestamp: the channel + time settings only shape TimeWindowEvent windows, never the raw container metrics.""" + db = _timestamp_boundaries_db(basic_narrow_db) + query = db.query + previous_tz = spark.conf.get("spark.sql.session.timeZone") + spark.conf.set("spark.sql.session.timeZone", "UTC") + try: + result = query.select( + query.channel(channel_name="Engine RPM") + .apply(_grab_start_ts, container_metrics=["start_ts"]) + .alias("start") + ).solve(spark, solver=DefaultSolver(spark, config=config)) + rows = {row.container_id: row.start for row in result.collect()} + finally: + spark.conf.set("spark.sql.session.timeZone", previous_tz) + + expected = { + r.container_id: float(r.us) + for r in db.container_metrics(spark) + .select("container_id", F.unix_micros("start_ts").alias("us")) + .collect() + } + assert rows and all(rows[cid] == expected[cid] for cid in rows), (rows, expected) + + +def test_time_window_expression_and_udf_share_a_solve( + spark: SparkSession, basic_narrow_db: MeasurementDB +): + """In one solve, a TimeWindowExpression tiles the container in the channel time frame + (here relative ms) while a UDF still reads the absolute TIMESTAMP start_ts.""" + db = _timestamp_boundaries_db(basic_narrow_db) + query = db.query + rpm = query.channel(channel_name="Engine RPM") + windows = StatsAggregator( + input_expressions=[rpm], + event_expression=TimeWindowExpression(10_000), + statistics=["mean"], + ).alias("windows") + config = SolverConfig(channel_time_unit="ms", channel_time_origin="container_start") + result = query.select( + windows, rpm.apply(_grab_start_ts, container_metrics=["start_ts"]).alias("start") + ).solve(spark, solver=DefaultSolver(spark, config=config)) + rows = {row.container_id: row for row in result.collect()} + + durations = { + r.container_id: float(r.d) + for r in basic_narrow_db.container_metrics(spark) + .select("container_id", (F.col("stop_ts") - F.col("start_ts")).alias("d")) + .collect() + } + assert rows + for container_id, row in rows.items(): + event_timestamps = row.windows.event_timestamps + # Relative windows: from 0 to the container's duration in ms, 10 s apart. + assert event_timestamps[0] == [0.0, 10_000.0] + assert event_timestamps[-1][1] == durations[container_id] + assert len(event_timestamps) == -(-durations[container_id] // 10_000) + assert row.start > 0 # absolute epoch microseconds, not -1 (non-timestamp) diff --git a/tests/impulse_query_engine/unit/analyze/query/solvers/utils/window_bounds_test.py b/tests/impulse_query_engine/unit/analyze/query/solvers/utils/window_bounds_test.py new file mode 100644 index 00000000..1254833e --- /dev/null +++ b/tests/impulse_query_engine/unit/analyze/query/solvers/utils/window_bounds_test.py @@ -0,0 +1,291 @@ +# pylint: disable=missing-function-docstring, redefined-outer-name +"""Tests for solvers.utils.window_bounds.with_window_bounds. + +``TimeWindowEvent`` windows are computed in the channel time frame +(``channel_time_unit`` / ``channel_time_origin``). ``with_window_bounds`` derives the container +start/stop in that frame as two extra columns and leaves the raw ``start_ts`` / ``stop_ts`` +untouched for everyone else (UDFs, ``ContainerEvent``, ``measurement_dimension``). + +All frames are the ``basic_narrow_db`` fixture's ``container_metrics`` boundaries (epoch-ms +longs), recast per test; container 2's boundaries are nulled to cover missing values. +""" + +import pyspark.sql.functions as F +import pyspark.sql.types as T +import pytest +from pyspark.sql import DataFrame, SparkSession + +from impulse_query_engine.analyze.query.solvers.solver_config import SolverConfig +from impulse_query_engine.analyze.query.solvers.utils.window_bounds import with_window_bounds +from impulse_query_engine.measurement_db import MeasurementDB +from tests.conftest import basic_narrow_db, spark # noqa: F401 (pytest fixtures) + +_START, _STOP = "__window_start", "__window_stop" +_NULL_CONTAINER = 2 + + +def _ms_boundaries(spark: SparkSession, db: MeasurementDB) -> DataFrame: # noqa: F811 + """The fixture's container boundaries (epoch-ms longs), container 2's set to null.""" + + def unless_null_container(name: str): + return F.when(F.col("container_id") != _NULL_CONTAINER, F.col(name)).alias(name) + + return db.container_metrics(spark).select( + "container_id", unless_null_container("start_ts"), unless_null_container("stop_ts") + ) + + +def _recast(df: DataFrame, cast) -> DataFrame: + """*df* with ``start_ts`` / ``stop_ts`` passed through *cast* (a Column -> Column).""" + return df.select( + "container_id", + cast(F.col("start_ts")).alias("start_ts"), + cast(F.col("stop_ts")).alias("stop_ts"), + ) + + +def _timestamp_boundaries(spark: SparkSession, db: MeasurementDB) -> DataFrame: # noqa: F811 + return _recast(_ms_boundaries(spark, db), F.timestamp_millis) + + +def _raw_ms(spark: SparkSession, db: MeasurementDB) -> dict: # noqa: F811 + """``{container_id: (start_ms, stop_ms)}`` of the containers with boundaries.""" + return { + r.container_id: (r.start_ts, r.stop_ts) + for r in _ms_boundaries(spark, db).collect() + if r.start_ts is not None + } + + +def _bounds(cfg: SolverConfig, df: DataFrame) -> dict: + out = with_window_bounds(df, cfg) + return {r.container_id: (r[_START], r[_STOP]) for r in out.collect()} + + +def test_window_bound_column_names(): + cfg = SolverConfig() + assert (cfg.window_start_col, cfg.window_stop_col) == (_START, _STOP) + + +@pytest.mark.parametrize( + "unit, expected_type, from_micros", + [ + ("s", T.DoubleType(), lambda us: us / 1e6), + ("ms", T.DoubleType(), lambda us: us / 1e3), + ("us", T.LongType(), lambda us: us), + ("ns", T.LongType(), lambda us: us * 1000), + ], +) +@pytest.mark.parametrize("session_tz", ["UTC", "Europe/Berlin"]) +def test_epoch_origin_converts_timestamps_to_unit( + spark, basic_narrow_db, unit, expected_type, from_micros, session_tz # noqa: F811 +): + previous_tz = spark.conf.get("spark.sql.session.timeZone") + spark.conf.set("spark.sql.session.timeZone", session_tz) + try: + df = _timestamp_boundaries(spark, basic_narrow_db) + out = with_window_bounds(df, SolverConfig(channel_time_unit=unit)) + bounds = {r.container_id: (r[_START], r[_STOP]) for r in out.collect()} + finally: + spark.conf.set("spark.sql.session.timeZone", previous_tz) + + assert out.schema[_START].dataType == expected_type + # Exact and independent of the session time zone. + for cid, (start_ms, stop_ms) in _raw_ms(spark, basic_narrow_db).items(): + assert bounds[cid] == (from_micros(start_ms * 1000), from_micros(stop_ms * 1000)) + assert bounds[_NULL_CONTAINER] == (None, None) + + +def test_epoch_seconds_match_spark_cast_to_double(spark, basic_narrow_db): # noqa: F811 + # "s" equals Spark's cast(timestamp as double), bit for bit. + df = _timestamp_boundaries(spark, basic_narrow_db) + casted = { + r.container_id: (r.start_ts, r.stop_ts) + for r in _recast(df, lambda c: c.cast("double")).collect() + } + assert _bounds(SolverConfig(channel_time_unit="s"), df) == casted + + +@pytest.mark.parametrize( + "unit, from_micros", + [("s", lambda us: us / 1e6), ("ms", lambda us: us / 1e3), ("us", lambda us: us)], +) +@pytest.mark.parametrize("session_tz", ["UTC", "Europe/Berlin"]) +def test_container_start_origin_gives_relative_bounds( + spark, basic_narrow_db, unit, from_micros, session_tz # noqa: F811 +): + previous_tz = spark.conf.get("spark.sql.session.timeZone") + spark.conf.set("spark.sql.session.timeZone", session_tz) + try: + cfg = SolverConfig(channel_time_unit=unit, channel_time_origin="container_start") + bounds = _bounds(cfg, _timestamp_boundaries(spark, basic_narrow_db)) + finally: + spark.conf.set("spark.sql.session.timeZone", previous_tz) + + for cid, (start_ms, stop_ms) in _raw_ms(spark, basic_narrow_db).items(): + assert bounds[cid] == (0, from_micros((stop_ms - start_ms) * 1000)) + # A null boundary leaves a null stop bound, so the container gets no windows. + assert bounds[_NULL_CONTAINER][1] is None + + +def test_numeric_boundaries_epoch_as_is_and_container_start_shifted( + spark, basic_narrow_db # noqa: F811 +): + df = _ms_boundaries(spark, basic_narrow_db) + raw = _raw_ms(spark, basic_narrow_db) + epoch = _bounds(SolverConfig(), df) + assert all(epoch[cid] == bounds for cid, bounds in raw.items()) + # Shift only: numeric boundaries are already in the channels' unit, so no unit is needed. + relative = _bounds(SolverConfig(channel_time_origin="container_start"), df) + assert all(relative[cid] == (0, stop - start) for cid, (start, stop) in raw.items()) + + +def test_numeric_ms_boundaries_converted_to_finer_channel_unit_exactly( + spark, basic_narrow_db # noqa: F811 +): + # Boundaries in epoch ms, channels in µs: an integer factor keeps the longs exact. + df = _ms_boundaries(spark, basic_narrow_db) + raw = _raw_ms(spark, basic_narrow_db) + cfg = SolverConfig(channel_time_unit="us", container_time_unit="ms") + assert with_window_bounds(df, cfg).schema[_START].dataType == T.LongType() + bounds = _bounds(cfg, df) + assert all(bounds[cid] == (start * 1000, stop * 1000) for cid, (start, stop) in raw.items()) + + relative = SolverConfig( + channel_time_unit="us", channel_time_origin="container_start", container_time_unit="ms" + ) + # The difference is taken in ms first, then converted. + bounds = _bounds(relative, df) + assert all(bounds[cid] == (0, (stop - start) * 1000) for cid, (start, stop) in raw.items()) + + +@pytest.mark.parametrize("ansi", ["true", "false"]) +def test_int_boundaries_widen_to_long_instead_of_overflowing( + spark, basic_narrow_db, ansi # noqa: F811 +): + """INT epoch seconds * 1000 exceeds int32. Spark keeps int * int as int, which raised + ARITHMETIC_OVERFLOW under ANSI and silently wrapped to negative bounds without it.""" + int_seconds = _recast( + _ms_boundaries(spark, basic_narrow_db), lambda c: (c / F.lit(1000)).cast("int") + ) + raw_s = { + cid: (start // 1000, stop // 1000) + for cid, (start, stop) in _raw_ms(spark, basic_narrow_db).items() + } + previous_ansi = spark.conf.get("spark.sql.ansi.enabled") + spark.conf.set("spark.sql.ansi.enabled", ansi) + try: + assert int_seconds.schema["start_ts"].dataType == T.IntegerType() + cfg = SolverConfig(channel_time_unit="ms", container_time_unit="s") + assert with_window_bounds(int_seconds, cfg).schema[_START].dataType == T.LongType() + bounds = _bounds(cfg, int_seconds) + assert all(bounds[cid] == (s * 1000, e * 1000) for cid, (s, e) in raw_s.items()) + + relative = SolverConfig( + channel_time_unit="ms", channel_time_origin="container_start", container_time_unit="s" + ) + bounds = _bounds(relative, int_seconds) + assert all(bounds[cid] == (0, (e - s) * 1000) for cid, (s, e) in raw_s.items()) + finally: + spark.conf.set("spark.sql.ansi.enabled", previous_ansi) + + +def test_double_boundaries_keep_fractions_when_converted_to_finer_unit( + spark, basic_narrow_db # noqa: F811 +): + # Seconds as doubles (the fixture's ms values / 1000, so with a fractional part). + seconds = _recast(_ms_boundaries(spark, basic_narrow_db), lambda c: c / F.lit(1000.0)) + cfg = SolverConfig(channel_time_unit="ms", container_time_unit="s") + assert with_window_bounds(seconds, cfg).schema[_START].dataType == T.DoubleType() + bounds = _bounds(cfg, seconds) + for cid, (start_ms, stop_ms) in _raw_ms(spark, basic_narrow_db).items(): + assert bounds[cid] == ((start_ms / 1000.0) * 1000, (stop_ms / 1000.0) * 1000) + # Not truncated to whole seconds before the conversion. + assert bounds[cid][0] != (start_ms // 1000) * 1000 + + +def test_numeric_boundaries_converted_to_coarser_channel_unit( + spark, basic_narrow_db # noqa: F811 +): + # Boundaries in epoch µs, channels in ms: a division, giving doubles. + micros = _recast(_ms_boundaries(spark, basic_narrow_db), lambda c: c * F.lit(1000)) + cfg = SolverConfig(channel_time_unit="ms", container_time_unit="us") + assert with_window_bounds(micros, cfg).schema[_START].dataType == T.DoubleType() + bounds = _bounds(cfg, micros) + for cid, (start_ms, stop_ms) in _raw_ms(spark, basic_narrow_db).items(): + assert bounds[cid] == (float(start_ms), float(stop_ms)) + + +def test_numeric_boundaries_unchanged_for_equal_or_unset_container_unit( + spark, basic_narrow_db # noqa: F811 +): + df = _ms_boundaries(spark, basic_narrow_db) + raw = {r.container_id: (r.start_ts, r.stop_ts) for r in df.collect()} + assert _bounds(SolverConfig(channel_time_unit="ms", container_time_unit="ms"), df) == raw + assert _bounds(SolverConfig(channel_time_unit="us"), df) == raw + + +def test_container_time_unit_rejected_for_timestamp_boundaries( + spark, basic_narrow_db # noqa: F811 +): + cfg = SolverConfig(channel_time_unit="s", container_time_unit="ms") + with pytest.raises(ValueError, match="container_time_unit only applies to numeric"): + with_window_bounds(_timestamp_boundaries(spark, basic_narrow_db), cfg) + + +def test_container_time_unit_requires_channel_time_unit(): + with pytest.raises(ValueError, match="container_time_unit requires channel_time_unit"): + SolverConfig.model_validate({"container_time_unit": "ms"}) + cfg = SolverConfig.model_validate({"channel_time_unit": "us", "container_time_unit": "ms"}) + assert (cfg.channel_time_unit, cfg.container_time_unit) == ("us", "ms") + + +def test_raw_boundaries_stay_unchanged(spark, basic_narrow_db): # noqa: F811 + df = _timestamp_boundaries(spark, basic_narrow_db) + cfg = SolverConfig(channel_time_unit="s", channel_time_origin="container_start") + out = with_window_bounds(df, cfg) + assert isinstance(out.schema["start_ts"].dataType, T.TimestampType) + assert out.select("container_id", "start_ts", "stop_ts").collect() == df.collect() + + +def test_timestamp_boundaries_without_unit_rejected(spark, basic_narrow_db): # noqa: F811 + for origin in ("epoch", "container_start"): + with pytest.raises(ValueError, match=r"TimeWindowEvent.*channel_time_unit"): + with_window_bounds( + _timestamp_boundaries(spark, basic_narrow_db), + SolverConfig(channel_time_origin=origin), + ) + + +@pytest.mark.parametrize("zone_less_type", ["timestamp_ntz", "date"]) +def test_zone_less_types_rejected(spark, basic_narrow_db, zone_less_type): # noqa: F811 + df = _recast(_timestamp_boundaries(spark, basic_narrow_db), lambda c: c.cast(zone_less_type)) + with pytest.raises(ValueError, match="start_ts"): + with_window_bounds(df, SolverConfig(channel_time_unit="s")) + + +def test_mixed_and_missing_boundaries_rejected(spark, basic_narrow_db): # noqa: F811 + timestamps = _timestamp_boundaries(spark, basic_narrow_db) + mixed = timestamps.withColumn("stop_ts", F.unix_millis("stop_ts")) + with pytest.raises(ValueError, match="both be TIMESTAMP or both be numeric"): + with_window_bounds(mixed, SolverConfig(channel_time_unit="s")) + with pytest.raises(ValueError, match="stop_ts"): + with_window_bounds(timestamps.drop("stop_ts"), SolverConfig()) + # An unmapped physical name: the error lists it and points to the column mapping. + unmapped = timestamps.withColumnRenamed("stop_ts", "measurement_end") + with pytest.raises( + ValueError, match=r"'measurement_end'.*container_metrics\.column_name_mapping" + ): + with_window_bounds(unmapped, SolverConfig(channel_time_unit="s")) + + +def test_channel_time_settings_validated(): + cfg = SolverConfig.model_validate( + {"channel_time_unit": "ns", "channel_time_origin": "container_start"} + ) + assert (cfg.channel_time_unit, cfg.channel_time_origin) == ("ns", "container_start") + assert SolverConfig().channel_time_origin == "epoch" + with pytest.raises(ValueError): + SolverConfig.model_validate({"channel_time_unit": "minutes"}) + with pytest.raises(ValueError): + SolverConfig.model_validate({"channel_time_origin": "recording_start"}) diff --git a/tests/impulse_reporting/integration/time_window_event_test.py b/tests/impulse_reporting/integration/time_window_event_test.py new file mode 100644 index 00000000..cee850e5 --- /dev/null +++ b/tests/impulse_reporting/integration/time_window_event_test.py @@ -0,0 +1,967 @@ +"""Integration tests for TimeWindowEvent with end-to-end Report usage.""" + +import math +from unittest.mock import create_autospec + +import pandas as pd +import pyspark.sql.functions as F +import pyspark.sql.types as T +import pytest +from databricks.sdk import WorkspaceClient +from pyspark.sql import Window + +from impulse_query_engine.analyze.query.solvers.solver_config import RawEncoder, SolverConfig +from impulse_reporting.aggregations.stats_aggregator import StatsAggregator +from impulse_reporting.config.config_parser import ( + Comparator, + ContainerFilters, + DataType, + ImpulseConfig, + IncrementalConfig, + MetricFilter, + QueryEngine, + Solvers, + Source, + UnitySink, +) +from impulse_reporting.core.page import Page +from impulse_reporting.core.report import Report +from impulse_reporting.events.container_event import ContainerEvent +from impulse_reporting.events.time_window_event import TimeWindowEvent +from tests.conftest import setup_basic_db, spark # noqa: F401 (pytest fixtures) + +# Container boundaries (epoch ms) for the Seat_Leon measurements in +# container_metrics.csv, as documented in container_event_test.py. +# c1: span 107545 ms, c2: 108752 ms, c3: 110083 ms +EXPECTED_CONTAINERS = { + 1: {"start_ts": 1751528502708, "stop_ts": 1751528610253}, + 2: {"start_ts": 1751528501483, "stop_ts": 1751528610235}, + 3: {"start_ts": 1751528500169, "stop_ts": 1751528610252}, +} +WINDOW_LENGTH = 10_000 # 10 seconds, in epoch-ms units + + +def _config(table_prefix: str) -> ImpulseConfig: + return ImpulseConfig( + source=Source( + container_metrics_table="spark_catalog.silver.container_metrics", + channel_metrics_table="spark_catalog.silver.channel_metrics", + channels_uri="spark_catalog.silver.channels", + ), + unity_sink=UnitySink( + catalog="spark_catalog", + schema="gold", + table_prefix=table_prefix, + ), + container_filters=ContainerFilters( + metric_filters=[ + [ + MetricFilter( + column_name="vehicle_key", comparator=Comparator.EQ, value="Seat_Leon" + ), + MetricFilter( + column_name="start_dt", + comparator=Comparator.GE, + value="2025-07-03T07:00:00.000Z", + ), + ] + ] + ), + query_engine=QueryEngine(solver=Solvers.KEY_VALUE_STORE_SOLVER), + measurement_dimensions=["container_id", "start_ts", "stop_ts"], + ) + + +def _expected_window_count(container_id: int) -> int: + span = ( + EXPECTED_CONTAINERS[container_id]["stop_ts"] + - EXPECTED_CONTAINERS[container_id]["start_ts"] + ) + return -(-span // WINDOW_LENGTH) # ceil division + + +def test_time_window_event_in_report(spark, basic_narrow_db): + """A TimeWindowEvent registered on a report tiles each container exactly into windows, + with one id per window, and writes its event_dimension row. Per-window statistics are + covered by test_time_window_event_aggregation_join (the basic fixture's container + boundaries don't overlap its samples, so they would all be NaN here).""" + my_report = Report( + name="time_window_event_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=dict(_config("time_window_event_test")), + ) + + window_evt = TimeWindowEvent( + name="ten_sec", window_length=WINDOW_LENGTH, desc="Ten second windows" + ) + my_report.add_event(window_evt) + + my_report.determine_report() + + event_dfs = my_report.event_dfs + assert "TIME_WINDOW_EVENT" in event_dfs + rows = event_dfs["TIME_WINDOW_EVENT"]["changed"].collect() + + total_expected = sum(_expected_window_count(cid) for cid in EXPECTED_CONTAINERS) + assert len(rows) == total_expected + + _assert_windows_for_all_containers(rows) + for container_id in EXPECTED_CONTAINERS: + # Per-window instances are distinct (unlike ContainerEvent's single id). + instance_ids = [r.event_instance_id for r in rows if r.container_id == container_id] + assert len(set(instance_ids)) == len(instance_ids) + + dim_rows = my_report.event_metadata_dfs["TIME_WINDOW_EVENT"].collect() + assert len(dim_rows) == 1 + assert dim_rows[0].event_type == "TIME_WINDOW_EVENT" + assert dim_rows[0].attributes["window_length"] == str(float(WINDOW_LENGTH)) + + +# Window length (channel-sample time unit, µs) for the aligned-boundary test. +# Per-container sample spans are ~3.9–5.4e9 µs, so 600s (=6e8 µs) yields several +# windows per container, each overlapping RPM samples. +ALIGNED_WINDOW_LENGTH = 600_000_000 +_ALIGNED_SCHEMA = "spark_catalog.silver_tw_aligned" + + +# Customer-shaped time bases for the id-join test, all derived from the basic db's µs epochs. +# Each entry: (transform for channel tstart/tend, transform for container start_ts/stop_ts, +# window length in the samples' unit, SolverConfig channel time settings). +# us: the native µs epochs (< 2^53, every boundary exactly representable). +# ns: ns epochs (~1.5e18, beyond 2^53) with a window that is NOT a multiple of the +# 256 ns double spacing there, so the boundaries round. +# sec: seconds as doubles with a fractional window, so the boundaries round. +# sec_ts: samples as seconds-as-double, container boundaries as TIMESTAMP (windows in +# epoch seconds via channel_time_unit="s"). +# us_ts: native µs samples, container boundaries as TIMESTAMP (windows in epoch µs via +# channel_time_unit="us", the long path of the conversion). +# rel_sec: samples as seconds since the container start (double), container boundaries as +# TIMESTAMP (windows from 0 via channel_time_origin="container_start"). +# ms_bounds: native µs samples, container boundaries as epoch-ms longs (windows in µs via +# container_time_unit="ms"); without the conversion no window overlaps a sample. +def _to_seconds(c): + return c.cast("double") / F.lit(1e6) + + +def _to_ns(c): + return c.cast("long") * F.lit(1000) + + +def _to_timestamp(c): + return F.timestamp_micros(c.cast("long")) + + +def _to_ms(c): + return F.floor(c.cast("long") / F.lit(1000)).cast("long") + + +_RELATIVE_SECONDS = {"channel_time_unit": "s", "channel_time_origin": "container_start"} + +_TIME_BASES = { + "us": (lambda c: c, lambda c: c, ALIGNED_WINDOW_LENGTH, {}), + "ns": (_to_ns, _to_ns, 600_000_000_007, {}), + "sec": (_to_seconds, _to_seconds, 600.3, {}), + "sec_ts": (_to_seconds, _to_timestamp, 600.3, {"channel_time_unit": "s"}), + "us_ts": (lambda c: c, _to_timestamp, ALIGNED_WINDOW_LENGTH, {"channel_time_unit": "us"}), + "rel_sec": (_to_seconds, _to_timestamp, 600.3, _RELATIVE_SECONDS), + "ms_bounds": ( + lambda c: c, + _to_ms, + ALIGNED_WINDOW_LENGTH, + {"channel_time_unit": "us", "container_time_unit": "ms"}, + ), +} + + +def _clone_aligned_silver( + spark, + schema: str, + to_time_base=lambda c: c, + boundaries_to_time_base=None, + relative_channels: bool = False, +) -> None: + """Clone the basic silver tables into *schema* with container_metrics start_ts / stop_ts + recomputed from each container's channel-sample range (so the container boundaries, + and thus the windows, share the samples' time base). Channel timestamps are then mapped + through *to_time_base* (after subtracting the container start when *relative_channels*) + and the container boundaries through *boundaries_to_time_base* (default: the same + transform).""" + boundaries_to_time_base = boundaries_to_time_base or to_time_base + spark.sql(f"CREATE SCHEMA IF NOT EXISTS {schema}") + channels = spark.read.table("spark_catalog.silver.channels") + bounds = channels.groupBy("container_id").agg( + F.min("tstart").alias("_agg_start"), + F.max("tend").alias("_agg_stop"), + ) + container_metrics = spark.read.table("spark_catalog.silver.container_metrics") + start_type = container_metrics.schema["start_ts"].dataType + stop_type = container_metrics.schema["stop_ts"].dataType + aligned_cm = ( + container_metrics.join(bounds, on="container_id", how="left") + .withColumn("start_ts", F.coalesce("_agg_start", "start_ts").cast(start_type)) + .withColumn("stop_ts", F.coalesce("_agg_stop", "stop_ts").cast(stop_type)) + .drop("_agg_start", "_agg_stop") + .withColumn("start_ts", boundaries_to_time_base(F.col("start_ts"))) + .withColumn("stop_ts", boundaries_to_time_base(F.col("stop_ts"))) + ) + aligned_cm.write.format("delta").mode("overwrite").option( + "overwriteSchema", "true" + ).saveAsTable(f"{schema}.container_metrics") + if relative_channels: + channels = ( + channels.join(bounds, on="container_id") + .withColumn("tstart", F.col("tstart") - F.col("_agg_start")) + .withColumn("tend", F.col("tend") - F.col("_agg_start")) + .drop("_agg_start", "_agg_stop") + ) + channels.withColumn("tstart", to_time_base(F.col("tstart"))).withColumn( + "tend", to_time_base(F.col("tend")) + ).write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable( + f"{schema}.channels" + ) + spark.read.table("spark_catalog.silver.channel_metrics").write.format("delta").mode( + "overwrite" + ).saveAsTable(f"{schema}.channel_metrics") + + +@pytest.fixture +def setup_tw_aligned_db(spark, setup_basic_db, request): # noqa: F811 + """Aligned silver clone in the time base given by ``request.param`` (default ``us``). + + Yields ``(schema, window_length, channel_time)``, where *channel_time* holds the + SolverConfig channel time settings for that time base. + """ + time_base = getattr(request, "param", "us") + to_time_base, boundaries_to_time_base, window_length, channel_time = _TIME_BASES[time_base] + schema = f"{_ALIGNED_SCHEMA}_{time_base}" + _clone_aligned_silver( + spark, + schema, + to_time_base, + boundaries_to_time_base, + relative_channels=channel_time.get("channel_time_origin") == "container_start", + ) + yield schema, window_length, channel_time + spark.sql(f"DROP SCHEMA IF EXISTS {schema} CASCADE") + + +def _aligned_config( + schema: str, + table_prefix: str, + channel_time: dict | None = None, + raw_encoder: RawEncoder | None = None, + channels_table: str = "channels", + **extra, +) -> dict: + """Report config over the aligned clone; a *raw_encoder* switches to ``data_type=RAW``.""" + return dict( + ImpulseConfig( + source=Source( + container_metrics_table=f"{schema}.container_metrics", + channel_metrics_table=f"{schema}.channel_metrics", + channels_uri=f"{schema}.{channels_table}", + ), + unity_sink=UnitySink( + catalog="spark_catalog", schema="gold", table_prefix=table_prefix + ), + container_filters=ContainerFilters( + metric_filters=[ + [ + MetricFilter( + column_name="vehicle_key", comparator=Comparator.EQ, value="Seat_Leon" + ) + ] + ] + ), + query_engine=QueryEngine( + solver=Solvers.KEY_VALUE_STORE_SOLVER, + solver_config=SolverConfig(**channel_time) if channel_time else None, + data_type=DataType.RAW if raw_encoder else DataType.RLE, + raw_encoder=raw_encoder, + ), + measurement_dimensions=["container_id", "start_ts", "stop_ts"], + **extra, + ) + ) + + +def _rpm_stats( + report: Report, + event: TimeWindowEvent, + statistics=("min", "max", "mean"), + name: str = "rpm_stats_per_window", +): + query = report.get_db().query + return StatsAggregator( + name=name, + input_expressions=[query.channel(channel_name="Engine RPM")], + channel_names=["Engine RPM"], + statistics=list(statistics), + event=event, + desc="Engine RPM stats per window", + ) + + +def _assert_ids_join(spark, table_prefix: str) -> tuple[set, set]: # noqa: F811 + """Assert every stats event_instance_id exists in event_instance_fact, with real values. + + Returns ``(stats_event_ids, event_ids)`` for further checks. + """ + stats_fact = spark.read.table(f"spark_catalog.gold.{table_prefix}_stats_aggregator_fact") + event_instance_fact = spark.read.table( + f"spark_catalog.gold.{table_prefix}_event_instance_fact" + ) + + # Real computed values: with aligned boundaries every window overlaps RPM samples, + # so the windows produce a positive max. + max_values = [ + r.statistic_value + for r in stats_fact.filter(F.col("aggregation_label") == "max").collect() + if r.statistic_value is not None + ] + assert len(max_values) > 0 + assert any(v > 0 for v in max_values) + + stats_event_ids = { + r.event_instance_id + for r in stats_fact.filter(F.col("event_instance_id").isNotNull()) + .select("event_instance_id") + .distinct() + .collect() + } + event_ids = { + r.event_instance_id + for r in event_instance_fact.select("event_instance_id").distinct().collect() + } + assert len(stats_event_ids) > 0 + # Every per-window stats instance must map to a materialized window instance. + assert stats_event_ids.issubset( + event_ids + ), f"stats event_instance_ids not in event_instance_fact: {stats_event_ids - event_ids}" + return stats_event_ids, event_ids + + +def _assert_window_stats_match_samples( + spark, schema: str, table_prefix: str, channels_table: str = "channels" # noqa: F811 +): + """Each window's RPM min / max equal those of the silver samples overlapping that window, + and windows without RPM samples carry no value (RPM only covers each container's first + minute, so most windows are empty). + + This pins every stats row to the window whose boundaries event_instance_fact stores, so + windows that differed between the event fact and the solve would show up here. + """ + rpm_channels = ( + spark.read.table(f"{schema}.channel_metrics") + .filter(F.col("channel_name") == "Engine RPM") + .select("container_id", "channel_id") + ) + channels = spark.read.table(f"{schema}.{channels_table}") + if "tend" not in channels.columns: + # RAW points: each sample is valid until the next one, the last one only at its own + # timestamp (the documented raw->interval rule of both encoders). + by_time = Window.partitionBy("container_id", "channel_id").orderBy("tstart") + channels = channels.withColumnRenamed("timestamp", "tstart").withColumn( + "tend", F.coalesce(F.lead("tstart").over(by_time), F.col("tstart")) + ) + samples = channels.join(rpm_channels, ["container_id", "channel_id"]).select( + "container_id", + F.col("tstart").cast("double").alias("tstart"), + F.col("tend").cast("double").alias("tend"), + F.col("value").cast("double").alias("value"), + ) + windows = spark.read.table(f"spark_catalog.gold.{table_prefix}_event_instance_fact") + expected = ( + windows.join(samples, "container_id") + .filter((F.col("tstart") < F.col("end_ts")) & (F.col("tend") > F.col("start_ts"))) + .groupBy("container_id", "event_instance_id") + .agg(F.min("value").alias("expected_min"), F.max("value").alias("expected_max")) + ) + actual = ( + spark.read.table(f"spark_catalog.gold.{table_prefix}_stats_aggregator_fact") + .groupBy("container_id", "event_instance_id") + .pivot("aggregation_label", ["min", "max"]) + .agg(F.first("statistic_value")) + ) + rows = actual.join(expected, ["container_id", "event_instance_id"], "left").collect() + + def _is_missing(value) -> bool: + return value is None or math.isnan(value) + + with_samples = [r for r in rows if r.expected_max is not None] + assert with_samples and len(with_samples) < len(rows) + mismatches = [ + r + for r in rows + if ( + (r["min"], r["max"]) != (r.expected_min, r.expected_max) + if r.expected_max is not None + else not (_is_missing(r["min"]) and _is_missing(r["max"])) + ) + ] + assert not mismatches, mismatches[:5] + + +@pytest.mark.parametrize( + "setup_tw_aligned_db", + ["us", "ns", "sec", "sec_ts", "us_ts", "rel_sec", "ms_bounds"], + indirect=True, +) +def test_time_window_event_aggregation_join(spark, setup_tw_aligned_db): + """Stats scoped to a TimeWindowEvent yield per-window values whose event_instance_id + joins to the natively computed event fact, for µs, ns and seconds-as-double time bases, + for TIMESTAMP container boundaries (channel_time_unit), for channel timestamps relative + to the container start (channel_time_origin="container_start"), and for numeric + boundaries in another unit than the channels (container_time_unit).""" + schema, window_length, channel_time = setup_tw_aligned_db + table_prefix = f"time_window_join_test_{schema.removeprefix(_ALIGNED_SCHEMA + '_')}" + my_report = Report( + name="time_window_join_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=_aligned_config(schema, table_prefix, channel_time=channel_time), + ) + + window_evt = TimeWindowEvent(name="ten_min", window_length=window_length) + my_report.add_event(window_evt) + + page = Page(page_number=1) + my_report.add_page(page) + page.add_aggregation(_rpm_stats(my_report, window_evt)) + + my_report.determine_report() + my_report.persist_results() + + _assert_ids_join(spark, table_prefix) + _assert_window_stats_match_samples(spark, schema, table_prefix) + if channel_time.get("channel_time_origin") == "container_start": + _assert_windows_start_at_zero(spark, table_prefix) + + +def _assert_windows_start_at_zero(spark, table_prefix: str) -> None: # noqa: F811 + """Relative channel time: every container's first window starts at 0.""" + first = ( + spark.read.table(f"spark_catalog.gold.{table_prefix}_event_instance_fact") + .groupBy("container_id") + .agg(F.min("start_ts").alias("first")) + .collect() + ) + assert first and all(r.first == 0.0 for r in first), first + + +def _write_raw_channels(spark, schema: str) -> str: # noqa: F811 + """Write the aligned channels in the raw format (one ``timestamp`` per sample, no + ``tend``) next to the RLE table, and return the new table's name.""" + table = "channels_raw" + spark.read.table(f"{schema}.channels").select( + "container_id", "channel_id", F.col("tstart").alias("timestamp"), "value" + ).write.format("delta").mode("overwrite").saveAsTable(f"{schema}.{table}") + return table + + +@pytest.mark.parametrize("raw_encoder", [RawEncoder.RLE, RawEncoder.INTERVAL]) +@pytest.mark.parametrize("setup_tw_aligned_db", ["us", "us_ts"], indirect=True) +def test_time_window_event_aggregation_join_raw(spark, setup_tw_aligned_db, raw_encoder): + """With data_type=RAW both encoders derive [tstart, tend) from the raw ``timestamp`` + column without changing its unit, so windows over numeric or TIMESTAMP + (channel_time_unit="us") container boundaries line up with the samples exactly as for + RLE silver data.""" + schema, window_length, channel_time = setup_tw_aligned_db + channels_table = _write_raw_channels(spark, schema) + time_base = schema.removeprefix(_ALIGNED_SCHEMA + "_") + table_prefix = f"time_window_raw_test_{time_base}_{raw_encoder.value.lower()}" + my_report = Report( + name="time_window_raw_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=_aligned_config( + schema, + table_prefix, + channel_time=channel_time, + raw_encoder=raw_encoder, + channels_table=channels_table, + ), + ) + + window_evt = TimeWindowEvent(name="ten_min", window_length=window_length) + my_report.add_event(window_evt) + page = Page(page_number=1) + my_report.add_page(page) + page.add_aggregation(_rpm_stats(my_report, window_evt)) + + my_report.determine_report() + my_report.persist_results() + + _assert_ids_join(spark, table_prefix) + _assert_window_stats_match_samples(spark, schema, table_prefix, channels_table) + + +def _container_boundaries(spark, schema: str) -> dict: # noqa: F811 + """``{container_id: (start_ts, stop_ts)}`` as doubles, for the containers in the report's + scope (``_aligned_config`` filters on ``vehicle_key == "Seat_Leon"``).""" + return { + r.container_id: (float(r.start_ts), float(r.stop_ts)) + for r in spark.read.table(f"{schema}.container_metrics") + .filter(F.col("vehicle_key") == "Seat_Leon") + .select("container_id", "start_ts", "stop_ts") + .collect() + } + + +def _assert_windows_tile_containers(rows, boundaries: dict, window_length: float) -> None: + """Each container's windows tile its ``[start_ts, stop_ts]`` exactly: window ``i`` spans + ``[start + i * W, min(start + (i + 1) * W, stop)]``, so the windows start at + ``start_ts``, are contiguous and the last one is clamped to ``stop_ts``. + + The expected boundaries use the same double arithmetic as the event, so the comparison + is exact for every time base (ns epochs, fractional windows). A container without a + positive span expects no windows. + """ + for container_id, (start, stop) in boundaries.items(): + windows = sorted((r.start_ts, r.end_ts) for r in rows if r.container_id == container_id) + count = math.ceil((stop - start) / window_length) if stop > start else 0 + expected = [ + (start + i * window_length, min(start + (i + 1) * window_length, stop)) + for i in range(count) + ] + expected = [(s, e) for s, e in expected if s < e] + assert windows == expected, (container_id, windows[:3], expected[:3]) + + +def test_multiple_time_window_events_coexist(spark, setup_tw_aligned_db): + """Two TimeWindowEvents with different window lengths coexist in one report: each tiles + every container with its own windows, and the statistics scoped to each event carry + the values of that event's windows. The ids include the event name, so a window of one + event never joins the other event's windows, even where their boundaries coincide.""" + schema, window_length, _ = setup_tw_aligned_db + table_prefix = "time_window_multi_test" + my_report = Report( + name="time_window_multi_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=_aligned_config(schema, table_prefix), + ) + + evt_short = TimeWindowEvent(name="ten_min", window_length=window_length) + evt_long = TimeWindowEvent(name="thirty_min", window_length=3 * window_length) + my_report.add_event(evt_short) + my_report.add_event(evt_long) + + page = Page(page_number=1) + my_report.add_page(page) + page.add_aggregation(_rpm_stats(my_report, evt_short, name="rpm_stats_ten_min")) + page.add_aggregation(_rpm_stats(my_report, evt_long, name="rpm_stats_thirty_min")) + + my_report.determine_report() + my_report.persist_results() + + event_fact = spark.read.table(f"spark_catalog.gold.{table_prefix}_event_instance_fact") + event_rows = event_fact.collect() + boundaries = _container_boundaries(spark, schema) + for event, length in ((evt_short, window_length), (evt_long, 3 * window_length)): + _assert_windows_tile_containers( + [r for r in event_rows if r.event_id == event.get_id()], boundaries, length + ) + + # Every stats row joins a window of its own event, with its window's sample values. + _assert_ids_join(spark, table_prefix) + stats_fact = spark.read.table(f"spark_catalog.gold.{table_prefix}_stats_aggregator_fact") + cross_event = ( + stats_fact.select("event_instance_id", F.col("event_id").alias("stats_event_id")) + .join(event_fact.select("event_instance_id", "event_id"), "event_instance_id") + .filter(F.col("stats_event_id") != F.col("event_id")) + ) + assert cross_event.count() == 0 + assert {r.event_id for r in stats_fact.select("event_id").distinct().collect()} == { + evt_short.get_id(), + evt_long.get_id(), + } + _assert_window_stats_match_samples(spark, schema, table_prefix) + + dim_rows = my_report.event_metadata_dfs["TIME_WINDOW_EVENT"].collect() + assert {d.event_name for d in dim_rows} == {"ten_min", "thirty_min"} + + +# --------------------------------------------------------------------------- +# Coverage: windows for every filtered container, independent of channel data +# --------------------------------------------------------------------------- +_PARTIAL_SCHEMA = "spark_catalog.silver_tw_partial" +_RPM_CHANNEL_ID = 5 + + +@pytest.fixture +def setup_tw_partial_db(spark, setup_basic_db): # noqa: F811 + """Basic silver clone where container 3 has no Engine RPM channel (metrics or data).""" + spark.sql(f"CREATE SCHEMA IF NOT EXISTS {_PARTIAL_SCHEMA}") + no_rpm_on_3 = ~((F.col("container_id") == 3) & (F.col("channel_id") == _RPM_CHANNEL_ID)) + spark.read.table("spark_catalog.silver.container_metrics").write.format("delta").mode( + "overwrite" + ).saveAsTable(f"{_PARTIAL_SCHEMA}.container_metrics") + for table in ("channel_metrics", "channels"): + spark.read.table(f"spark_catalog.silver.{table}").filter(no_rpm_on_3).write.format( + "delta" + ).mode("overwrite").saveAsTable(f"{_PARTIAL_SCHEMA}.{table}") + yield + spark.sql(f"DROP SCHEMA IF EXISTS {_PARTIAL_SCHEMA} CASCADE") + + +def _assert_windows_for_all_containers(rows) -> None: + """Every filtered container is tiled into WINDOW_LENGTH windows over its boundaries.""" + boundaries = { + cid: (float(b["start_ts"]), float(b["stop_ts"])) for cid, b in EXPECTED_CONTAINERS.items() + } + _assert_windows_tile_containers(rows, boundaries, WINDOW_LENGTH) + + +def test_time_window_event_covers_containers_without_aggregated_channel( + spark, setup_tw_partial_db +): + """Windows exist for every filtered container even when the scoped aggregation's + channel is missing on some of them and the solve is split into single-channel batches. + + Previously the windows came from the channel solve, so container 3 (no Engine RPM) + got none whenever the window expression landed in the RPM batch.""" + config = _config("time_window_partial_test") + config.source = Source( + container_metrics_table=f"{_PARTIAL_SCHEMA}.container_metrics", + channel_metrics_table=f"{_PARTIAL_SCHEMA}.channel_metrics", + channels_uri=f"{_PARTIAL_SCHEMA}.channels", + ) + config.query_engine = QueryEngine( + solver=Solvers.KEY_VALUE_STORE_SOLVER, max_channels_per_batch=1 + ) + my_report = Report( + name="time_window_partial_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=dict(config), + ) + window_evt = TimeWindowEvent(name="ten_sec", window_length=WINDOW_LENGTH) + my_report.add_event(window_evt) + page = Page(page_number=1) + my_report.add_page(page) + page.add_aggregation(_rpm_stats(my_report, window_evt)) + + my_report.determine_report() + + rows = my_report.event_dfs["TIME_WINDOW_EVENT"]["changed"].collect() + _assert_windows_for_all_containers(rows) + + # The stats cover every window of the containers that have Engine RPM, and none of + # container 3, whose windows exist regardless. + stats_rows = my_report.aggregation_dfs["STATS_AGGREGATOR"]["changed"].collect() + assert {r.container_id for r in stats_rows} == {1, 2} + assert {r.event_instance_id for r in stats_rows} == { + r.event_instance_id for r in rows if r.container_id in (1, 2) + } + + +def test_standalone_time_window_event_covers_all_containers(spark, basic_narrow_db): + """A TimeWindowEvent with no aggregation (nothing to solve) still materializes windows.""" + my_report = Report( + name="time_window_standalone_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=dict(_config("time_window_standalone_test")), + ) + my_report.add_event(TimeWindowEvent(name="ten_sec", window_length=WINDOW_LENGTH)) + + my_report.determine_report() + + rows = my_report.event_dfs["TIME_WINDOW_EVENT"]["changed"].collect() + _assert_windows_for_all_containers(rows) + assert all(r.start_ts < r.end_ts for r in rows) + + +# --------------------------------------------------------------------------- +# Incremental: ids still join when the aggregation and the event use different scopes +# --------------------------------------------------------------------------- +def test_time_window_event_ids_join_after_incremental_run(spark, setup_tw_aligned_db): + """Run 1 (full) on containers 1-2; run 2 (incremental) adds container 3 and changes the + aggregation's definition. The changed aggregation recomputes over all containers while + the unchanged event only computes container 3, yet every stats id must still join.""" + schema, window_length, _ = setup_tw_aligned_db + table_prefix = "time_window_inc_test" + cm_run_1 = f"{schema}.container_metrics_run_1" + cm_run_2 = f"{schema}.container_metrics_run_2" + past = F.lit("2020-01-01 00:00:00").cast("timestamp") + cm = spark.read.table(f"{schema}.container_metrics") + cm.filter(F.col("container_id").isin([1, 2])).withColumn("timestamp", past).write.format( + "delta" + ).mode("overwrite").saveAsTable(cm_run_1) + + def _run(cm_table: str, is_incremental: bool, statistics) -> None: + config = _aligned_config( + schema, + table_prefix, + incremental=IncrementalConfig( + enabled=is_incremental, + silver_last_modified_column="timestamp", + gold_last_modified_column="_created_at", + ), + ) + config["source"].container_metrics_table = cm_table + report = Report( + name="time_window_inc_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=config, + ) + window_evt = TimeWindowEvent(name="ten_min", window_length=window_length) + report.add_event(window_evt) + page = Page(page_number=1) + report.add_page(page) + page.add_aggregation(_rpm_stats(report, window_evt, statistics)) + report.determine_report() + report.persist_results() + + _run(cm_run_1, is_incremental=False, statistics=("min", "max", "mean")) + + # Container 3 is new (recent timestamp); 1 and 2 are unchanged. + cm.withColumn( + "timestamp", F.when(F.col("container_id") == 3, F.current_timestamp()).otherwise(past) + ).write.format("delta").mode("overwrite").saveAsTable(cm_run_2) + # Adding a statistic changes the aggregation's definition hash (event unchanged). + _run(cm_run_2, is_incremental=True, statistics=("min", "max", "mean", "median")) + + stats_event_ids, _ = _assert_ids_join(spark, table_prefix) + + event_fact = spark.read.table(f"spark_catalog.gold.{table_prefix}_event_instance_fact") + stats_fact = spark.read.table(f"spark_catalog.gold.{table_prefix}_stats_aggregator_fact") + assert {r.container_id for r in event_fact.select("container_id").distinct().collect()} == { + 1, + 2, + 3, + } + # The changed aggregation was recomputed for all containers (incl. the new one). + assert {r.container_id for r in stats_fact.select("container_id").distinct().collect()} == { + 1, + 2, + 3, + } + assert stats_fact.filter(F.col("aggregation_label") == "median").count() > 0 + + +# --------------------------------------------------------------------------- +# TIMESTAMP container boundaries: channel_time_unit is required only by TimeWindowEvent +# --------------------------------------------------------------------------- +@pytest.mark.parametrize("setup_tw_aligned_db", ["sec_ts"], indirect=True) +def test_time_window_event_timestamp_boundaries_require_channel_time_unit( + spark, setup_tw_aligned_db +): + """A TimeWindowEvent over TIMESTAMP boundaries without channel_time_unit fails fast and + clearly.""" + schema, window_length, _ = setup_tw_aligned_db + my_report = Report( + name="time_window_no_unit_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=_aligned_config(schema, "time_window_no_unit_test"), + ) + my_report.add_event(TimeWindowEvent(name="ten_min", window_length=window_length)) + + with pytest.raises(ValueError, match=r"TimeWindowEvent.*channel_time_unit"): + my_report.determine_report() + + +@pytest.mark.parametrize( + "channel_time", + [None, {"channel_time_unit": "ms"}, _RELATIVE_SECONDS], + ids=["unset", "ms", "relative_s"], +) +@pytest.mark.parametrize("setup_tw_aligned_db", ["sec_ts"], indirect=True) +def test_container_event_timestamp_boundaries(spark, setup_tw_aligned_db, channel_time): + """A ContainerEvent ignores the channel time settings: over TIMESTAMP boundaries it + always writes the raw boundaries as epoch seconds (as on main). measurement_dimension + keeps the TIMESTAMP type.""" + schema, _, _ = setup_tw_aligned_db + unit = (channel_time or {}).get("channel_time_unit", "unset") + origin = (channel_time or {}).get("channel_time_origin", "epoch") + table_prefix = f"container_event_ts_test_{unit}_{origin}" + my_report = Report( + name="container_event_ts_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=_aligned_config(schema, table_prefix, channel_time=channel_time), + ) + my_report.add_event(ContainerEvent(name="full_container")) + my_report.determine_report() + my_report.persist_results() + + expected = { + r.container_id: (r.s, r.e) + for r in spark.read.table(f"{schema}.container_metrics") + .select( + "container_id", + F.col("start_ts").cast("double").alias("s"), + F.col("stop_ts").cast("double").alias("e"), + ) + .collect() + } + event_fact = spark.read.table(f"spark_catalog.gold.{table_prefix}_event_instance_fact") + actual = {r.container_id: (r.start_ts, r.end_ts) for r in event_fact.collect()} + assert actual and all(actual[cid] == expected[cid] for cid in actual), (actual, expected) + + measurement_dim = spark.read.table(f"spark_catalog.gold.{table_prefix}_measurement_dimension") + assert isinstance(measurement_dim.schema["start_ts"].dataType, T.TimestampType) + + +@pytest.mark.parametrize("setup_tw_aligned_db", ["sec_ts"], indirect=True) +def test_channel_time_change_recomputes_time_window_event_only(spark, setup_tw_aligned_db): + """Changing channel_time_unit between incremental runs moves the definition hashes of the + TimeWindowEvent and the aggregation scoped to its windows, so they recompute over all + containers and the windows never mix units. The ContainerEvent writes the raw boundaries, + so it keeps its hash and its rows.""" + schema, window_length, channel_time = setup_tw_aligned_db + assert channel_time == {"channel_time_unit": "s"} + table_prefix = "time_window_channel_time_test" + cm_run_1 = f"{schema}.container_metrics_run_1" + cm_run_2 = f"{schema}.container_metrics_run_2" + past = F.lit("2020-01-01 00:00:00").cast("timestamp") + cm = spark.read.table(f"{schema}.container_metrics") + cm.filter(F.col("container_id").isin([1, 2])).withColumn("timestamp", past).write.format( + "delta" + ).mode("overwrite").saveAsTable(cm_run_1) + # Container 3 is new in run 2 (recent timestamp); 1 and 2 are unchanged. + cm.withColumn( + "timestamp", F.when(F.col("container_id") == 3, F.current_timestamp()).otherwise(past) + ).write.format("delta").mode("overwrite").saveAsTable(cm_run_2) + + def _run(cm_table: str, unit: str, is_incremental: bool): + config = _aligned_config( + schema, + table_prefix, + channel_time={"channel_time_unit": unit}, + incremental=IncrementalConfig( + enabled=is_incremental, + silver_last_modified_column="timestamp", + gold_last_modified_column="_created_at", + ), + ) + config["source"].container_metrics_table = cm_table + report = Report( + name="time_window_channel_time_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=config, + ) + window_evt = TimeWindowEvent(name="ten_min", window_length=window_length) + container_evt = ContainerEvent(name="full_container") + report.add_event(window_evt) + report.add_event(container_evt) + page = Page(page_number=1) + report.add_page(page) + stats = _rpm_stats(report, window_evt) + page.add_aggregation(stats) + report.determine_report() + report.persist_results() + return report, window_evt, container_evt, stats + + def _event_rows(event_id: int): + return ( + spark.read.table(f"spark_catalog.gold.{table_prefix}_event_instance_fact") + .filter(F.col("event_id") == event_id) + .collect() + ) + + _run(cm_run_1, "s", is_incremental=False) + report, window_evt, container_evt, stats = _run(cm_run_2, "ms", is_incremental=True) + + changed_events = {i for ids in report._changed_event_ids.values() for i in ids} + changed_aggs = {i for ids in report._changed_aggregation_ids.values() for i in ids} + assert window_evt.get_id() in changed_events + assert stats.get_id() in changed_aggs + assert container_evt.get_id() not in changed_events + + boundaries = { + r.container_id: r + for r in cm.select( + "container_id", + F.col("start_ts").cast("double").alias("start_s"), + (F.unix_micros("start_ts") / F.lit(1e3)).alias("start_ms"), + ).collect() + } + # ContainerEvent: the raw boundaries as epoch seconds, for the old and the new containers. + container_rows = _event_rows(container_evt.get_id()) + assert {r.container_id for r in container_rows} == {1, 2, 3} + assert all(r.start_ts == boundaries[r.container_id].start_s for r in container_rows) + + # Every window tiles the container's ms span: none is left over from the seconds run. + window_rows = _event_rows(window_evt.get_id()) + assert {r.container_id for r in window_rows} == {1, 2, 3} + first_window = {} + for r in window_rows: + first_window[r.container_id] = min( + first_window.get(r.container_id, r.start_ts), r.start_ts + ) + assert first_window == {cid: b.start_ms for cid, b in boundaries.items()} + + +def _absolute_start_micros(ts, container_metrics): + """Customer-style UDF: rebuilding absolute time needs the TIMESTAMP start_ts, which + arrives as a pd.Timestamp (naive, in the session time zone). Returns it as a constant + series of epoch microseconds.""" + value = container_metrics["start_ts"] + if value is None: # type-inference pass on the empty cache + return ts * 0 + micros = (value - pd.Timestamp("1970-01-01")) // pd.Timedelta(microseconds=1) + return ts * 0 + float(micros) + + +@pytest.mark.parametrize("setup_tw_aligned_db", ["rel_sec"], indirect=True) +def test_udf_reads_absolute_start_ts_next_to_relative_time_window_event( + spark, setup_tw_aligned_db +): + """Customer pattern: channels hold seconds since the container start, start_ts / stop_ts + are TIMESTAMP. A TimeWindowEvent tiles each container in that relative frame, while a UDF + in the same report still reads the absolute start_ts as a pd.Timestamp.""" + schema, window_length, channel_time = setup_tw_aligned_db + table_prefix = "time_window_udf_absolute_test" + previous_tz = spark.conf.get("spark.sql.session.timeZone") + spark.conf.set("spark.sql.session.timeZone", "UTC") + try: + report = Report( + name="time_window_udf_absolute_report", + spark=spark, + workspace_client=create_autospec(WorkspaceClient), + config=_aligned_config(schema, table_prefix, channel_time=channel_time), + ) + window_evt = TimeWindowEvent(name="ten_min", window_length=window_length) + report.add_event(window_evt) + page = Page(page_number=1) + report.add_page(page) + page.add_aggregation(_rpm_stats(report, window_evt)) + rpm = report.get_db().query.channel(channel_name="Engine RPM") + absolute_start = StatsAggregator( + name="absolute_start_per_window", + input_expressions=[rpm.apply(_absolute_start_micros, container_metrics=["start_ts"])], + channel_names=["absolute_start_us"], + statistics=["max"], + event=window_evt, + ) + page.add_aggregation(absolute_start) + report.determine_report() + report.persist_results() + finally: + spark.conf.set("spark.sql.session.timeZone", previous_tz) + + _assert_ids_join(spark, table_prefix) + _assert_windows_start_at_zero(spark, table_prefix) + + # The UDF saw the absolute start: in every window with samples, its value equals + # unix_micros(start_ts) of that container. + expected = { + r.container_id: float(r.us) + for r in spark.read.table(f"{schema}.container_metrics") + .select("container_id", F.unix_micros("start_ts").alias("us")) + .collect() + } + values = ( + spark.read.table(f"spark_catalog.gold.{table_prefix}_stats_aggregator_fact") + .filter(F.col("visual_id") == absolute_start.get_id()) + .filter(F.col("statistic_value").isNotNull() & ~F.isnan("statistic_value")) + .collect() + ) + assert values + assert all(r.statistic_value == expected[r.container_id] for r in values), values[:3] diff --git a/tests/impulse_reporting/unit/aggregations/definition_hash_test.py b/tests/impulse_reporting/unit/aggregations/definition_hash_test.py index c901f3b0..b68fe493 100644 --- a/tests/impulse_reporting/unit/aggregations/definition_hash_test.py +++ b/tests/impulse_reporting/unit/aggregations/definition_hash_test.py @@ -17,6 +17,7 @@ ) from impulse_reporting.aggregations.stats_aggregator import StatsAggregator from impulse_reporting.events.basic_event import BasicEvent +from impulse_reporting.events.time_window_event import TimeWindowEvent class TestHistogramDefinitionHash: @@ -354,6 +355,28 @@ def test_hash_without_custom_stats_matches_formula(self): assert stats_agg.determine_definition_hash() == expected + def test_time_window_channel_time_changes_hash(self): + """Statistics scoped to a TimeWindowEvent are computed per window, and the windows + lie in the report's channel time frame, so changing it must move the aggregation's + hash too (it does so through the event expression string).""" + event = TimeWindowEvent(name="windows", window_length=10_000) + stats_agg = StatsAggregator( + name="stats", + input_expressions=[TimeSeriesSelector(None)], + channel_names=["ch_a"], + statistics=["min", "max"], + event=event, + ) + hist = HistogramDuration( + name="hist", base_expr=TimeSeriesSelector(None), bins=[0.0, 1.0], event=event + ) + before = (stats_agg.determine_definition_hash(), hist.determine_definition_hash()) + + event.set_channel_time("ms", "container_start") + + assert stats_agg.determine_definition_hash() != before[0] + assert hist.determine_definition_hash() != before[1] + def test_renaming_channel_names_changes_hash(self): """channel_names is the fact-table merge key, so a rename must force recompute.""" agg1 = self._make(channel_names=["ch_a", "ch_b"]) diff --git a/tests/impulse_reporting/unit/aggregations/stats_aggregator_test.py b/tests/impulse_reporting/unit/aggregations/stats_aggregator_test.py index 646d350b..a12db25e 100644 --- a/tests/impulse_reporting/unit/aggregations/stats_aggregator_test.py +++ b/tests/impulse_reporting/unit/aggregations/stats_aggregator_test.py @@ -4,6 +4,8 @@ Tests follow the same pattern as histogram_test.py. """ +import math + import pyspark.sql.functions as f import pyspark.sql.types as T import pytest @@ -14,10 +16,12 @@ PerChannelStatistic, ) from impulse_query_engine.analyze.query.solvers.default_solver import DefaultSolver +from impulse_query_engine.measurement_db import MeasurementDB, MeasurementDBConfig from impulse_reporting.aggregations.stats_aggregator import StatsAggregator from impulse_reporting.events.basic_event import BasicEvent from impulse_reporting.events.container_event import ContainerEvent from impulse_reporting.events.points_in_time_event import PointsInTimeEvent +from impulse_reporting.events.time_window_event import TimeWindowEvent from impulse_reporting.persist.dimension_schema import STATS_AGGREGATOR_DIMENSION_SCHEMA @@ -530,6 +534,116 @@ def test_determine_aggregations_container_event_instance_id(spark, basic_narrow_ assert all(row.event_instance_id != row.expected_container_id for row in basic_rows) +def _aligned_boundaries_db(basic_narrow_db: MeasurementDB) -> MeasurementDB: + """Clone of basic_narrow_db whose container_metrics start_ts / stop_ts span each + container's channel samples (µs), so time windows actually overlap the data. In the + original, the boundaries (2025, epoch ms) and the samples (2017, epoch µs) never meet.""" + tables = dict(basic_narrow_db.config.debug_tables) + bounds = ( + tables["channels"] + .groupBy("container_id") + .agg(f.min("tstart").alias("start_ts"), f.max("tend").alias("stop_ts")) + ) + tables["container_metrics"] = ( + tables["container_metrics"].drop("start_ts", "stop_ts").join(bounds, "container_id") + ) + return MeasurementDB(MeasurementDBConfig.for_debug(tables), ws=basic_narrow_db.ws) + + +def test_determine_aggregations_time_window_event_instance_id(spark, basic_narrow_db): + """Time-window stats rows carry the timestamp-based id that + ``TimeWindowEvent.determine_events`` writes to ``event_instance_fact`` (one id per window, + all of them materialized) and the statistics of their own window. Basic-event stats in the + same frame keep their own ids.""" + db = _aligned_boundaries_db(basic_narrow_db) + window_length = 600_000_000 # 10 min in µs + eng_rpm = db.query.channel(channel_name="Engine RPM") + + window_event = TimeWindowEvent(name="ten_min", window_length=window_length) + basic_event = BasicEvent(name="rpm_event", expr=eng_rpm > 500) + window_stats = StatsAggregator( + name="window_stats", + input_expressions=[eng_rpm], + channel_names=["Engine RPM"], + statistics=["min", "max"], + event=window_event, + ) + basic_stats = StatsAggregator( + name="basic_stats", + input_expressions=[eng_rpm], + channel_names=["Engine RPM"], + statistics=["min", "max"], + event=basic_event, + ) + + solver = DefaultSolver(spark) + solved_df = db.query.select(window_stats.get_expression(), basic_stats.get_expression()).solve( + spark, solver + ) + df = StatsAggregator.determine_aggregations( + spark=spark, aggregations=[window_stats, basic_stats], solved_df=solved_df + ) + windows = TimeWindowEvent.determine_events( + spark, [window_event], query=db.query, solver=solver + ) + + window_rows = windows.collect() + window_ids = {r.event_instance_id for r in window_rows} + window_count = { + cid: sum(1 for r in window_rows if r.container_id == cid) + for cid in {r.container_id for r in window_rows} + } + stats_rows = df.filter(f.col("visual_id") == window_stats.get_id()).collect() + assert stats_rows + assert {r.event_instance_id for r in stats_rows} <= window_ids + # Every window of every solved container gets its own id. + for cid, n in window_count.items(): + assert len({r.event_instance_id for r in stats_rows if r.container_id == cid}) == n + + # Real values. The RPM channel only covers each container's first minute, so only the + # first window holds samples: its min / max are those of the RPM samples, all other + # windows carry no value. + rpm_ids = ( + db.channel_metrics(spark) + .filter(f.col("channel_name") == "Engine RPM") + .select("container_id", "channel_id") + ) + rpm = { + r.container_id: r + for r in db.channels(spark) + .join(rpm_ids, ["container_id", "channel_id"]) + .groupBy("container_id") + .agg( + f.min(f.col("value").cast("double")).alias("min"), + f.max(f.col("value").cast("double")).alias("max"), + f.max("tend").alias("last_tend"), + ) + .collect() + } + first_window = {} + for r in window_rows: + if r.container_id not in first_window or r.start_ts < first_window[r.container_id][0]: + first_window[r.container_id] = (r.start_ts, r.event_instance_id) + for cid, (start, first_id) in first_window.items(): + assert rpm[cid].last_tend - start < window_length, "fixture: RPM beyond window 0" + values = { + r.aggregation_label: r.statistic_value + for r in stats_rows + if r.event_instance_id == first_id + } + assert values == {"min": rpm[cid].min, "max": rpm[cid].max}, (cid, values) + later = [ + r for r in stats_rows if r.event_instance_id not in {i for _, i in first_window.values()} + ] + # No samples: the statistic is null (or NaN). + assert later + assert all(r.statistic_value is None or math.isnan(r.statistic_value) for r in later) + + basic_rows = df.filter(f.col("visual_id") == basic_stats.get_id()).collect() + assert len(basic_rows) > 0 + assert not {r.event_instance_id for r in basic_rows} & window_ids + + def test_determine_metadata_df(spark, basic_narrow_db): """Test that determine_metadata_df returns a DataFrame with expected columns.""" eng_rpm = basic_narrow_db.query.channel(channel_name="Engine RPM") diff --git a/tests/impulse_reporting/unit/core/report_utils_test.py b/tests/impulse_reporting/unit/core/report_utils_test.py index 01551557..61faab4c 100644 --- a/tests/impulse_reporting/unit/core/report_utils_test.py +++ b/tests/impulse_reporting/unit/core/report_utils_test.py @@ -495,7 +495,7 @@ def test_empty_events_by_type_returns_empty(self): query=MagicMock(), solver=MagicMock(), pre_filtered_containers_df=None, - container_event_cls=object, + boundary_event_cls=object, ) assert event_dfs == {} @@ -532,7 +532,7 @@ def determine_metadata_df(cls, spark, events): query=MagicMock(), solver=MagicMock(), pre_filtered_containers_df=MagicMock(spec=DataFrame), - container_event_cls=FakeContainerBase, + boundary_event_cls=FakeContainerBase, ) assert received_kwargs == {"solved_df": mock_solved} @@ -573,7 +573,7 @@ def determine_metadata_df(cls, spark, events): query=mock_query, solver=mock_solver, pre_filtered_containers_df=mock_pre_filtered, - container_event_cls=FakeContainerBase, + boundary_event_cls=FakeContainerBase, ) assert received_kwargs["query"] is mock_query @@ -594,7 +594,7 @@ def test_empty_event_list_for_type_is_skipped(self): query=MagicMock(), solver=MagicMock(), pre_filtered_containers_df=None, - container_event_cls=object, + boundary_event_cls=object, ) assert "BASIC_EVENT" not in event_dfs @@ -633,7 +633,7 @@ def determine_metadata_df(cls, spark, events): query=MagicMock(), solver=MagicMock(), pre_filtered_containers_df=None, - container_event_cls=FakeContainerBase, + boundary_event_cls=FakeContainerBase, ) assert len(meta_calls) == 0 diff --git a/tests/impulse_reporting/unit/events/container_event_test.py b/tests/impulse_reporting/unit/events/container_event_test.py index 9a9914da..ad85a4a3 100644 --- a/tests/impulse_reporting/unit/events/container_event_test.py +++ b/tests/impulse_reporting/unit/events/container_event_test.py @@ -1,5 +1,7 @@ """Unit tests for ContainerEvent.""" +import hashlib + import pyspark.sql.functions as f from impulse_query_engine.analyze.query.solvers.default_solver import DefaultSolver @@ -141,6 +143,18 @@ def test_definition_hash_ignores_description(): assert ev1.determine_definition_hash() == ev2.determine_definition_hash() +def _sha256_long(text: str) -> int: + return int.from_bytes(hashlib.sha256(text.encode()).digest()[:8], "big", signed=True) + + +def test_definition_hash_is_name_only(): + """The channel time settings only shape TimeWindowEvent windows; a ContainerEvent writes + the raw boundaries, so its hash stays the name alone and never forces a recompute.""" + event = ContainerEvent(name="ev") + assert event.determine_definition_hash() == _sha256_long("ev") + assert event.as_dict()["definition_hash"] == _sha256_long("ev") + + # --------------------------------------------------------------------------- # determine_events (integration-ish, needs Spark) # --------------------------------------------------------------------------- diff --git a/tests/impulse_reporting/unit/events/time_window_event_test.py b/tests/impulse_reporting/unit/events/time_window_event_test.py new file mode 100644 index 00000000..849b97d1 --- /dev/null +++ b/tests/impulse_reporting/unit/events/time_window_event_test.py @@ -0,0 +1,157 @@ +"""Unit tests for TimeWindowEvent.""" + +import pytest + +from impulse_query_engine.analyze.query.events.time_window_expression import ( + MAX_WINDOWS_PER_CONTAINER, + TimeWindowExpression, +) +from impulse_reporting.events.container_boundary_event import ContainerBoundaryEvent +from impulse_reporting.events.container_event import ContainerEvent +from impulse_reporting.events.time_window_event import TimeWindowEvent + + +# --------------------------------------------------------------------------- +# Constructor / basic attributes +# --------------------------------------------------------------------------- +def test_init(): + event = TimeWindowEvent(name="w10", window_length=10000) + assert event.name == "w10" + assert event.window_length == 10000 + assert event.description is None + assert isinstance(event.get_expression(), TimeWindowExpression) + + +def test_init_surfaces_window_length_attribute(): + event = TimeWindowEvent(name="w10", window_length=10000) + assert event.attributes["window_length"] == "10000.0" + + +def test_window_length_normalized_across_int_and_float(): + # 10000 and 10000.0 are the same windows: event_dimension must not differ between them. + a = TimeWindowEvent(name="w", window_length=10000) + b = TimeWindowEvent(name="w", window_length=10000.0) + assert isinstance(a.window_length, float) + assert a.window_length == a.get_expression().window_length + assert a.attributes == b.attributes + assert a.as_dict() == b.as_dict() + + +def test_init_does_not_override_user_window_length_attribute(): + event = TimeWindowEvent( + name="w10", window_length=10000, attributes={"window_length": "custom"} + ) + assert event.attributes["window_length"] == "custom" + + +def test_is_container_boundary_event_but_not_container_event(): + # Routed via the filter pipeline like ContainerEvent, but a sibling (not a subclass), so + # it gets its own (window-index) instance ids and is not limited to one per report. + event = TimeWindowEvent(name="w", window_length=10) + assert isinstance(event, ContainerBoundaryEvent) + assert not isinstance(event, ContainerEvent) + assert issubclass(ContainerEvent, ContainerBoundaryEvent) + + +@pytest.mark.parametrize("bad", [0, -1, -5.5, None, float("inf"), float("-inf"), float("nan")]) +def test_non_positive_window_length_raises(bad): + with pytest.raises(ValueError, match="strictly positive"): + TimeWindowEvent(name="bad", window_length=bad) + + +# --------------------------------------------------------------------------- +# get_id / type string +# --------------------------------------------------------------------------- +def test_get_id_is_positive_int_and_deterministic(): + a = TimeWindowEvent(name="same", window_length=10) + b = TimeWindowEvent(name="same", window_length=99) # id keys on name only + assert isinstance(a.get_id(), int) and a.get_id() > 0 + assert a.get_id() == b.get_id() + + +def test_event_type_str(): + assert TimeWindowEvent(name="w", window_length=10).get_event_type_str() == "TIME_WINDOW_EVENT" + + +# --------------------------------------------------------------------------- +# definition hash — must move with window_length, stable otherwise +# --------------------------------------------------------------------------- +def test_definition_hash_changes_with_window_length(): + a = TimeWindowEvent(name="w", window_length=10000) + b = TimeWindowEvent(name="w", window_length=60000) + assert a.determine_definition_hash() != b.determine_definition_hash() + + +def test_definition_hash_stable_across_desc_and_attributes(): + a = TimeWindowEvent(name="w", window_length=10000, desc="a", attributes={"k": "1"}) + b = TimeWindowEvent(name="w", window_length=10000, desc="b", attributes={"k": "2"}) + assert a.determine_definition_hash() == b.determine_definition_hash() + + +def test_definition_hash_stable_across_int_and_float_window_length(): + # 10000 and 10000.0 describe identical windows; the hash must not change between them + # (otherwise an int/float re-run forces a spurious full recompute in incremental mode). + a = TimeWindowEvent(name="w", window_length=10000) + b = TimeWindowEvent(name="w", window_length=10000.0) + assert a.determine_definition_hash() == b.determine_definition_hash() + + +def test_definition_hash_changes_with_channel_time_frame(): + # The channel time frame decides where the windows lie, so changing the unit or the + # origin must force a full recompute. It reaches the hash through the expression string. + def event(unit=None, origin="epoch", container_unit=None) -> TimeWindowEvent: + e = TimeWindowEvent(name="w", window_length=10000) + e.set_channel_time(unit, origin, container_unit) + return e + + unset = TimeWindowEvent(name="w", window_length=10000) + s, ms, ms_relative = event("s"), event("ms"), event("ms", "container_start") + ms_from_s = event("ms", container_unit="s") + + assert ms_relative.get_expression().channel_time_unit == "ms" + assert ms_relative.get_expression().channel_time_origin == "container_start" + assert "channel_time_origin=container_start" in ms_relative.as_dict()["event_expression"] + # The defaults keep today's hash. + assert unset.determine_definition_hash() == event().determine_definition_hash() + assert ms_from_s.get_expression().container_time_unit == "s" + hashes = {e.determine_definition_hash() for e in (unset, s, ms, ms_relative, ms_from_s)} + assert len(hashes) == 5 + + +# --------------------------------------------------------------------------- +# max_windows_per_container — guard rail, not part of the definition +# --------------------------------------------------------------------------- +def test_max_windows_per_container_default_and_override(): + assert TimeWindowEvent(name="w", window_length=10).max_windows_per_container == ( + MAX_WINDOWS_PER_CONTAINER + ) + event = TimeWindowEvent(name="w", window_length=10, max_windows_per_container=5) + assert event.max_windows_per_container == event.get_expression().max_windows == 5 + + +def test_max_windows_per_container_excluded_from_hash(): + a = TimeWindowEvent(name="w", window_length=10) + b = TimeWindowEvent(name="w", window_length=10, max_windows_per_container=5) + assert a.determine_definition_hash() == b.determine_definition_hash() + + +@pytest.mark.parametrize("bad", [0, -1, 2.5, True, None]) +def test_invalid_max_windows_per_container_raises(bad): + with pytest.raises(ValueError, match="^max_windows_per_container must be a positive integer"): + TimeWindowEvent(name="w", window_length=10, max_windows_per_container=bad) + + +# --------------------------------------------------------------------------- +# metadata dict shape +# --------------------------------------------------------------------------- +def test_as_dict_shape(): + event = TimeWindowEvent( + name="w10", window_length=10000, desc="ten second windows", required_channels=["c1"] + ) + d = event.as_dict() + assert d["event_type"] == "TIME_WINDOW_EVENT" + assert d["event_name"] == "w10" + assert d["event_description"] == "ten second windows" + assert d["required_channels"] == ["c1"] + assert d["event_expression"] != "NA" + assert d["attributes"]["window_length"] == "10000.0"