Skip to content

Bulk actions on all matching events, tracked as background tasks - #2713

Open
ejsmith wants to merge 22 commits into
mainfrom
feature/bulk-operations
Open

ejsmith wants to merge 22 commits into
mainfrom
feature/bulk-operations

Conversation

@ejsmith

@ejsmith ejsmith commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Bulk actions on all matching events, tracked as background tasks

Delete every event matching a filter in the background, and follow bulk actions, project resets, and sample-data generation in one task panel.

Changes

  • All-matching selection with confirmation, progress, and cooperative cancellation.
  • The stack-level filter is resolved once when the task is created and stored with it, so every page uses the same stacks. Relative dates (events and stacks) are frozen to the creation time before counting.
  • Checkpointed batches resume from a saved cursor after retries; a task that makes no progress for two hours is retired.
  • One active destructive task per project, enforced under a per-organization lock.
  • Reuses the existing Foundatio queue, cache, lock, messaging, and resilience primitives.
  • Client: confirmation closes if the selection changes; dismissing a task is tied to its updated_utc, so a retry shows again.
API Purpose
POST /api/v2/bulk-operations Start an action; HTTP 202 with its task
GET /api/v2/tasks/{id} Read progress and outcome
GET /api/v2/organizations/{id}/tasks Active and recent tasks
POST /api/v2/tasks/{id}/cancel Request cancellation
POST /api/v2/bulk-operations
{ "type": "events.delete", "organization_id": "<id>",
  "selection": { "kind": "Filter", "filter": "type:error", "time": "[now-24h TO now]" } }

Failure handling

Case Result
Plan downgraded to no premium search mid-task Task fails with the upgrade message
Unparseable time 422 on selection.time instead of widening the delete
More than 50 recent tasks Older active tasks stay in the list
Cancel racing with progress Eligibility rechecked under the lock
Failed cache write Task add/save/cancel throws instead of being silently lost

Notes

  • No migration or new index. Task records live in the cache for seven days.
  • Deletion, checkpoints, cancellation, and indexing are separate writes; handlers tolerate replay.

Testing

  • Backend: bulk operation, resolver, freezer, task service, tracking handler, and OpenAPI snapshot tests.
  • Frontend: npm run validate and the task and bulk-actions unit tests.

Add a generic bulk operation framework and use it for events.delete.
The events list can now select everything matching the current filter
(Gmail style) and delete it in the background with a progress panel.

- Operation record in the cache (7 days) with a per-organization index.
- IBulkOperationHandler and IBulkSelectionResolver, discovered by DI.
- BulkOperationWorkItem job: batches, saved cursor, cancellation,
  one active destructive operation per project.
- Endpoints under /api/v2/bulk-operations and BulkOperationUpdated push.
- Single-item delete and the bulk batch share EventDeletionService.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T01:04:08.843557Z ee02498 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

The table body also contains a hidden row, so count the row checkboxes instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a59ac4f238

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…r, retries

- Paging no longer counts total hits; the total is counted once at creation.
- Active operations get their own index so the one-destructive-operation
  guard sees every one of them; the list shows the newest 50.
- The stack level filter is resolved once at creation and pinned, so
  selection stays fixed and pages stay cheap. A filter that matches too
  many stacks is rejected with a 422 and the real message.
- The work item carries organization, type and description so a lost
  record still produces a final Failed update.
- Temporary errors are retried from the saved cursor; permanent errors
  and exhausted retries fail the operation.
- Panel: Cancel names its operation and progress is no longer re-announced.
- Guard the select-all count against a filter change during the refetch.
- Split the e2e steps so assertions stay out of the Act sections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74cbb576ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The queue's retries cover a whole work item, so rethrowing let an operation that made progress between errors run out of deliveries while still marked running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d8cd35e19d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Without it, a run longer than the queue's work item timeout could be delivered to a second worker while the first was still deleting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 70801469e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…election

An undefined selection kind fell through to an empty filter that selected everything in scope. A restart after the last id batch but before completion started the ids over, because the saved cursor was empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dff385c438

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Otherwise it stayed queued with no work item to run it and blocked new operations on the project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8928092148

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A filter such as data.date:[now/d-4d TO now/d+1d} kept its relative dates, so a run that crossed midnight or resumed later could select different events than the user confirmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 11181b5141

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Relative bounds on string fields such as environment:[now TO z] are left alone, and custom data date fields keep their date type by naming the stored field. A stale destructive operation is now failed before a replacement is admitted, so a delayed delivery cannot run alongside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 794c82ec7b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Grouped filters such as date:(>now-2h AND <=now) put the field on the group, and custom data groups are typed by all of their terms, as the events index does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@niemyjski niemyjski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, wondered if this could be more of a tasks api for anything we return a task id / worker id for so we could track task progress in the ui as a whole.

Also feels like we need more integration / unit test coverage for all the backend code.

Replace the bulk operation record with a generic BackgroundTask stored by
BackgroundTaskService. Bulk specific logic stays in BulkOperationService.
Reset data and sample data return a task id and report progress through a
decorating work item handler. Add GET/list/cancel task endpoints.
Rename the bulk operations feature to tasks and read the generic task
endpoints and BackgroundTaskUpdated messages. Percent-only tasks show their
percent and message, Cancel only appears when the task can be cancelled, and
Reset data and Generate sample data refresh the panel.
Cover the task service (storage, index cleanup, cancel rules, lock retries),
bulk conflict and enqueue-failure rules, the event selection resolver, the
tracking work item handler, the tasks endpoints and the websocket broker.
…eared

A failed run shows a generic message instead of the exception text. Cancelling
an abandoned running task says it stopped responding, and the panel offers
Cancel for any task that has not reported progress for two hours.
private async Task<BulkSelectionPage> GetPageAsync(BackgroundTask task, IBulkSelectionResolver resolver, string? pinnedSelection, CancellationToken cancellationToken)
{
int size = Math.Max(1, _options.BatchSize);
if (task.Bulk!.Cursor == ExhaustedCursor)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Always use String.Equals and never ==

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The worker now uses String.Equals(task.Bulk!.Cursor, ExhaustedCursor, StringComparison.Ordinal) in the proposed fix. That cursor is a protocol token and should be compared ordinally.

/// Either an explicit list of ids or a filter. A filter selection always carries the absolute UTC window that was
/// resolved when the operation was created, so items that arrive afterwards are never included.
/// </summary>
public class BulkOperationSelection

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be a record, so we get equality as well as debugger displays

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed selection DTO is a record. One detail about equality matters here: its List<string> compares by reference. No current caller relies on two separately loaded ID lists comparing equal.

namespace Exceptionless.Core.Models.BackgroundTasks;

/// <summary>The types of tasks that are not bulk actions. Bulk actions use the key of their handler, e.g. <c>events.delete</c>.</summary>
public static class BackgroundTaskTypes

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure where we use this but could this be inferred a different way automatically? Not opposed to this, but just want to give more context

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ProjectHandler uses projects.reset-data and SampleDataService uses projects.sample-data. Both identifiers are stored in task records and sent to clients. Keeping their values explicit lets us rename a CLR handler without changing the wire value.

public class EventFieldsQueryVisitor : ChainableQueryVisitor
{
public override async Task VisitAsync(GroupNode node, IQueryVisitorContext context)
{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there tests for this change?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. PersistentEventQueryValidatorTests.CanProcessQueryAsync exercises the visitor. RelativeDateFreezerTests covers grouped custom dates and non-date terms, and EventBulkSelectionResolverTests checks the resulting selection against Elasticsearch.


namespace Exceptionless.Core.Services.BackgroundTasks;

public class BackgroundTaskOptions

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have options that live in a folder, they shouldn't be grouped like this, please look at how we place our services and options etcc.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed fix moves BackgroundTaskOptions into src/Exceptionless.Core/Configuration/BackgroundTaskOptions.cs, alongside the other Core options.


public static string GetTaskCacheKey(string id) => $"{KeyPrefix}:{id}";
public static string GetCancelCacheKey(string id) => $"{KeyPrefix}:{id}:cancel";
public static string GetOrganizationIndexCacheKey(string organizationId) => $"{KeyPrefix}:organization:{organizationId}";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally I'm all for this key prefix, but normally we structure all organization specific data so it can be deleted when an org is cleaned up. We need to make sure that we are able to clean this up easy. Also we should be making sure anything with an organization scope is consistent like GetCreateLockResource is inconsistent.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed hard-delete path removes the organization’s indexed task records, saved selections, and cancellation keys under each task’s update lock. A two-organization integration test verifies the other tenant’s task survives. The creation lock keeps the existing resource name so two API versions still exclude each other during rollout.

Comment on lines +137 to +138
task.CreatedUtc = utcNow;
task.UpdatedUtc = utcNow;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't these get set automatically on save by the repos

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tasks are written through ICacheClient. Repository save hooks never see them. The task service therefore needs to stamp their creation and update times.

if (!update(task))
return task;

task.UpdatedUtc = _timeProvider.GetUtcNow().UtcDateTime;

@niemyjski niemyjski Nov 18, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd think repositories would automatically be doing this date management for us. (No because it's all cache based)

But fels like these should be moved to SaveAsync and AddAsync...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed change sets CreatedUtc in AddAsync and UpdatedUtc in SaveAsync. Enqueue failure and progress updates use that same save path.


// Progress may have resumed after the initial read. Decide eligibility against the locked state.
var currentUtcNow = _timeProvider.GetUtcNow().UtcDateTime;
bool isAbandonedNow = currentUtcNow - current.UpdatedUtc >= StaleAfter;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to have Utc in the name

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timestamp is currentUtcNow. The proposed change removes the ambiguous isAbandonedNow name and uses IsStale(current, currentUtcNow) for the under-lock decision.


namespace Exceptionless.Core.Services.BulkOperations;

public class BulkOperationOptions

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this options, just kind of weird these are here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BulkOperationOptions moves into its own file under Core’s Configuration directory. The worker’s retry limit moves there too.

_timeProvider = timeProvider;
}

public static string GetPinnedSelectionCacheKey(string id) => $"{BackgroundTaskService.KeyPrefix}:{id}:selection";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why wouldn't all these keys be in one spot?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The task service now defines the selection key alongside the record, cancellation, and organization-index keys in the proposed patch. Tenant cleanup uses the same definitions.

return BulkOperationCreateResult.Conflicting(conflict);
}

await _taskService.AddAsync(task);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are doing date operations outside of this call here too.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The creation timestamp and selection cutoff serve different purposes. The proposed change keeps the cutoff on the selection before counting, so later pages share the same window. AddAsync owns the task’s actual creation time.

{
Description = workItem.Description ?? "Bulk operation",
Id = workItem.TaskId,
Message = "The operation's progress was lost and it could not continue.",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Careful what we put in here, there is a cost to all of this. who sees this?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This final update goes to the organization’s WebSocket group if the worker can’t find its task record. The queue item carries the task ID, organization, type, and description so the panel can identify the failed operation. The filter, selected IDs, and event payloads are excluded.

if (!overlaps)
continue;

if (utcNow - existing.UpdatedUtc >= BackgroundTaskService.StaleAfter)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Always prefer Exceptionless.DateTimeExtensions for all date comparisons, check prs and agent rules.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed IsStale check uses IsBeforeOrEqual with the two-hour UTC cutoff, including the boundary itself. The backend skill also documents using Exceptionless.DateTimeExtensions for date ordering.

/// </summary>
public class DeleteEventsBulkOperationHandler : IBulkOperationHandler
{
public const string OperationType = "events.delete";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be on the interface as a prop to make it more discoverable?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or do we just have two props tracking it......

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IBulkOperationHandler already exposes Type. The separate OperationType constant has no other caller. The proposed fix removes it and keeps the wire value on the interface implementation.

/// </summary>
public class EventBulkSelectionResolver : IBulkSelectionResolver
{
private const string CreatedUtcField = "created_utc";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really do not like magic strings like this.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed query uses (PersistentEvent e) => e.CreatedUtc, so the repository resolves the mapped field name. The cutoff test checks that an old-dated event ingested later is excluded.

}

private ValueTask<bool> IsDateFieldAsync(string field)
{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is argument validaiton, also check pr because private methods should come last in class

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The private mapping lookup now follows the public methods in the proposed patch. Its caller already checks the field name before invoking it. Filter syntax and premium eligibility are checked by the resolver before task creation.

public async Task<BulkSelectionPreparation> PrepareAsync(BackgroundTask task, CancellationToken cancellationToken)
{
var (appFilter, check) = await GetAppFilterAsync(task);
if (!check.IsOk)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is IsOk?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IsOk means check.Status is BulkOperationCheckStatus.Ok. A failed check carries the rejection reason so the caller can distinguish invalid input, denied access, and a plan restriction.

query.ExcludeStack(pinned.ExcludedStackIds);

// Paging only needs the next ids; the total was counted once when the task was created.
var options = new CommandOptions<PersistentEvent>().SearchAfterToken(cursor, _serializer).PageLimit(size).TrackTotalHits(false);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason this couldn't have been inlined into FindAsync param

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The proposed change passes the command options directly to FindAsync, retaining the search-after cursor, page size, and disabled hit count.

}

/// <summary>What the stack level filter resolved to when the task was created.</summary>
private sealed class PinnedEventSelection

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Always record type if possible

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed PinnedEventSelection is a private sealed record with init-only properties. Its serialized field names stay the same.

/// <param name="filter">The filter to pin.</param>
/// <param name="utcNow">The time that <c>now</c> stands for.</param>
/// <param name="isDateFieldAsync">Whether a built-in (non custom data) field is mapped as a date.</param>
public static async Task<string?> FreezeAsync(string? filter, DateTime utcNow, Func<string, ValueTask<bool>> isDateFieldAsync)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this have unit tests?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. RelativeDateFreezerTests covers relative terms and ranges, grouped and custom date fields, non-date fields, wildcards, and culture-independent UTC formatting. The resolver integration tests also exercise the frozen filter against Elasticsearch.

return null;
}

private static bool IsRelative(string? value)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like we have DateTimeExtensions and parsers just a little worried we are creating tech debt here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordinary time window uses DateTimeRange.Parse. The freezer handles a different input: a Lucene filter that can contain both mapped date fields and strings. It walks the parsed expression so a value like now is anchored only when the field is a date.

/// Removes events and records the deleted usage. Shared by the single-item delete endpoint and the bulk delete
/// operation so both have identical side effects.
/// </summary>
public class EventDeletionService

@niemyjski niemyjski Nov 18, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this for one simple thing, don't we have an existing service this could have lived in? If not this is ok.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two consumers: the existing event-delete endpoint and the bulk worker. Both remove events and increment per-project deleted usage. Keeping that operation in one Core service lets both entry points share the accounting behavior.

app.MapWebHookEndpoints();
app.MapStripeEndpoints();
app.MapSavedViewEndpoints();
app.MapBackgroundTaskEndpoints();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: these should be sorted alphabetically

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried sorting the registry. Route registration order changes the emitted OpenAPI path order, which failed the existing snapshot. My recommendation is to preserve that order in this PR and avoid a document-wide diff for this nit.

_subscriber.SubscribeAsync<EntityChanged>(OnEntityChangedAsync, shutdownToken),
_subscriber.SubscribeAsync<PlanChanged>(OnPlanChangedAsync, shutdownToken),
_subscriber.SubscribeAsync<PlanOverage>(OnPlanOverageAsync, shutdownToken),
_subscriber.SubscribeAsync<BackgroundTaskUpdated>(OnBackgroundTaskUpdatedAsync, shutdownToken),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: alpha

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed patch sorts the message-bus subscriptions alphabetically. The organization group and subscription handlers stay the same.

@niemyjski niemyjski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixes for this PR are in #2734, which targets feature/bulk-operations. Merging #2734 puts those commits on this PR for its final review and CI run.

I recommend holding #2713 until the bulk delete rejects malformed windows and unreadable selections, delayed resets honor cancellation after cache expiry, and failed task registration cleans up its partial writes. The regressions reproduced locally. The proposed changes passed the targeted backend tests and Redis checks.

I’ve answered the individual questions in their original threads. Code-change threads can be resolved once the changes are present here.

private readonly ILockProvider _lockProvider;
private readonly TimeProvider _timeProvider;

public BulkOperationWorkItemHandler(BulkOperationService service, BackgroundTaskService taskService, IEnumerable<IBulkSelectionResolver> resolvers, BulkOperationOptions options,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The proposed change puts the consecutive failure limit beside batch size and retry delay in BulkOperationOptions. The limit stays at three. A successful batch resets the counter.


if (await _taskService.IsCancelRequestedAsync(task.Id))
{
await FinishAsync(task.Id, BackgroundTaskStatus.Cancelled, "Cancelled by user.");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These sentences reach users. The worker writes them to BackgroundTask.Message, the tasks API returns them, and task-item.svelte renders the message alongside the status. I’d keep the punctuation.

}

var page = await GetPageAsync(task, resolver, pinnedSelection, context.CancellationToken);
if (page.Ids.Count == 0)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed changes use is and is not for constant patterns throughout the added C# and tests. The repository’s backend skill now covers that convention.

private async Task<BulkSelectionPage> GetPageAsync(BackgroundTask task, IBulkSelectionResolver resolver, string? pinnedSelection, CancellationToken cancellationToken)
{
int size = Math.Max(1, _options.BatchSize);
if (task.Bulk!.Cursor == ExhaustedCursor)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The worker now uses String.Equals(task.Bulk!.Cursor, ExhaustedCursor, StringComparison.Ordinal) in the proposed fix. That cursor is a protocol token and should be compared ordinally.

/// Either an explicit list of ids or a filter. A filter selection always carries the absolute UTC window that was
/// resolved when the operation was created, so items that arrive afterwards are never included.
/// </summary>
public class BulkOperationSelection

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed selection DTO is a record. One detail about equality matters here: its List<string> compares by reference. No current caller relies on two separately loaded ID lists comparing equal.

/// <param name="filter">The filter to pin.</param>
/// <param name="utcNow">The time that <c>now</c> stands for.</param>
/// <param name="isDateFieldAsync">Whether a built-in (non custom data) field is mapped as a date.</param>
public static async Task<string?> FreezeAsync(string? filter, DateTime utcNow, Func<string, ValueTask<bool>> isDateFieldAsync)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. RelativeDateFreezerTests covers relative terms and ranges, grouped and custom date fields, non-date fields, wildcards, and culture-independent UTC formatting. The resolver integration tests also exercise the frozen filter against Elasticsearch.

return null;
}

private static bool IsRelative(string? value)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordinary time window uses DateTimeRange.Parse. The freezer handles a different input: a Lucene filter that can contain both mapped date fields and strings. It walks the parsed expression so a value like now is anchored only when the field is a date.

/// Removes events and records the deleted usage. Shared by the single-item delete endpoint and the bulk delete
/// operation so both have identical side effects.
/// </summary>
public class EventDeletionService

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two consumers: the existing event-delete endpoint and the bulk worker. Both remove events and increment per-project deleted usage. Keeping that operation in one Core service lets both entry points share the accounting behavior.

app.MapWebHookEndpoints();
app.MapStripeEndpoints();
app.MapSavedViewEndpoints();
app.MapBackgroundTaskEndpoints();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried sorting the registry. Route registration order changes the emitted OpenAPI path order, which failed the existing snapshot. My recommendation is to preserve that order in this PR and avoid a document-wide diff for this nit.

_subscriber.SubscribeAsync<EntityChanged>(OnEntityChangedAsync, shutdownToken),
_subscriber.SubscribeAsync<PlanChanged>(OnPlanChangedAsync, shutdownToken),
_subscriber.SubscribeAsync<PlanOverage>(OnPlanOverageAsync, shutdownToken),
_subscriber.SubscribeAsync<BackgroundTaskUpdated>(OnBackgroundTaskUpdatedAsync, shutdownToken),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposed patch sorts the message-bus subscriptions alphabetically. The organization group and subscription handlers stay the same.

@niemyjski

Copy link
Copy Markdown
Member

@ejsmith, when you asked me to take over this PR and get it done, I understood that to include working through the bugs I found. I should have brought the design and rollout changes back to you for review before pushing them to your branch. Here’s what prompted each one.

  • Selection snapshot. Freezing the date bounds and stack IDs still lets a late-arriving event match before the worker runs. I wanted “delete all matching” to delete only events that matched when the task was created. That’s a product decision about what the user is confirming.
  • Idle polling. If a task starts in another tab and this tab misses the WebSocket notification, its panel can stay empty. The one-minute refresh recovered from that, at the cost of a request per visible tab even when nobody has work running.
  • Deploy order. During a rolling deploy, an old worker could pick up a new bulk payload it didn’t understand, or run a reset without reporting progress. Updating workers first closed that window. Changing the production workflow warranted its own review.
  • Benchmark JSON. I kept the before-and-after measurements so reviewers could check what larger task histories and selections cost in cache reads, allocations, and stored bytes. Those numbers could have gone in the review instead of the repository.

All three commits I’ve made on this work since your comment are in #2734, stacked on this PR. Thanks for pulling the fixes you wanted back into #2713.

continue;
}

if (GetHandler(existing.Type) is not { IsDestructive: true })

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the task record’s organization before treating it as a conflict or marking it failed. The list endpoint already checks ownership after reading from the cache. This path trusts the index alone, so a stale index entry pointing at another organization’s task can block this request or mark that other task failed.

Suggested change
if (GetHandler(existing.Type) is not { IsDestructive: true })
if (!String.Equals(existing.OrganizationId, candidate.OrganizationId, StringComparison.Ordinal) || GetHandler(existing.Type) is not { IsDestructive: true })

@niemyjski niemyjski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please hold this PR. The local worker deleted an event outside the selection the API had accepted, and the confirmation dialog sent two bulk requests while the first was pending. The inline comments give the reproduction and regression coverage for each. There is also a mixed-version rollout gap. #2734 addresses earlier recovery issues and is still open.

On ee0249868, all 56 existing bulk and background-task endpoint tests passed. Both Chromium journeys passed against the local development stack with retries disabled, including deleting five events across pages. Two additional regression probes failed as described below. The rollout finding comes from the workflow and handler registration.


// Paging only needs the next ids; the total was counted once when the task was created.
var options = new CommandOptions<PersistentEvent>().SearchAfterToken(cursor, _serializer).PageLimit(size).TrackTotalHits(false);
var results = await _eventRepository.FindAsync(query, options);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 · Preserve the event membership accepted for deletion. I queued a delete for sessions lasting 60–120 seconds. The API reported one match. Before running the worker, I updated a second existing session from 30 to 90 seconds through UpdateSessionStart. After the queue drained, both events were gone. The creation-time cutoff admits both records, and this search re-evaluates their current values. Capture membership at acceptance (for example, bounded IDs or a point-in-time search), keep checking the current organization/project scope, and add an endpoint-to-worker test asserting the second event survives.

}

async function remove() {
if (confirmedSelectionIdentity !== selectionIdentity) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Prevent a second confirmation while deletion is pending. In Chromium I held the first request open and clicked the still-enabled “Delete 5 Events” button again. The same dialog emitted two POST /api/v2/bulk-operations requests. The probe intercepted both requests. An admission conflict only protects overlapping execution, so a fast first task can finish before the second request arrives. Guard submission synchronously in the action owner, disable the control until the request settles, and cover the delayed-response case by asserting exactly one POST.

services.AddSingleton<WorkItemHandlers>(s =>
{
var handlers = new WorkItemHandlers();
handlers.Register<BulkOperationWorkItem>(s.GetRequiredService<BulkOperationWorkItemHandler>);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 · Roll out workers before enabling the new producer. This handler exists only in the new worker image. The production workflow updates the API and work-item Deployment together in one helm upgrade --wait. Kubernetes can serve a new API replica while an old worker is still consuming the shared queue. That worker cannot handle the new bulk payload, and can run a reset without updating its task. Gate producer rollout on upgraded workers being ready and old consumers exiting. Cover a delayed or failed worker rollout and verify the web deployment does not start in either case. The same-version E2E suite cannot exercise this upgrade window.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants