Skip to content

Undo deletion_queue insert on failure in next tx#279

Draft
sfc-gh-npuka wants to merge 1 commit into
mainfrom
naisila/deletion-queue
Draft

Undo deletion_queue insert on failure in next tx#279
sfc-gh-npuka wants to merge 1 commit into
mainfrom
naisila/deletion-queue

Conversation

@sfc-gh-npuka
Copy link
Copy Markdown
Collaborator

How it works:

Transaction N:
  PRE_COMMIT:
    1. UndoFailedRestCatalogDeletionQueueEntries()
       → DELETEs paths from deletion_queue that failed in tx N-1
    2. ApplyIcebergMetadataChanges()
       → InsertDeletionQueueRecord(metadataPath, ...) as before (optimistic)
       → reports the inserted path back via output parameter
    3. The path is saved in RestCatalogRequestPerTable.deletionQueueMetadataPath
  COMMIT:
    4. PostAllRestCatalogRequests() → HTTP POST to REST catalog
       If 204: done (deletion queue entry was correct all along)
       If not 204: save all deletionQueueMetadataPath values to
               FailedRestCatalogDeletionPaths (TopMemoryContext)

Transaction N+1:
  PRE_COMMIT:
    1. UndoFailedRestCatalogDeletionQueueEntries()
       → DELETEs the stale paths from FailedRestCatalogDeletionPaths
    2. ApplyIcebergMetadataChanges()
       → same metadataPath is fetched again (REST catalog unchanged)
       → InsertDeletionQueueRecord succeeds (no duplicate since it was deleted in step 1)
   ...

Partial fix of #226 : we would still run into the same error if backend exits without executing any subsequent transaction


Checklist

  • I have tested my changes and added tests if necessary
  • I updated documentation if needed
  • I confirm that all my commits are signed off (DCO)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant