CloudKit sync can stall when syncing large datasets, resulting in missing records #565

Activity

1. jsutula

   added

   bugSomething isn't workingSomething isn't working

   on Oct 27, 2026
2. jsutula commented on Oct 27, 2026

   

   jsutula

   on Oct 27, 2026

   ContributorAuthor

   More actions

   @mbrandonw has started working on this with #564
3. jonphillips commented on Nov 2, 2026

   

   jonphillips

   on Nov 2, 2026

   More actions

   We hit this on a production app (about 50k records, mostly in single-parent NOT NULL REFERENCES child tables) and worked around it in a vendored copy. Here's one data point on what drives the 429s, in case it's useful for the fix stacked on #564.

   What we saw. A fresh device's first sync stopped with 11,753 rows in sqlitedata_icloud_unsyncedRecordIDs. Child tables were at 0.4–15% of the other device's counts and stopped moving for over a week, including across relaunches. In the device log, each fetch cycle produced one SQLiteData failure, then CloudKit's requestRateLimited ended the engine's own fetch 2–3 seconds later.

   Why the re-fetch causes the throttling (SyncEngine.swift L1531–1581 at 1.12.1):

   1. On every fetched batch, handleFetchedRecordZoneChanges re-downloads every parked record ID with records(for:), 150 per request, before it applies the batch. The request count grows with the parked set, not with the batch: 11,753 parked is about 79 requests per incoming batch.
   2. On a first sync the parked set keeps growing, because zone changes arrive in server order and children routinely come before their parents.
   3. The whole loop runs inside one withErrorReporting. A single 429 throws away everything the loop had already fetched and returns []. The batch then parks more children, and the next burst is larger.

   So retrying on requestRateLimited would stop the loss. But each batch would still cost O(parked) requests, which is what provokes the limit in the first place.

   The network fetch isn't needed in the common case. upsertFromServerRecord (L1925) writes the record's SyncMetadata, with the full server record in lastKnownServerRecord, before the upsert that fails on the foreign key. A parked record's server copy is therefore already on the device. One catch: the FK-failure branch (L1984–1996) doesn't refresh that saved copy. If a newer version of a still-parked record arrives, the older one stays.

   Our workaround, in case the shape helps (not proposing it as the upstream fix):

   • In the FK-failure branch, also setLastKnownServerRecord(serverRecord), so the saved copy is always the newest version received.
   • Retry parked records from their saved copies, merged into the batch's parents-first sort and applied in the same write. On an ordinary batch, only retry records whose table has a foreign key to a table that received records in that batch. On the first batch after the engine starts, retry every parked record once.
   • Go to the network only for records whose saved copy is missing or unusable (for example, a CKAsset whose file is gone): at most one 150-record request per batch. An error there ends the fallback for that batch only and never discards the local retries.

   With that, the same device finished its first sync in about 15 minutes, and every table's count matched the other device. The patch and its tests (MockCloudDatabase request counting plus a simulated throttle) are in jonphillips/yes-chef@ab47f8f, under Vendor/sqlite-data. Happy to help with a reproduction against the rate-limited mock once #564 lands.
4. jonphillips

   added a commit that references this issue on Nov 2, 2026

   SQLiteData 1.12.1 rebase (#425)

   ...

   d34be8d
5. mbrandonw commented on Nov 4, 2026

   

   mbrandonw

   on Nov 4, 2026

   Member

   More actions

   Hi @jsutula and @jonphillips, would y'all be able to give #569 a test? We think we have this problem fixed.
6. jsutula commented on Nov 5, 2026

   

   jsutula

   on Nov 5, 2026

   ContributorAuthor

   More actions

   I've just finished testing it with my repro scenario (Release build, fresh install + initial sync of thousands of records, 100% repro on 1.12.1):

   Pinned to #569 I can no longer reproduce the stall and resultant missing records. In 4/4 attempts all records are eventually pulled down after leaving the app in the foreground long enough.

   One caveat: it appears the first part of the fix (#567 Cache Unsynced Records) completely eliminates occurrence of CKError.requestRateLimited in my repro scenario, so my testing did not actually exercise the rate limit error handling portion of the fix.
7. mbrandonw commented on Nov 5, 2026

   

   mbrandonw

   on Nov 5, 2026

   Member

   More actions

   Hi @jsutula, great thanks for testing it out.

   And yeah, fwiw, #567 is the real fix but #569 is necessary as a migration step for people who have only cached record IDs instead full records, and also just overall rate limiting is a good thing for us to have in various places.
8. mbrandonw commented on Nov 5, 2026

   

   mbrandonw

   on Nov 5, 2026

   Member

   More actions

   Ok, this has been merged and will be released later today/tomorrow, so going to close this out for now. Please do report back if you find any issues!
9. mbrandonw

   closed this as completedon Nov 5, 2026