Data divergence between devices after concurrent offline edits #274

Activity

1. lukaskubanek

   added

   bugSomething isn't workingSomething isn't working

   on Oct 24, 2025
2. lukaskubanek commented on Nov 23, 2025

   

   lukaskubanek

   on Nov 23, 2025

   Last edited by lukaskubanek

   ContributorAuthor

   More actions

   As part of elaborating on the customizable conflict-resolution work in #272, I looked deeper into the divergence described in this issue, especially scenario 2. I initially assumed that device A wouldn’t process the fetch correctly when it had a local modification and would stale it out, but it turns out that device A does send its change to the server. The divergence actually emerges on device B, which doesn’t accept the change when it arrives from the server.

   ---

   Let’s zoom in on the scenario from the Reminders sample app:

   1. Both devices start with a reminder with ID 3169aa9e-0b40-48af-8f0b-a50c3fb04a72 and the title set to foo. The server also holds foo.

   2. Device A goes offline.

   3. Device A changes the title of the reminder to fooA.

   INSERT INTO "reminders" ("id", "title", …)
   VALUES ('3169aa9e-0b40-48af-8f0b-a50c3fb04a72', 'fooA', …)
   ON CONFLICT ("id") DO UPDATE SET
     "title" = "excluded"."title",
     …;

   4. Device B (still online) changes the title of the reminder to fooB.

   INSERT INTO "reminders" ("id", "title", …)
   VALUES ('3169aa9e-0b40-48af-8f0b-a50c3fb04a72', 'fooB', …)
   ON CONFLICT ("id") DO UPDATE SET
     "title" = "excluded"."title",
     …;

   5. Device B’s sync engine uploads fooB to the server. CloudKit then reports the same record back to the device (as it always does after a successful save), but this round-trip is effectively a no-op. The server now holds fooB.

   SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
     ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ event     ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ➡️ Sending │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └───────────┴────────────┴────────────────────────────────────────────────┘
   
   SQLiteData (private.db) sentRecordZoneChanges
     ┏━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ action  ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ✅ Saved │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └─────────┴────────────┴────────────────────────────────────────────────┘
   
   SQLiteData (private.db) fetchedDatabaseChanges
     ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
     ┃ action     ┃ zoneName                            ┃ ownerName        ┃
     ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
     │ ✅ Modified │ co.pointfree.SQLiteData.defaultZone │ __defaultOwner__ │
     └────────────┴─────────────────────────────────────┴──────────────────┘
     
   SQLiteData (private.db) fetchedRecordZoneChanges
     ┏━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ action      ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ✅ Modified │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └─────────────┴────────────┴────────────────────────────────────────────────┘
   

   6. Device A comes back online.

   7. Device A schedules a send that includes its local change. Since it hasn’t yet seen device B’s update, it tries to apply fooA over the last-known server record value foo. But because the server now holds fooB, the save fails. (Depending on timing, this failure may occur before or after device A receives the server’s updates in the next step.)

   SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
     ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ event     ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ➡️ Sending │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └───────────┴────────────┴────────────────────────────────────────────────┘
   
   SQLiteData (private.db) sentRecordZoneChanges
     ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ action        ┃ recordType ┃ recordName                                     ┃ error                    ┃
     ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ 🛑 Save failed │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │ serverRecordChanged (14) │
     └───────────────┴────────────┴────────────────────────────────────────────────┴──────────────────────────┘
   

   8. Device A receives the updated record from the server and updates its last-known server record to fooB. Because the local modification is still present and isRowValueModified is true, the upsert logic in SyncEngine.upsertFromServerRecord(…) and CKRecord.update(with:row:columnNames:parentForeignKey:) skips applying the server value to the row in the database. The server now holds fooB, device A’s last-known server record reflects fooB, but the row itself still contains fooA.

   SQLiteData (private.db) fetchedDatabaseChanges
     ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
     ┃ action     ┃ zoneName                            ┃ ownerName        ┃
     ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
     │ ✅ Modified │ co.pointfree.SQLiteData.defaultZone │ __defaultOwner__ │
     └────────────┴─────────────────────────────────────┴──────────────────┘
     
   SQLiteData (private.db) fetchedRecordZoneChanges
     ┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ action     ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ✅ Modified │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └────────────┴────────────┴────────────────────────────────────────────────┘
   

   9. Device A schedules another save, as the sync engine is still trying to push the pending save through. At this point, the last-known server record reflects fooB, so the outgoing change is treated as updating from fooB to fooA. This time the save succeeds. The server, device A’s last-known server record, and the row in the database all now contain fooA.

   SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
     ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ event     ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ➡️ Sending │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └───────────┴────────────┴────────────────────────────────────────────────┘
     
   SQLiteData (private.db) sentRecordZoneChanges
     ┏━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ action  ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ✅ Saved │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └─────────┴────────────┴────────────────────────────────────────────────┘
   

   10. Next, device B fetches this change from the server and attempts to upsert from the fetched server record. Remember that device B holds onto fooB in both the last-known server record as well as the row. Although a modification is reported, the upsert is a no-op: updating the title value via CKRecord.setValue(…) on the fetched server record with values from device B’s last-known server record overwrites fooA with fooB, because fooB has the newer timestamp. Since the didSet flag is now set, the title column is excluded from updates. As a result, the last-known server record ends up storing fooB (even though the server now holds fooA), and the row also continues to contain fooB.

   SQLiteData (private.db) fetchedDatabaseChanges
     ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
     ┃ action     ┃ zoneName                            ┃ ownerName        ┃
     ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
     │ ✅ Modified │ co.pointfree.SQLiteData.defaultZone │ __defaultOwner__ │
     └────────────┴─────────────────────────────────────┴──────────────────┘
     
   SQLiteData (private.db) fetchedRecordZoneChanges
     ┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
     ┃ action     ┃ recordType ┃ recordName                                     ┃
     ┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
     │ ✅ Modified │ reminders  │ 3169aa9e-0b40-48af-8f0b-a50c3fb04a72:reminders │
     └────────────┴────────────┴────────────────────────────────────────────────┘
   

   INSERT INTO "reminders" ("id", "title", …)
   VALUES ('3169aa9e-0b40-48af-8f0b-a50c3fb04a72', 'fooB', …)
   ON CONFLICT ("id") DO UPDATE SET
     "title" = "excluded"."title",
     …;

   11. And boom, the value has diverged! The state only repairs itself after modifying the broken field directly. Changing any other field does not cause the sync engine to work it out.

   ---

   Obviously, ending up with diverged values is suboptimal. But the question is what the correct behavior should be here. According to my mental model, we’re running into a conflict in step 7 or 8, be it on send or on fetch, as the last-known server (aka ancestor) version (foo), the server version (fooB), and the database row (aka client) version (fooA) differ. With a customizable three-way merge per #272, one could plug in any strategy to resolve it, e.g. a string-based diff that produces fooAB. We’re not there yet, but I’d say this logic applies to the current built-in per-field LWW strategy as well. In other words, receiving a newer fooB from the server in steps 7 or 8 should replace the local modification.

   @mbrandonw @stephencelis Does this make sense?
3. lukaskubanek commented on Nov 24, 2025

   

   lukaskubanek

   on Nov 24, 2025

   ContributorAuthor

   More actions

   Over in #272, I’ve broken down the exact (non-)conflict scenarios that get executed on each device, ultimately leading to the divergence described in this issue. It comes down to the combination of scenario C1 on device A and scenario NC4 on device B. I also propose some high-level adjustments to how the LWW strategy could be applied to avoid this class of divergence.
4. lukaskubanek commented on Nov 11, 2026

   

   lukaskubanek

   on Nov 11, 2026

   ContributorAuthor

   More actions

   Now that #414 is up for review, here is how scenario 2 from above maps onto its tests. The tests run a single sync engine playing device A, so device B’s edits appear as server edits, and “server newer” means device B’s edit is the newer one. Steps 1 to 6 are the shared setup. The tests differ in how device A processes the failed send and the fetch:

   Test	Ordering on Device A
   sameFieldChange_conflictOnSend_fetchBeforeRetry_serverNewer	The ordering logged above. The send fails, the fetch arrives, then the retry goes out.
   sameFieldChange_conflictOnSend_retryBeforeFetch_serverNewer	The last two steps swapped. The retry succeeds before the fetch arrives.
   sameFieldChange_conflictOnFetch_serverNewer	The fetch arrives before device A attempts any send, so there is no failed send at all.

   The first two orderings (conflict-on-send) pass since #574, as the failed send now stores the outgoing record as the last-known server record, which carries device A’s timestamp into the merge. So the sequence logged above no longer reproduces on main. The third one (conflict-on-fetch) still fails and is marked as a known issue in #414. It is the case reported in #354, and I have a small fix ready for it that should close both issues.

   Scenario 1, where device A’s edit is newer, maps onto the three sameFieldChange_*_clientNewer siblings, which all pass as expected.