The failure we shipped first
Our first offline sync was optimistic: the tablet wrote a bill locally, assumed the server would accept it, and pushed when the connection returned. It worked in testing because testing never loses the network mid-order.
In service it produced duplicate bills. A tablet that reconnected halfway through an order would replay writes the server had already accepted. A duplicate bill in front of a guest is not a bug you get to explain away at the table.
Why an append-only log fixes it
We rewrote the client to keep an append-only log of intents rather than a mutable copy of state. Each entry carries an ID generated on the device before the write is attempted.
The server then reconciles: it applies entries it has not seen, ignores entries it has, and returns the authoritative state. Replaying the same entry twice is a no-op because the ID has already been recorded.
- Generate the ID on the client, before the first attempt — not on the server.
- Make the write endpoint idempotent on that ID. This is the whole mechanism.
- Keep the log append-only. A mutable local cache reintroduces the race you just removed.
- Reconcile server-side and treat the server as authoritative on conflict.
What it costs
It is slower to build than optimistic sync, and it is more code. That is the honest trade: you are paying in build time to make a class of bug impossible rather than unlikely.
For a restaurant floor or a hotel front desk, that trade is obviously correct. For a low-stakes draft-saving feature it may not be.