Skip to main content
A re-bootstrap re-snapshots the source as it is now. That correctly re-creates and updates every row that still exists — but it can’t see rows that were deleted while the agent wasn’t watching. Those rows’ documents would linger in the index as orphans. Reconciliation removes them.
Delete reconciliation currently applies to OpenSearch and Elasticsearch sinks. Redis uses a different rebuild contract: normal pause and resume preserve the cursor, while an exclusively owned keyspace can be cleared by finite routed targets and restored from a source snapshot. Shared Redis keyspaces reject destructive drain and rebootstrap operations.

​
When it runs

On a drain → resume cycle, before the bootstrap re-emits documents. If rows were deleted during the drain window, the agent reconciles them away first, so the re-bootstrap starts from a clean index. It does not run on a normal lossless resume — there, the replayed change stream carries the deletes directly.

​
How it works

The pass is before-bootstrap by design — it needs no engine internals and runs once per drain-resume:
1

List source keys

Query the source for the current set of primary keys / element IDs (streamed via a server-side cursor, so a 100k-row table never materializes in memory at once).
2

Scroll the index

Page through every document in the target index using the scroll API.
3

Diff + bulk-delete

Any document whose key isn’t in the source key-set is an orphan. Bulk-delete them (1000 per request).
Because documents use deterministic IDs, the diff is a clean set-membership test: extract the key from each doc ID, check it against the source set.

​
Cost characteristics

The scan cost is proportional to the number of documents in the target index, not the number of orphans. Deletes are issued in batches of 1,000. The source-key set is held in memory for the duration of the pass, so measure both runtime and peak memory with production-like data before scheduling reconciliation for a large index. Run it during a lower-traffic window when the scan would compete with search traffic.

​
Composite keys

Reconciliation handles both single-column and composite primary keys (Postgres) and element IDs (Neo4j) — no table is skipped for having a multi-column key. The live key set the source returns is folded into the same canonical form as the deterministic doc ID, so the diff works regardless of key arity:
  • a single column keys the set by the bare value (shop.orders:["uuid"]),
  • a composite key serializes every key column, in declared order, as a JSON array (shop.cart_items:["cart-42","prod-9"]).
The source-side live set and the document-ID parser use the same canonical key encoder, so a single pass handles single- and composite-key tables.
The deterministic doc-ID format differs per source — a JSON array for Postgres (shop.orders:["uuid"]) and a raw element-id suffix for Neo4j (products_denormalized:4:uuid:42). The reconciler knows both, so the same pass works for either backend.