> ## Documentation Index
> Fetch the complete documentation index at: https://ventstream.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Delete reconciliation

> Why a re-bootstrap can leave orphan documents, and how the engine removes them.

A re-bootstrap re-snapshots the source as it is **now**. That correctly
re-creates and updates every row that still exists — but it can't see
rows that were **deleted** while the agent wasn't watching. Those rows'
documents would linger in the index as orphans.

Reconciliation removes them.

<Info>
  Delete reconciliation currently applies to OpenSearch and Elasticsearch
  sinks. Redis uses a different rebuild contract: normal pause and resume
  preserve the cursor, while an exclusively owned keyspace can be cleared by
  finite routed targets and restored from a source snapshot. Shared Redis
  keyspaces reject destructive drain and rebootstrap operations.
</Info>

## When it runs

On a **drain → resume** cycle, before the bootstrap re-emits documents.
If rows were deleted during the drain window, the agent reconciles them
away first, so the re-bootstrap starts from a clean index.

```mermaid theme={null}
flowchart LR
    d["drained"] --> r["resume"] --> rec["reconcile<br/>deletes"] --> b["bootstrap"] --> t["tailing"]
    classDef n fill:#0b1220,stroke:#2563eb,color:#e6edf3;
    class d,r,rec,b,t n;
```

It does not run on a normal lossless resume — there, the replayed
change stream carries the deletes directly.

## How it works

The pass is **before-bootstrap** by design — it needs no engine
internals and runs once per drain-resume:

<Steps>
  <Step title="List source keys">
    Query the source for the current set of primary keys / element IDs
    (streamed via a server-side cursor, so a 100k-row table never
    materializes in memory at once).
  </Step>

  <Step title="Scroll the index">
    Page through every document in the target index using the scroll
    API.
  </Step>

  <Step title="Diff + bulk-delete">
    Any document whose key isn't in the source key-set is an orphan.
    Bulk-delete them (1000 per request).
  </Step>
</Steps>

Because documents use [deterministic IDs](/docs/concepts/architecture#deterministic-document-ids),
the diff is a clean set-membership test: extract the key from each
doc ID, check it against the source set.

## Cost characteristics

The scan cost is proportional to the number of documents in the target index,
not the number of orphans. Deletes are issued in batches of 1,000. The
source-key set is held in memory for the duration of the pass, so measure both
runtime and peak memory with production-like data before scheduling
reconciliation for a large index. Run it during a lower-traffic window when the
scan would compete with search traffic.

## Composite keys

Reconciliation handles both **single-column and composite primary keys**
(Postgres) and **element IDs** (Neo4j) — no table is skipped for having a
multi-column key. The live key set the source returns is folded into the
*same* canonical form as the deterministic doc ID, so the diff works
regardless of key arity:

* a **single** column keys the set by the bare value
  (`shop.orders:["uuid"]`),
* a **composite** key serializes every key column, in declared order, as
  a JSON array (`shop.cart_items:["cart-42","prod-9"]`).

The source-side live set and the document-ID parser use the same canonical key
encoder, so a single pass handles single- and composite-key tables.

<Tip>
  The deterministic doc-ID format differs per source — a JSON array for
  Postgres (`shop.orders:["uuid"]`) and a raw element-id suffix for
  Neo4j (`products_denormalized:4:uuid:42`). The reconciler knows both,
  so the same pass works for either backend.
</Tip>