> ## Documentation Index
> Fetch the complete documentation index at: https://ventstream.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Architecture
> How VentStream separates engine data paths from optional Fleet management.
VentStream has an open-source data plane and an optional Cloud management plane.
The separation is operational, security-sensitive, and deliberate.
The purple path is management traffic. Teal and green show the CDC path from
source connectors through projection and sink acknowledgement. Orange shows the
independent broker-to-WebSocket/GraphQL delivery path. Neither data path enters
Fleet. The same engine binary can also run standalone without a supervisor or
control-plane dependency.
## Data plane
An engine process runs one or more explicit roles:
* `cdc` reads a source cursor/change stream, transforms records and joins, and
writes destinations such as OpenSearch.
* `ws` and `graphql` consume NATS subjects or Redis Streams and serve realtime
clients.
Connector credentials, source data, destination documents, broker messages, and
client traffic remain in the workload environment. Cursor and join state are
persisted locally or in the source system so restarts can resume correctly.
## Deterministic document IDs
CDC documents use IDs derived from the source relation and primary key. A
bootstrap record, live update, retry, and replay therefore target the same sink
document. Source deletes use the same ID for a sink tombstone.
Examples:
* PostgreSQL and MySQL: `shop.orders:["order-42"]`
* MongoDB: `shop.orders:["64f0..."]`
* Neo4j projection: `products_denormalized:`
This makes repeated delivery idempotent at the document boundary. It does not
make arbitrary downstream side effects idempotent; consumers of realtime events
should use the event ID for that purpose.
## Crash safety: the cursor watermark
For durable CDC sources, VentStream does not confirm source progress beyond the
contiguous prefix that the sink has handled. A crash before confirmation replays
from the previous checkpoint. Deterministic document IDs make those repeated
writes converge on the same result.
Checkpoint storage differs by source: PostgreSQL uses its replication slot,
Kafka uses committed consumer-group offsets, and Neo4j, MongoDB, and MySQL keep
cursor state on durable local storage. Retention still sets the recovery limit:
WAL, binlogs, oplogs, transaction logs, and Kafka topics must retain changes long
enough for the engine to catch up.
## Management plane
Fleet stores tenant inventory, authorization, desired state, immutable
configuration metadata, operations, agent status, and audit events. Managed agents
initiate outbound mTLS connections to the gateway. Fleet never connects to source
databases and never proxies engine traffic.
## Managed process boundary
The Fleet supervisor is PID 1 in the managed image. It owns workload identity,
the control stream, cached desired state, operation receipts, and configuration
staging. It supervises the unmodified open-source engine as a child process.
The child receives connector settings and a local configuration path, but no Fleet
private key or protocol implementation. Pause stops the child while retaining
cursor state. Drain stops it and invokes connector-specific state cleanup.
## Outage behavior
* A standalone engine has no Fleet dependency.
* A managed first boot without trusted desired state fails closed.
* A running managed deployment continues its cached state during a temporary Fleet
outage.
* A deployment cached as paused or drained remains stopped across restarts.
* API acceptance never implies agent completion; durable operation state records
delivery and execution.
See [Fleet overview](/docs/fleet/overview) and [Resource model](/docs/fleet/resource-model).