Skip to main content
VentStream has an open-source data plane and an optional Cloud management plane. The separation is operational, security-sensitive, and deliberate. VentStream architecture showing the optional Fleet management plane above a customer-hosted data plane. Fleet administers managed agents over outbound mTLS while CDC records and realtime broker events flow only through the VentStream engine inside the workload environment. The purple path is management traffic. Teal and green show the CDC path from source connectors through projection and sink acknowledgement. Orange shows the independent broker-to-WebSocket/GraphQL delivery path. Neither data path enters Fleet. The same engine binary can also run standalone without a supervisor or control-plane dependency.

​
Data plane

An engine process runs one or more explicit roles:
  • cdc reads a source cursor/change stream, transforms records and joins, and writes destinations such as OpenSearch.
  • ws and graphql consume NATS subjects or Redis Streams and serve realtime clients.
Connector credentials, source data, destination documents, broker messages, and client traffic remain in the workload environment. Cursor and join state are persisted locally or in the source system so restarts can resume correctly.

​
Deterministic document IDs

CDC documents use IDs derived from the source relation and primary key. A bootstrap record, live update, retry, and replay therefore target the same sink document. Source deletes use the same ID for a sink tombstone. Examples:
  • PostgreSQL and MySQL: shop.orders:["order-42"]
  • MongoDB: shop.orders:["64f0..."]
  • Neo4j projection: products_denormalized:<element-id>
This makes repeated delivery idempotent at the document boundary. It does not make arbitrary downstream side effects idempotent; consumers of realtime events should use the event ID for that purpose.

​
Crash safety: the cursor watermark

For durable CDC sources, VentStream does not confirm source progress beyond the contiguous prefix that the sink has handled. A crash before confirmation replays from the previous checkpoint. Deterministic document IDs make those repeated writes converge on the same result. Checkpoint storage differs by source: PostgreSQL uses its replication slot, Kafka uses committed consumer-group offsets, and Neo4j, MongoDB, and MySQL keep cursor state on durable local storage. Retention still sets the recovery limit: WAL, binlogs, oplogs, transaction logs, and Kafka topics must retain changes long enough for the engine to catch up.

​
Management plane

Fleet stores tenant inventory, authorization, desired state, immutable configuration metadata, operations, agent status, and audit events. Managed agents initiate outbound mTLS connections to the gateway. Fleet never connects to source databases and never proxies engine traffic.

​
Managed process boundary

The Fleet supervisor is PID 1 in the managed image. It owns workload identity, the control stream, cached desired state, operation receipts, and configuration staging. It supervises the unmodified open-source engine as a child process. The child receives connector settings and a local configuration path, but no Fleet private key or protocol implementation. Pause stops the child while retaining cursor state. Drain stops it and invokes connector-specific state cleanup.

​
Outage behavior

  • A standalone engine has no Fleet dependency.
  • A managed first boot without trusted desired state fails closed.
  • A running managed deployment continues its cached state during a temporary Fleet outage.
  • A deployment cached as paused or drained remains stopped across restarts.
  • API acceptance never implies agent completion; durable operation state records delivery and execution.
See Fleet overview and Resource model.