Architecture

VentStream has an open-source data plane and an optional Cloud management plane. The separation is operational, security-sensitive, and deliberate. Green is the engine and its capabilities; blue is everything you own — sources, targets, brokers, and clients. The only line between the planes is a single outbound mTLS stream carrying desired state, never data: records, documents, and events stay inside your environment. Remove the agent key and the same binary runs fully standalone without a control-plane dependency.

​

Data plane

An engine process runs one or more explicit roles:

• cdc reads a source cursor/change stream, transforms records and joins, and writes destinations: OpenSearch / Elasticsearch, Meilisearch, or Redis.
• ws and graphql consume NATS subjects or Redis Streams and serve realtime clients.
• mcp serves the MCP server role; it must run alone, without the other roles.

Connector credentials, source data, destination documents, broker messages, and client traffic remain in the workload environment. Cursor and join state are persisted locally or in the source system so restarts can resume correctly.

​

Deterministic document IDs

CDC documents use IDs derived from the source relation and primary key. A bootstrap record, live update, retry, and replay therefore target the same sink document. Source deletes use the same ID for a sink tombstone. Examples:

• PostgreSQL and MySQL: shop.orders:["order-42"]
• MongoDB (from _id): shop.orders:["64f0..."]
• Kafka: the same form, keyed from the message envelope key
• Neo4j projection: products_denormalized:<element-id> (raw CDC uses the node’s elementId)

Every source stamps this id even without a joins or denormalize spec, so flat per-row documents upsert in place and deletes always find their document. An update that changes a row’s primary key removes the old document and writes the new one. This makes repeated delivery idempotent at the document boundary. It does not make arbitrary downstream side effects idempotent; consumers of realtime events should use the event ID for that purpose.

​

Crash safety: the cursor watermark

For durable CDC sources, VentStream does not confirm source progress beyond the contiguous prefix that the sink has handled. A crash before confirmation replays from the previous checkpoint. Deterministic document IDs make those repeated writes converge on the same result. Checkpoint storage differs by source: PostgreSQL uses its replication slot, Kafka uses committed consumer-group offsets, and Neo4j, MongoDB, and MySQL keep cursor state on durable local storage. Retention still sets the recovery limit: WAL, binlogs, oplogs, transaction logs, and Kafka topics must retain changes long enough for the engine to catch up.

​

Management plane

VentStream Cloud stores tenant inventory, authorization, desired state, immutable configuration metadata, operations, agent status, and audit events. Managed agents initiate outbound mTLS connections to the gateway. Cloud never connects to source databases and never proxies engine traffic.

​

Managed mode boundary

There is one public engine artifact for both modes. An engine started with an agent key (VS_AGENT_KEY, vsa1. prefix) is managed: it connects outbound to the gateway, fetches its selected configuration revision, reports status, and executes operations. Without a key, no code path opens a connection to the platform. Pause stops the pipeline while retaining cursor state; drain stops it and invokes connector-specific state cleanup. See Managed mode.

​

Outage behavior

• A standalone engine has no VentStream Cloud dependency.
• A managed first boot without trusted desired state fails closed.
• A running managed deployment continues its cached state during a temporary Cloud outage.
• A deployment cached as paused or drained remains stopped across restarts.
• API acceptance never implies agent completion; durable operation state records delivery and execution.

See VentStream Cloud overview and Resource model.