Skip to main content
VentStream is an open-source data-streaming engine with two independent workloads in one binary:
  • Capture & denormalize — read a system’s native change stream and keep a derived, denormalized view continuously in sync in a target store. You declare the target document shape, and the engine updates affected documents as source records change.
  • Real-time data delivery — publish domain events once and stream them to any number of subscribers (browsers, services) in real time over native WebSocket or GraphQL subscriptions.
Both pipelines are the same idea — turn a stream of changes into the exact shape a consumer needs. The difference is how you consume it: a queryable derived view you read from a store (capture), or a live event stream you subscribe to (delivery). Run either, or both, selected by role.
Supported today. VentStream is source / sink / transport-pluggable; these are the first implemented backends, and more are planned:
  • Capture sources: PostgreSQL logical replication, Neo4j CDC, MongoDB change streams, MySQL/MariaDB row binlog, and Kafka/Redpanda topics
  • Target stores: OpenSearch / Elasticsearch and Redis
  • Real-time brokers: NATS Core, NATS JetStream, and Redis Streams
The rest of these docs use those backends in examples, but the concepts are backend-agnostic.

​
The problems it solves

Derived data drifts. The moment data is copied into a search index, cache, or read model, the copy can drift from its source:
  • Periodic re-sync jobs re-scan everything on a timer — minutes to hours behind, expensive, and they still miss anything that changed mid-run.
  • Application dual-writes couple every write path to the derived store and rot the moment one path forgets to update it.
  • Hand-rolled triggers become an unmaintainable web of per-table glue as the projection grows.
Real-time delivery repeats infrastructure work. Teams need connection lifecycle, subscription routing, reconnect, replay, and slow-consumer handling before they can deliver an application event. VentStream replaces both with one declarative model and a streaming engine. Only the affected data is recomputed or delivered, and latency is controlled by the source polling, projection, dispatch, broker, and sink settings selected for the workload.

​
What makes it different

Direct native CDC

For database sources, one binary reads the source’s native change stream directly and writes to the target. Kafka/Redpanda is also supported when it is already your event backbone. No separate stream processor is required for projection and denormalization.

Joins & field selection, built in

Most CDC tools emit raw per-table row changes — you bolt on a stream processor (Kafka Streams, Flink, ksqlDB) to embed related rows and pick fields. VentStream does the projection inline: embed related records, select columns, walk multiple hops — declared in YAML, with bounded fan-out so a change recomputes only what depends on it.

Focused data path

The engine streams and reshapes data — nothing else. It isn’t an app server, a broker, or a database.

Single box or distributed — same model

Run one agent on one server or a fleet behind a router; the model doesn’t change. The GraphQL gateway composes into a federated router (Apollo / Cosmo), so it drops into a distributed graph as cleanly as it runs standalone.

WebSocket and GraphQL delivery

Native WebSocket and GraphQL subscription gateways provide routing, replay-capable broker cursors, backpressure, and consumer cleanup.

Sink-gated checkpoints

Durable source progress advances only after the sink handles the contiguous output prefix. A restart can replay unconfirmed work using deterministic document IDs.

​
How it fits together

Two planes, deployed independently:
  • Data plane — engine processes. Each runs one or more roles: cdc (capture & denormalize), ws / graphql (live fan-out). An engine reads its input stream and does its work entirely in the customer data plane.
  • Fleet management plane — an API, CLI, gateway, and metadata store. Managed supervisors connect outbound with renewable mTLS identity, reconcile revisioned desired state, and report bounded status. Fleet never touches source systems or data traffic.
See Architecture for the full picture.

​
Where to go next

Quickstart

A running capture pipeline — source change to synced document — in minutes.

Real-time subscriptions

Fan live events out to clients over WebSocket or GraphQL.

Concepts

How capture, projections, and fan-out work.

Deploy

Run locally with Docker Compose, or ship the stack to Kubernetes.