Skip to main content
The Kafka source is available on Linux and macOS (and in the container image). It is not included in the native Windows build — its native library has no supported Windows toolchain.
The Kafka source is a consumer: it reads change data that’s already on a Kafka topic — typically produced by a Debezium source connector — and unwraps each message into a sink document. The current connector maps each message to one sink document (raw 1:1), keyed by the row’s primary key. Works against Apache Kafka, Redpanda, AWS MSK, Confluent Cloud, and any Kafka-API broker. Set VS_CDC_SOURCE=kafka (or redpanda).
VentStream is not a Kafka Connect plugin — it’s a standalone consumer that talks to the brokers directly, with its own lifecycle. It also doesn’t capture anything: a Debezium source connector (which you run) puts change data on the topic; VentStream reads it out. If you only have a database and no Kafka, use the native Postgres / MySQL sources instead — don’t stand up Kafka just for this.

​
How it works

  • The message is the body. Unlike the database sources there’s no capture and no re-read — the Debezium after image (or the raw value) is the document. No database connection is opened.
  • Unwrap. In debezium mode the source reads the change envelope: op c/r → insert, u → update, d → delete; the body is after (before’s key for deletes). It handles both the JSON-Converter {schema, payload} wrapper and the schemaless form. In raw mode the value is the document as-is and a null value is a delete.
  • Bootstrap = consume from earliest. There’s no separate snapshot scan; the topic is the log. Start at earliest to replay history (Debezium snapshot records arrive as op:"r" inserts), or latest for live only.
  • Resume = committed offsets, sink-gated. Each message gets a monotonic consume-seq; the source commits the consumer group’s offsets only up to the seq the sink has durably written (via the shared sink-progress watermark). A crash redelivers anything not yet sink-confirmed, and the deterministic doc id makes redelivery an idempotent overwrite — no-loss at-least-once. No state directory.
  • Deletes become tombstones — the doc id targets the exact document to remove. Debezium’s trailing null-value (log-compaction) tombstone is skipped.

​
The doc-id mapping

The sink document id is the canonical, namespaced form {namespace}.{relation}:[<pk>] — e.g. shop.orders:["1"], matching the Postgres/MySQL sources. The primary-key components come from the message key (Debezium keys by PK); namespace/relation come from the envelope source block (schema/db + table/collection), or from the topic name as a fallback. Composite keys honor the key’s schema field order.

​
Run the engine

The index template renders one index per relation (orders, order_items, …).

​
Connecting to a secured cluster (SASL/TLS)

This covers AWS MSK with SASL/SCRAM, Confluent Cloud (PLAIN), and Redpanda Cloud. MSK IAM auth (and therefore MSK Serverless, which is IAM-only) is not supported in the current release because the connector does not provide an AWS SigV4 token provider. AWS documents the authentication requirement in MSK Serverless.

​
Key environment variables

Full list in the engine env reference.

​
Current limitations

  • Raw 1:1 only. One message → one sink document; there are no cross-topic joins, so a change on a referenced table’s topic doesn’t recompose a parent document.
  • JSON values only. Debezium JSON (with-schema and schemaless) and raw JSON. Avro + Confluent Schema Registry — the most common enterprise Debezium setup — is not supported.
  • Schema changes aren’t tracked. A Debezium DDL/schema-change event carries no row image and is skipped; the next data record reflects the new shape if the target mapping accepts it.
  • Primary key required. A message with no key (and no VS_KAFKA_RAW_KEY_FIELD match in raw mode) can’t form a doc id and is logged + skipped.
  • MSK IAM / MSK Serverless require an AWS SigV4 token provider, which the current connector does not include.
  • Single consumer per agent. No partition-parallel scaling / rebalance handling yet; run one agent per consumer group.
  • Offsets commit on a single global prefix. Offsets advance only through the contiguous sink-durable prefix across all partitions, so an event stalled on one partition holds back commits on the others until it’s durable. This prevents committing past unwritten data but reduces commit granularity during a multi-partition stall.

​
Debezium and unavailable values

Debezium replaces values it could not read (Postgres TOAST columns under REPLICA IDENTITY DEFAULT, and equivalents) with the __debezium_unavailable_value placeholder. VentStream rejects and skips any change event carrying it — writing the placeholder would permanently corrupt the destination document, and unlike VentStream’s native Postgres source there is no database connection here to re-read the real row. Be clear about what “skip” means: that update is dropped entirely, including its non-placeholder field changes, and the offset commits past it. On a PG-fed topic under REPLICA IDENTITY DEFAULT, every update touching a TOASTed row is skipped — watch vs_kafka_skipped_total{reason="unmappable"}. The fix belongs at the source database: ALTER TABLE t REPLICA IDENTITY FULL; (or the connector’s equivalent), after which images arrive complete. If you cannot change the source, prefer VentStream’s native Postgres source, which transparently re-reads TOASTed rows instead of skipping.