MySQL / MariaDB source

​

Source requirements

• binlog_format=ROW (with binlog_row_image=FULL, the default). Statement or mixed formats don’t carry per-row images.
• A server_id unique across the replication topology — the agent registers as a replica. Set it with VS_MYSQL_SERVER_ID.
• The login user needs REPLICATION SLAVE and REPLICATION CLIENT plus SELECT on the watched tables.

-- my.cnf
[mysqld]
server-id        = 1
log-bin          = mysql-bin
binlog-format    = ROW
binlog-row-image = FULL

​

How it works

• Trigger, then re-read. A binlog event is only a trigger. The source re-SELECTs the current row by primary key and serializes that, so the sink always reflects the latest committed state — the binlog value images are never decoded directly. This keeps column types correct and avoids JSON/blob internals.
• Bootstrap. On cold start the source reads SHOW MASTER STATUS to pin the current binlog position, keyset-paginates every in-scope table by primary key (WHERE (pk) > (?) ORDER BY pk LIMIT n), emits each row as an insert, then tails the binlog from the pinned position. A row mutated during the scan is re-emitted by the tail and de-duplicated by the deterministic doc id at the (idempotent) sink.
• Resume. The binlog position (file:pos) is persisted to a file in VS_MYSQL_STATE_DIR (a PVC in Kubernetes), flushed every VS_MYSQL_POS_FLUSH_MS. The source places an internal acknowledgement barrier after all output derived from a binlog position. The dispatcher consumes that barrier and confirms it only after the ordered sink batch prefix is durable; the barrier is never written as a customer document. MySQL sink batches are serialized so two changes to the same document cannot land out of order across parallel bulk requests. Batch size and recomposition concurrency still provide throughput without weakening document ordering. A crash before confirmation replays from the previous persisted position, so recovery is at-least-once and deterministic document IDs make duplicates idempotent. A restart resumes from the confirmed position with no re-bootstrap. If the saved position has been purged from the server (or is gone after a failover to a different primary), the source detects it (ER_MASTER_FATAL_ERROR_READING_BINLOG) and fails closed. It does not silently wipe the cursor and re-bootstrap, because a fresh snapshot cannot emit tombstones for rows deleted before that snapshot. Reconcile the destination or perform an explicit drain/reset, then bootstrap from the new position.
• Deletes become tombstones — the deterministic doc id targets the exact row to remove.

​

The doc-id mapping

​

Run the agent

VS_ROLES=cdc VS_CDC_SOURCE=mysql \
VS_MYSQL_HOST=db.example.net VS_MYSQL_PORT=3306 \
VS_MYSQL_USER=ventstream VS_MYSQL_PASSWORD=secret \
VS_MYSQL_DATABASE=shop \
VS_MYSQL_TABLES=orders,order_items \
VS_MYSQL_SERVER_ID=4000000001 \
VS_MYSQL_STATE_DIR=/var/lib/ventstream/state \
VS_MYSQL_BOOTSTRAP_MODE=snapshot \
VS_OS_ENDPOINT=http://localhost:9200 \
VS_INDEX_TEMPLATE='${header:ventstream.cdc.relation}' \
./target/release/ventstream

​

TLS

VS_MYSQL_TLS_MODE=verify_full
VS_MYSQL_TLS_TRUST_PROVIDER=aws_rds

tls:
  mode: verify_full
  trust:
    provider: aws_rds

​

Key environment variables

​

Denormalization / joins

# VS_JOINS_YAML
joins:
  - name: orders
    primary: { table: shop.orders, pk: id }
    related:
      - id: customer
        table: shop.customers
        pk: id
        join_on: { from: customer_id, to: id }
        embed_as: customer
        cardinality: one
        select: [id, name, tier]
      - id: items
        table: shop.order_items
        pk: id
        join_on: { from: id, to: order_id }
        embed_as: items
        cardinality: many
        sort_by: id
    backfill: { mode: sync_on_miss }

• A row arriving before its related rows is backfilled on demand: the source re-SELECTs the missing rows (backfill: sync_on_miss), so order-of-arrival doesn’t matter.
• Related tables are consumed into the join (they don’t get their own index); only the primary’s denormalized doc is written.
• All primary + related tables are watched/snapshotted automatically, even if VS_MYSQL_TABLES filters.
• Memory-mode joins require VS_JOINS_STATE_DIR on durable storage. The engine refuses to start without it so binlog progress cannot move beyond join state that would be lost on restart.

​

Memory modes: in-memory vs SQL

​

Current limitations

• Joins are raw, not recursive. One level of embedding per related entry (no nested joins-of-joins yet). Cross-source joins are not supported — all tables in a spec come from the one MySQL database.
• Primary key required. Doc ids and tombstones key on the primary key; a table without one is logged with a warning and skipped at bootstrap, and its binlog events are skipped in the tail.
• ROW binlog format required — statement/mixed formats carry no per-row images. The source tails positions, not GTIDs: position resume is robust on a single server, but a failover to a different primary is recovered through the controlled reset and bootstrap path rather than GTID repositioning.
• Schema cache is not invalidated on DDL. Per-table PK/column metadata is cached on first use. An ALTER TABLE that changes the primary key or columns mid-stream is not picked up until the agent restarts — restart after such a migration.
• Binlog-bounded resume. A resume reaches back only as far as the binlog is retained (binlog_expire_logs_seconds). If the saved position has been purged, the source fails closed and requires explicit reconciliation or drain/reset before bootstrap (see How it works).
• MySQL 8.4+ compatible. Uses SHOW BINARY LOG STATUS where available, falling back to the deprecated SHOW MASTER STATUS on older MySQL / MariaDB.