Skip to main content
The SurrealDB sink writes each change as a full-replace upsert on a native record id, so your tables in SurrealDB mirror the source — flat rows or composed documents with embedded joins — and stay consistent through updates, deletes, primary-key changes, and truncates. SurrealDB commits synchronously: a confirmed write is durable, with no task queue to poll. Requires SurrealDB 3.x.

​
1. Provision the target (one-time)

VentStream never needs root. In Surrealist (or any client with admin authority — for a local instance, surreal sql -u root -p root works), create the namespace, database, and a database-scoped user:
Those are the credentials the sink runs with: enough to write records and define table-level indexes, nothing more. Scoped users authenticate via /signin; the sink handles tokens and refresh automatically. If the database is missing at startup, the sink fails with exactly this DDL in the error message. (For a throwaway local instance you can instead set auto_create_database: true and connect with root — dev only.)

​
2. A complete pipeline

The sink is one half of a pipeline file; any source provides the other. A minimal, complete Postgres → SurrealDB config:
*_ref: env:NAME resolves each value from the environment at startup, so no secret lives in the file. Export them and run the engine:
The engine snapshots every published table into SurrealDB, then tails the WAL. Each published table becomes a SurrealDB table named after its relation — public.orders lands in table orders — and table_prefix: "pg-" would make that pg-orders. Verify in Surrealist:
For composed documents (an order carrying its customer and line items), add a joins spec exactly as in the Postgres source guide — everything upstream of the sink, including joins and schema-drift handling, is identical for every sink.

​
Record identity

VentStream’s deterministic doc id table:["pk",…] maps directly onto SurrealDB’s array record ids — orders:['299'], composite keys included — so upserts are idempotent and deletes always find their record. The canonical id is stamped on every document as _vs_id. A source column named id is preserved as source_id (SurrealDB reserves id for the record id itself). Declare embedding fields to make them KNN-searchable — the sink ensures the HNSW index at startup:
Embedding arrays flow through documents unchanged; query with WHERE embedding <|10,40|> $vec.

​
Large joined tables

For 1:many joined pipelines, declare the embedded join paths so child deletes filter a flat materialized field instead of evaluating a per-row closure (~5x cheaper today, index-ready in future SurrealDB versions):

​
Semantics to know

  • Concurrent writers converge. SurrealDB uses optimistic transactions; the sink classifies conflicts as transient and retries the ordered tail — replays are idempotent, so external writers hammering the same records cost retries, not corruption.
  • Full-replace materialization. Documents are replaced, not merged: fields written by other applications on the same records are overwritten on the next sync. Give VentStream its own tables.
  • table_routing.mode: fixed funnels every relation into one named table (records keep fully-qualified ids so relations can’t collide); a TRUNCATE of any relation then clears that shared table — prefer the default per-relation routing unless you need this.

​
Troubleshooting

  • namespace/database is not provisioned — run the DDL from step 1, or set auto_create_database: true with elevated credentials (dev only).
  • HTTP 401 at startup — wrong credentials, or the user was defined at a different scope than the configured namespace/database.
  • Writes retry with transient statement failure — normal under write contention; sustained retries mean another process is hammering the same records.