Skip to main content
The Meilisearch sink materializes each VentStream output document into a Meilisearch index. Source inserts and updates become add-or-replace documents, source deletes remove the document, and table truncates clear the index. Delivery is confirmed through Meilisearch’s asynchronous task API: source progress only advances after the indexing task reports succeeded, so a crash or outage on either side replays safely instead of losing writes. Deletes propagate natively — the classic failure mode of poll-based sync (deleted rows haunting the search index forever) does not exist here, because the engine reads the database’s replication log, which includes every delete.

​
Requirements

  • Meilisearch v1.x (validated against v1.52). The instance’s task queue and payload limits apply; the sink’s default 16 MiB request ceiling stays well under Meilisearch’s default 100 MB payload cap.
  • An API key with documents, indexes, tasks, and settings permissions on the target indexes. Use a scoped key, never the master key, for production.
  • Every event needs a stable document id (the ventstream.doc.id header). Postgres pipelines get this from a joins projection — see Postgres quickstart below. MongoDB, MySQL, Neo4j, and Kafka sources stamp it natively.

​
How documents are shaped

Meilisearch primary keys only allow [A-Za-z0-9_-], so the sink stores the canonical VentStream document id (orders:["123"]) base64url-encoded in the primary-key attribute (default _vs_pk), and keeps the readable original as a _vs_id field on the document:
Ids longer than Meilisearch’s 511-byte cap fall back to a SHA-256 digest. _vs_id is filterable like any other attribute if you declare it.

​
Ordering

Meilisearch has no external document versioning, so the sink writes with a concurrency of one and preserves source order through sequential same-index runs. Meilisearch’s own server-side task batching keeps throughput high; the connector test matrix delivers a 3,000-document single-transaction flood in about two seconds.

​
Configuration

That is a complete production configuration: per-table indexes named vs_<table>, auto-created on first write, primary key in _vs_pk. The full option set:
Environment fallbacks when no file config is present: VS_SINK=meilisearch, VS_MEILI_ENDPOINT, VS_MEILI_API_KEY, VS_MEILI_INDEX_PREFIX, VS_MEILI_INDEX (fixed routing).

​
Index routing

  • by_output_relation (default) — one index per source table: vs_shop_2Eorders for shop.orders. Characters outside Meilisearch’s index charset are escaped injectively (_ doubles, other bytes become _HH hex).
  • by_projection_target — route by the projection target declared in the joins spec.
  • fixed — every document into one index.

​
Managed settings

With fixed routing you can declare filterable_attributes and sortable_attributes; the sink applies them at startup and waits for the settings task to succeed before the pipeline starts. Attributes you do not declare are never modified, so hand-tuned relevancy settings survive.
MySQL DECIMAL columns materialize as precision-preserving strings ("49.00"). Declare a numeric cast in your projection, or expect lexicographic ordering if you make such a field sortable.

​
Startup checks

Before the pipeline starts, the sink verifies reachability and authentication, and for fixed routing validates that an existing index’s primaryKey matches the configured field — a mismatch blocks startup with drain-and-rebootstrap guidance instead of silently writing conflicting documents. A missing API key or bad endpoint fails fast at boot.

​
Failure behavior

  • Transport errors, 5xx, 429, and capacity-class task failures (task_queue_full, no_space_left_on_device) retry with jittered exponential backoff; source progress is pinned until delivery succeeds.
  • Deleting a document that does not exist, or deleting against an index that does not exist yet, is treated as the no-op it is.
  • An index deleted out from under the sink is recreated on the next write when auto_create_indexes is on.
  • Events the sink can prove are undeliverable client-side (payload is not a JSON object, missing document id) are routed to the dead-letter queue with per-event reasons and exact offsets; everything else fails closed rather than guessing.

​
Quickstart: Postgres to Meilisearch

Postgres events need a joins projection to carry a stable document id — even a single table with no joins:
Set VS_JOINS_STATE_DIR to a durable directory, create the publication and slot, and start the engine:
The snapshot bootstrap materializes existing rows; from then on inserts, updates, and deletes stream continuously. Delete a row and watch the document leave your search results in well under a second. Multi-table projections (embedded one-to-one objects and one-to-many arrays, with child changes recomposing parent documents) work exactly as they do for the OpenSearch sink — the same joins spec drives both.