> ## Documentation Index
> Fetch the complete documentation index at: https://ventstream.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Meilisearch sink

> Keep Meilisearch indexes continuously in sync with your database — inserts, updates, and deletes, no polling, no re-indexing.

The Meilisearch sink materializes each VentStream output document into a
Meilisearch index. Source inserts and updates become add-or-replace
documents, source deletes remove the document, and table truncates clear the
index. Delivery is confirmed through Meilisearch's asynchronous task API:
source progress only advances after the indexing task reports `succeeded`,
so a crash or outage on either side replays safely instead of losing writes.

Deletes propagate natively — the classic failure mode of poll-based sync
(deleted rows haunting the search index forever) does not exist here,
because the engine reads the database's replication log, which includes
every delete.

## Requirements

* Meilisearch v1.x (validated against v1.52). The instance's task queue and
  payload limits apply; the sink's default 16 MiB request ceiling stays
  well under Meilisearch's default 100 MB payload cap.
* An API key with documents, indexes, tasks, and settings permissions on
  the target indexes. Use a scoped key, never the master key, for
  production.
* Every event needs a stable document id (the `ventstream.doc.id` header).
  Postgres pipelines get this from a joins projection — see
  [Postgres quickstart](#quickstart-postgres-to-meilisearch) below.
  MongoDB, MySQL, Neo4j, and Kafka sources stamp it natively.

## How documents are shaped

Meilisearch primary keys only allow `[A-Za-z0-9_-]`, so the sink stores the
canonical VentStream document id (`orders:["123"]`) base64url-encoded in the
primary-key attribute (default `_vs_pk`), and keeps the readable original
as a `_vs_id` field on the document:

```json theme={null}
{
  "_vs_pk": "c2hvcC5vcmRlcnM6WyIxMDAiXQ",
  "_vs_id": "shop.orders:[\"100\"]",
  "order_id": 100,
  "status": "shipped",
  "customer": { "name": "Ada Lovelace", "tier": "gold" },
  "items": [ { "sku": "ESP-01", "qty": 1 } ]
}
```

Ids longer than Meilisearch's 511-byte cap fall back to a SHA-256 digest.
`_vs_id` is filterable like any other attribute if you declare it.

## Ordering

Meilisearch has no external document versioning, so the sink writes with a
concurrency of one and preserves source order through sequential
same-index runs. Meilisearch's own server-side task batching keeps
throughput high; the connector test matrix delivers a 3,000-document
single-transaction flood in about two seconds.

## Configuration

```yaml theme={null}
sink:
  kind: meilisearch
  meilisearch:
    endpoint_ref: env:VS_MEILI_ENDPOINT
    api_key_ref: env:VS_MEILI_API_KEY
```

That is a complete production configuration: per-table indexes named
`vs_<table>`, auto-created on first write, primary key in `_vs_pk`.

The full option set:

```yaml theme={null}
sink:
  kind: meilisearch
  meilisearch:
    endpoint_ref: env:VS_MEILI_ENDPOINT
    api_key_ref: env:VS_MEILI_API_KEY
    # by_output_relation (default) | by_projection_target | fixed
    index_routing:
      mode: fixed
      index: catalog
    index_prefix: "vs_"          # [A-Za-z0-9_-] only
    auto_create_indexes: true
    primary_key_field: "_vs_pk"
    max_batch_docs: 2000
    max_batch_bytes: 16777216
    task_deadline_ms: 120000     # give up polling a task after this long
    request_timeout_ms: 30000
    settings:                    # fixed routing only; only declared
      filterable_attributes: []  # attributes are ever touched
      sortable_attributes: []
    tls: {}                      # shared database TLS policy block
    insecure_tls: false
```

Environment fallbacks when no file config is present: `VS_SINK=meilisearch`,
`VS_MEILI_ENDPOINT`, `VS_MEILI_API_KEY`, `VS_MEILI_INDEX_PREFIX`,
`VS_MEILI_INDEX` (fixed routing).

### Index routing

* `by_output_relation` (default) — one index per source table:
  `vs_shop_2Eorders` for `shop.orders`. Characters outside Meilisearch's
  index charset are escaped injectively (`_` doubles, other bytes become
  `_HH` hex).
* `by_projection_target` — route by the projection target declared in the
  joins spec.
* `fixed` — every document into one index.

### Managed settings

With fixed routing you can declare `filterable_attributes` and
`sortable_attributes`; the sink applies them at startup and waits for the
settings task to succeed before the pipeline starts. Attributes you do not
declare are never modified, so hand-tuned relevancy settings survive.

<Note>
  MySQL `DECIMAL` columns materialize as precision-preserving strings
  (`"49.00"`). Declare a numeric cast in your projection, or expect
  lexicographic ordering if you make such a field sortable.
</Note>

## Startup checks

Before the pipeline starts, the sink verifies reachability and
authentication, and for fixed routing validates that an existing index's
`primaryKey` matches the configured field — a mismatch blocks startup with
drain-and-rebootstrap guidance instead of silently writing conflicting
documents. A missing API key or bad endpoint fails fast at boot.

## Failure behavior

* Transport errors, 5xx, 429, and capacity-class task failures
  (`task_queue_full`, `no_space_left_on_device`) retry with jittered
  exponential backoff; source progress is pinned until delivery succeeds.
* Deleting a document that does not exist, or deleting against an index
  that does not exist yet, is treated as the no-op it is.
* An index deleted out from under the sink is recreated on the next write
  when `auto_create_indexes` is on.
* Events the sink can prove are undeliverable client-side (payload is not
  a JSON object, missing document id) are routed to the dead-letter queue
  with per-event reasons and exact offsets; everything else fails closed
  rather than guessing.

## Quickstart: Postgres to Meilisearch

Postgres events need a joins projection to carry a stable document id —
even a single table with no joins:

```yaml theme={null}
# joins.yaml
joins:
  - name: products
    primary:
      table: public.products
      pk: id
    related: []
    state:
      backend: memory
```

```yaml theme={null}
# ventstream.yaml
schema_version: 1
roles: [cdc]
source:
  kind: postgres
  postgres:
    host_ref: env:VS_PG_HOST
    user_ref: env:VS_PG_USER
    password_ref: env:VS_PG_PASSWORD
    database_ref: env:VS_PG_DATABASE
    publication: vs_pub
    slot: vs_slot
    bootstrap:
      mode: snapshot
    denormalize_mode: sql
    sink_reverse_lookup: false
sink:
  kind: meilisearch
  meilisearch:
    endpoint_ref: env:VS_MEILI_ENDPOINT
    api_key_ref: env:VS_MEILI_API_KEY
specs:
  joins: joins.yaml
runtime:
  dlq_path: ./data/dlq.jsonl
```

Set `VS_JOINS_STATE_DIR` to a durable directory, create the publication and
slot, and start the engine:

```sql theme={null}
CREATE PUBLICATION vs_pub FOR TABLE public.products;
SELECT pg_create_logical_replication_slot('vs_slot', 'pgoutput');
```

```bash theme={null}
VS_ENGINE_CONFIG=./ventstream.yaml VS_JOINS_STATE_DIR=./joins-state ventstream
```

The snapshot bootstrap materializes existing rows; from then on inserts,
updates, and deletes stream continuously. Delete a row and watch the
document leave your search results in well under a second.

Multi-table projections (embedded one-to-one objects and one-to-many
arrays, with child changes recomposing parent documents) work exactly as
they do for the [OpenSearch sink](/docs/connectors/sinks/opensearch) — the same
joins spec drives both.