Search over a graph has a shape problem before it has a sync problem. OpenSearch wants self-contained documents; Neo4j gives you nodes whose meaning lives in their relationships. A product document worth searching carries its category, and the category is a different node shared by hundreds of products. So any pipeline you build has to answer two questions at once: how does a subgraph become a document, and who rewrites all those documents when the shared node changes? In VentStream both answers come from the same place.
You declare the document you want as a Cypher query. That projection is the mapping, and because the engine knows which nodes each document was built from, it also knows exactly which documents to rebuild when any of those nodes move.
The guide: Neo4j to OpenSearch in five steps
Neo4j’s change data capture feeds the pipeline, a projection spec shapes it, and a single Rust binary under the Apache-2.0 license does the work. Here is the full path:
1. Enable CDC on the database
The prerequisite to know up front: change data capture needs Neo4j 5.17 or later, Enterprise edition. Turning it on is a single statement against the system database:
ALTER DATABASE neo4j SET OPTION txLogEnrichment 'DIFF';2. Declare the document as a query
This step is where the graph-to-document question gets answered, and it is the heart of the whole setup. For each node label you want indexed, you write the Cypher that turns one node into one document. The engine anchors every Product node as p and runs your query from there; you return primaryEid and a doc map, and that map is, byte for byte, the OpenSearch document:
# denormalize.yaml
denormalize:
- primary_label: Product
output_table: products
cypher: |
OPTIONAL MATCH (p)-[:IN_CATEGORY]->(c:Category)
RETURN elementId(p) AS primaryEid, {
id: p.id,
name: p.name,
price: p.price,
category: CASE WHEN c IS NOT NULL THEN {id: c.id, name: c.name} ELSE null END
} AS docAnything Cypher can reach, the document can contain. Here that is just one hop to the category, embedded as a nested object.
3. Point the engine at both ends
The engine config names the source, the projection spec, and the sink:
schema_version: 1
roles: [cdc]
source:
kind: neo4j
neo4j:
uri_ref: env:VS_NEO4J_URI
user_ref: env:VS_NEO4J_USER
password_ref: env:VS_NEO4J_PASSWORD
database: neo4j
specs:
neo4j_denormalize: denormalize.yaml
sink:
kind: opensearch
opensearch:
endpoint_ref: env:VS_OS_ENDPOINT
index_routing:
strategy: fixed
name: productsAgainst an open development cluster this runs as written, since sink auth is optional. A secured cluster takes an auth block (mode: basic or mode: api_key with the matching *_ref entries) together with TLS verification on https endpoints; the full set is in the sink documentation. The same sink also speaks Elasticsearch: change kind: elasticsearch and keep the rest. On the graph side, encrypted Bolt is a URI scheme away — neo4j+s:// with verify_full, described in the source documentation.
4. Set the secrets
None of the files so far contain a credential. Each env: entry is a promise that a variable with that name will exist when the engine starts, and the engine looks nowhere else. Supply them however your deployment already supplies secrets; on a laptop the simplest thing is an env file the shell exports before launch:
# .env
VS_NEO4J_URI=bolt://graph.internal:7687
VS_NEO4J_USER=neo4j
VS_NEO4J_PASSWORD=…
VS_OS_ENDPOINT=https://search.internal:92005. Install and run
The installer targets macOS and Linux; on Windows, run it under WSL2 or use the container image instead.
curl -fsSL https://ventstream.dev/install.sh | sh
set -a && source ./.env && set +a
VS_ENGINE_CONFIG=./ventstream.yaml ventstreamWhat you see
On startup the engine captures a CDC cursor, then bootstraps the index by scanning every Product node through your projection. Once the scan completes, it polls the CDC log from that cursor and recomposes affected documents as the graph changes. Property updates land on the document they belong to, a freshly created node shows up already composed, and DETACH DELETE takes its document out of the index. In our verification run, a 2,000-product bootstrap finished with exact document parity and zero warnings.
The payoff is what happens on a shared node. In our test, renaming a single Category node caused every one of the 400 products embedding it to recompose in OpenSearch, and the graph-side count matched the index-side count exactly. That is the projection doing the work dual-write code never manages: the write touched one node, and the engine worked out the 400 documents whose contents it changed.
Relational sources get the same composed-document treatment: for PostgreSQL and MySQL the equivalent mechanism is a joins spec.
Why it stays correct
The mechanics underneath the recomposition:
- Documents are keyed by the primary node’s element id, so a recomposition always overwrites the document it replaces. However many times a product is rebuilt, the index holds one copy of it.
- Deletes travel with that same id, so removing a node removes precisely its document and nothing adjacent to it.
- The engine maintains a dependency map from nodes and relationships to the documents built from them, which is how one category rename fans out to exactly the 400 products that embed it. A hot-node threshold bounds this: a node referenced by a huge number of documents won’t trigger unbounded recomposition on every touch. The hot-nodes writeup goes deep on that mechanism.
How fast is it?
Composition speed scales with the resources and the projection you give the engine, so treat any single number as a point on a curve. In our verified bench runs, composing 4 million documents from a graph through Cypher projections proceeded at roughly 464k documents per minute. In steady state, a graph change surfaces in search within seconds: the CDC log is polled on a 500ms default interval, and the index refresh adds its usual beat on top.
Why VentStream for this sync
Graph-to-search is where most sync setups fall apart, because a composed document has many parents. The engineering that handles that here:
- Dependency-tracked recomposition. The engine knows which documents embed which nodes and relationships. When the shared
Categorychanged, exactly the 400 dependent products recomposed — not the whole index, and not zero. - Hot-node protection. A node embedded by tens of thousands of documents would otherwise turn every touch into a mass recompute. Fan-out anchoring is thresholded (
VS_NEO4J_HOT_NODE_THRESHOLD), and live multi-hop recomposition runs in bounded, concurrent Cypher chunks instead of one unbounded query — the deep dive covers a verified 59,498-document cascade. - No gap between snapshot and tail. The CDC cursor is captured before the bootstrap scan starts, so changes committed while the scan runs are replayed afterwards rather than lost between the two phases.
- Idempotent, fenced writes.Documents are keyed by the primary node’s element id, and each write carries an external version derived from the transaction watermark — a recomposition that arrives late cannot overwrite a newer one, and replays land on the same document instead of duplicating it.
- The cursor advances only on confirmed writes. The CDC position moves after OpenSearch acknowledges the bulk request, so a crash at any point resumes by redoing unconfirmed work — safe, because the writes are idempotent.
Run it wherever you like
All of the above is the open-source engine doing its normal work, and it will do the same on any host you give it. Attaching that engine to VentStream Cloud takes one agent key and buys you centrally managed configuration, health, and operations while your graph, your cluster, and the traffic between them stay in your infrastructure.
Browse the source, start from the quickstart, or sign up and enroll an agent against your own graph.