Managed agent on Kubernetes

A managed agent runs the VentStream data plane in your Kubernetes cluster while VentStream Cloud provides configuration, identity, lifecycle operations, and runtime visibility. VentStream Cloud does not create pods or receive source records. Your deployment automation installs the workload, and the agent makes outbound connections to the Cloud gateway.

Prerequisites

A VentStream Cloud organization and environment
ventstreamctl authenticated to that environment
kubectl and Helm access to the target cluster
A storage class for durable engine and agent state
Connector credentials stored in your secret manager
Outbound network access to the Cloud enrollment and control gateways

CDC deployments use one active replica. Realtime roles can be replicated, but each realtime instance needs its own deployment and its own agent key.

1. Create the pipeline and deployment

Create the logical pipeline, then create the deployment that represents this Kubernetes installation:

ventstreamctl pipelines create orders-cdc \
  --description "Orders search projection" \
  --workload-kind cdc \
  --source-kind postgres \
  --sink-kind opensearch

ventstreamctl agents create orders-primary \
  --pipeline orders-cdc

Pipeline and deployment names can be used by later CLI commands, so routine workflows do not require UUIDs.

2. Create the managed configuration

Author a non-secret ventstream.yaml. Credential fields reference environment variables that will come from a Kubernetes Secret:

schema_version: 1
roles: [cdc]

source:
  kind: postgres
  postgres:
    host_ref: env:VS_PG_HOST
    port: 5432
    user_ref: env:VS_PG_USER
    password_ref: env:VS_PG_PASSWORD
    database_ref: env:VS_PG_DATABASE
    publication_ref: env:VS_PG_PUBLICATION
    slot_ref: env:VS_PG_SLOT
    bootstrap:
      mode: snapshot
      chunk_size: 10000
    denormalize_mode: sql

sink:
  kind: opensearch
  opensearch:
    endpoint_ref: env:VS_OS_ENDPOINT
    auth:
      mode: api_key
      api_key_ref: env:VS_OS_API_KEY
    index_routing:
      strategy: fixed
      name: orders

specs:
  joins: projections/orders.yaml

runtime:
  health_listen: 0.0.0.0:4043
  dlq_path: /var/lib/ventstream/dlq.jsonl

Create and validate an immutable revision:

ventstreamctl pipelines configurations create orders-cdc \
  --engine-config ./ventstream.yaml \
  --file projections/orders.yaml=./orders.yaml

ventstreamctl pipelines configurations validate orders-cdc 1 \
  --reason "initial deployment"

ventstreamctl pipelines configurations select orders-cdc 1 \
  --reason "initial deployment"

The selected revision becomes the pipeline’s active desired configuration. A new agent receives it when enrollment completes; no configuration file is baked into the container image.

3. Create connector and agent-key Secrets

Create a namespace and a Secret containing only the environment variables referenced by the managed configuration:

kubectl create namespace ventstream-workloads

kubectl -n ventstream-workloads create secret generic orders-engine-secrets \
  --from-env-file=./orders-engine.env

Mint the deployment’s agent key directly into Kubernetes. The key is shown once; minting a new key revokes the previous one:

ventstreamctl agents key create orders-primary \
  --pipeline orders-cdc | \
kubectl -n ventstream-workloads create secret generic orders-agent-key \
  --from-file=key=/dev/stdin

Do not store the key in source control, CI logs, or a Helm values file. If the key leaks, revoke it (ventstreamctl agents key revoke) and mint a new one — the running agent picks up the replacement through the mounted Secret.

4. Install the public engine with the key

A managed engine is the same public image everyone runs — the agent key is what attaches it to VentStream Cloud. Install the chart with the key Secret:

helm upgrade --install orders-primary \
  oci://ghcr.io/ventstream/charts/ventstream-engine \
  --namespace ventstream-workloads \
  --create-namespace \
  --set image.repository=ghcr.io/ventstream/ventstream \
  --set image.tag=0.1.43 \
  --set fleet.agentKey.secret=orders-agent-key \
  --set engine.envFromSecret=orders-engine-secrets

The key identifies the deployment, so no pipeline ID, deployment ID, or gateway URL values are required. On first start the engine performs an invisible enrollment handshake with the key and from then on follows the pipeline’s selected configuration revision. If the pod loses its state volume, the key simply re-binds a fresh identity; the dashboard records the rotation. Cloud gateways use public WebPKI trust by default, so customers do not need to exchange a CA file.

5. Verify and start

Wait for the pod to enroll and report its runtime state:

kubectl -n ventstream-workloads get pods,pvc

ventstreamctl agents status \
  --pipeline orders-cdc \
  --deployment orders-primary

ventstreamctl operations list --pipeline orders-cdc

New pipelines remain paused until an operator starts them:

ventstreamctl pipelines resume orders-cdc --wait

Confirm the pipeline becomes running in both the dashboard and CLI before producing source changes.

Update configuration

Every edit creates another immutable revision:

ventstreamctl pipelines configurations create orders-cdc \
  --engine-config ./ventstream.yaml \
  --file projections/orders.yaml=./orders.yaml

ventstreamctl pipelines configurations validate orders-cdc 2 \
  --reason "add order status projection"

ventstreamctl pipelines configurations select orders-cdc 2 \
  --reason "add order status projection" \
  --wait

Selection activates the valid revision and delivers it to connected deployments. Use configuration history to roll back to an earlier valid revision.

Routine operations

ventstreamctl pipelines pause orders-cdc \
  --reason "source maintenance" --wait

ventstreamctl pipelines resume orders-cdc --wait

ventstreamctl pipelines reconcile orders-cdc \
  --reason "verify sink parity" --wait

ventstreamctl pipelines drain orders-cdc \
  --reason "retire deployment" --wait

Do not delete the workload PVC during routine upgrades. It contains managed identity, cached desired state, applied configuration, connector cursors, and join state.

Migrating from enrollment tokens

Existing deployments installed with a one-time enrollment grant keep working through the deprecation window. To migrate, mint an agent key for the deployment, replace the enrollment Secret with the key Secret, and switch the release to the public image in one upgrade:

ventstreamctl agents key create orders-primary --pipeline orders-cdc | \
kubectl -n ventstream-workloads create secret generic orders-agent-key \
  --from-file=key=/dev/stdin

helm upgrade orders-primary \
  oci://ghcr.io/ventstream/charts/ventstream-engine \
  --namespace ventstream-workloads \
  --reuse-values \
  --set image.repository=ghcr.io/ventstream/ventstream \
  --set image.tag=0.1.43 \
  --set fleet.agentKey.secret=orders-agent-key \
  --set fleet.enrollment.enabled=false

Get started

Concepts

Connectors

Real-time

Deploy

Guides

Deep dives

Reference

Managed agent on Kubernetes

Prerequisites

1. Create the pipeline and deployment

2. Create the managed configuration

3. Create connector and agent-key Secrets

4. Install the public engine with the key

5. Verify and start

Update configuration

Routine operations

Migrating from enrollment tokens

​Prerequisites

​1. Create the pipeline and deployment

​2. Create the managed configuration

​3. Create connector and agent-key Secrets

​4. Install the public engine with the key

​5. Verify and start

​Update configuration

​Routine operations

​Migrating from enrollment tokens

Prerequisites

1. Create the pipeline and deployment

2. Create the managed configuration

3. Create connector and agent-key Secrets

4. Install the public engine with the key

5. Verify and start

Update configuration

Routine operations

Migrating from enrollment tokens