Managed agent on Kubernetes

A managed agent runs the VentStream data plane in your Kubernetes cluster while VentStream Cloud provides configuration, identity, lifecycle operations, and runtime visibility. VentStream Cloud does not create pods or receive source records. Your deployment automation installs the workload, and the agent makes outbound connections to the Cloud gateway.

Prerequisites

A VentStream Cloud organization and environment
ventstreamctl authenticated to that environment
kubectl and Helm access to the target cluster
A storage class for durable engine and agent state
Connector credentials stored in your secret manager
Outbound network access to the Cloud enrollment and control gateways

CDC deployments use one active replica. Realtime workloads can use multiple deployments when each instance has its own managed identity.

1. Create the pipeline and deployment

Create the logical pipeline, then create the deployment that represents this Kubernetes installation:

ventstreamctl pipelines create orders-cdc \
  --description "Orders search projection" \
  --workload-kind cdc \
  --source-kind postgres \
  --sink-kind opensearch

ventstreamctl agents create orders-primary \
  --pipeline orders-cdc

Pipeline and deployment names can be used by later CLI commands, so routine workflows do not require UUIDs.

2. Create the managed configuration

Author a non-secret ventstream.yaml. Credential fields reference environment variables that will come from a Kubernetes Secret:

schema_version: 1
roles: [cdc]

source:
  kind: postgres
  postgres:
    host_ref: env:VS_PG_HOST
    port: 5432
    user_ref: env:VS_PG_USER
    password_ref: env:VS_PG_PASSWORD
    database_ref: env:VS_PG_DATABASE
    publication_ref: env:VS_PG_PUBLICATION
    slot_ref: env:VS_PG_SLOT
    bootstrap:
      mode: snapshot
      chunk_size: 10000
    denormalize_mode: sql

sink:
  kind: opensearch
  opensearch:
    endpoint_ref: env:VS_OS_ENDPOINT
    auth:
      mode: api_key
      api_key_ref: env:VS_OS_API_KEY
    index_routing:
      strategy: fixed
      name: orders

specs:
  joins: projections/orders.yaml

runtime:
  health_listen: 0.0.0.0:4043
  dlq_path: /var/lib/ventstream/dlq.jsonl

Create and validate an immutable revision:

ventstreamctl pipelines configurations create orders-cdc \
  --engine-config ./ventstream.yaml \
  --file projections/orders.yaml=./orders.yaml

ventstreamctl pipelines configurations validate orders-cdc 1 \
  --reason "initial deployment"

ventstreamctl pipelines configurations select orders-cdc 1 \
  --reason "initial deployment"

The selected revision becomes the pipeline’s active desired configuration. A new agent receives it when enrollment completes; no configuration file is baked into the container image.

3. Create connector and enrollment Secrets

Create a namespace and a Secret containing only the environment variables referenced by the managed configuration:

kubectl create namespace ventstream-workloads

kubectl -n ventstream-workloads create secret generic orders-engine-secrets \
  --from-env-file=./orders-engine.env

Generate a short-lived, single-use enrollment grant directly into Kubernetes:

ventstreamctl agents enroll-token create orders-primary \
  --pipeline orders-cdc | \
kubectl -n ventstream-workloads create secret generic orders-enrollment \
  --from-file=grant=/dev/stdin

Do not store the grant in source control, CI logs, or a Helm values file.

4. Generate the deployment values

Ask VentStream Cloud for the approved chart, managed image digest, gateway addresses, trust mode, pipeline ID, and deployment ID:

ventstreamctl agents manifest orders-primary \
  --pipeline orders-cdc \
  --target helm \
  --engine-secret orders-engine-secrets \
  --enrollment-secret orders-enrollment \
  --output orders-primary.values.yaml

The generated file contains no credentials or enrollment grant. It pins the managed image by digest and names the existing Kubernetes Secrets. Its header contains the chart URL, chart version, and suggested Helm release name. Review the file in deployment automation, then install it using those header values:

helm upgrade --install orders-primary \
  <chart-url-from-generated-header> \
  --version <chart-version-from-generated-header> \
  --namespace ventstream-workloads \
  --create-namespace \
  --values orders-primary.values.yaml

Cloud gateways use public WebPKI trust by default, so customers do not need to exchange a CA file. If a workspace is intentionally configured with private gateway trust, ventstreamctl agents manifest requires the corresponding Kubernetes trust Secret explicitly.

5. Verify and start

Wait for the pod to enroll and report its runtime state:

kubectl -n ventstream-workloads get pods,pvc

ventstreamctl agents status \
  --pipeline orders-cdc \
  --deployment orders-primary

ventstreamctl operations list --pipeline orders-cdc

New pipelines remain paused until an operator starts them:

ventstreamctl pipelines resume orders-cdc --wait

Confirm the pipeline becomes running in both the dashboard and CLI before producing source changes.

Update configuration

Every edit creates another immutable revision:

ventstreamctl pipelines configurations create orders-cdc \
  --engine-config ./ventstream.yaml \
  --file projections/orders.yaml=./orders.yaml

ventstreamctl pipelines configurations validate orders-cdc 2 \
  --reason "add order status projection"

ventstreamctl pipelines configurations select orders-cdc 2 \
  --reason "add order status projection" \
  --wait

Selection activates the valid revision and delivers it to connected deployments. Use configuration history to roll back to an earlier valid revision.

Routine operations

ventstreamctl pipelines pause orders-cdc \
  --reason "source maintenance" --wait

ventstreamctl pipelines resume orders-cdc --wait

ventstreamctl pipelines reconcile orders-cdc \
  --reason "verify sink parity" --wait

ventstreamctl pipelines drain orders-cdc \
  --reason "retire deployment" --wait

Do not delete the workload PVC during routine upgrades. It contains managed identity, cached desired state, applied configuration, connector cursors, and join state.

Get started

Concepts

Connectors

Guides

Engineering

VentStream Cloud

Deploy

Reference

Managed agent on Kubernetes

Prerequisites

1. Create the pipeline and deployment

2. Create the managed configuration

3. Create connector and enrollment Secrets

4. Generate the deployment values

5. Verify and start

Update configuration

Routine operations

​Prerequisites

​1. Create the pipeline and deployment

​2. Create the managed configuration

​3. Create connector and enrollment Secrets

​4. Generate the deployment values

​5. Verify and start

​Update configuration

​Routine operations

Prerequisites

1. Create the pipeline and deployment

2. Create the managed configuration

3. Create connector and enrollment Secrets

4. Generate the deployment values

5. Verify and start

Update configuration

Routine operations