Skip to main content

Collect Connector Gateway telemetry

The Connector Gateway exports OpenTelemetry metrics, traces, and logs to a collector, and can serve Prometheus metrics for scraping. Every exporter is off by default. Configure them under the connector-gateway.telemetry key in the values file you use for the platform chart.

Export to an OpenTelemetry collector​

Point the gateway at your collector's OpenTelemetry Protocol (OTLP) receiver and choose which signals to send:

values.yaml
connector-gateway:
  telemetry:
    otlp:
      endpoint: 'otel-collector.observability.svc:4318'
      insecure: true
      # Metrics
      enabled: true
      # Traces
      tracingEnabled: true
      # Logs
      loggingEnabled: true

The gateway sends OTLP over HTTP with protobuf encoding, so use your collector's HTTP receiver, usually on port 4318. Set insecure: true only for a collector on a trusted, cluster-local network; otherwise leave it false so the connection uses TLS. The chart refuses to render when any of the three signals is on and endpoint is empty.

If you set fleet-wide defaults under global.stacklok.telemetry.otlp, the gateway inherits them. A local endpoint overrides the global one. A signal that the global settings turn on stays on, even if you set it to false locally.

Exported logs carry the same text the gateway writes to standard error, with trace_id and span_id attached when a span is active.

What traces contain​

The gateway samples every request and reports the service name connector-gateway. The gateway is built on ToolHive's Virtual MCP Server (vMCP), so its spans use the vmcp. prefix. Each tool call produces a vmcp.call_tool span with these attributes:

AttributeValue
gen_ai.operation.nameexecute_tool
gen_ai.tool.nameThe tool's name
gen_ai.tool.call.idAn ID that also identifies the call in Tool Usage
enduser.idThe caller's platform user ID
mcp_serverThe connector that served the call

A failed call, or a tool result that reports an error, sets the span's status to error. Child spans cover the call to the connector's backend (backend.call_tool) and any token exchange (token_exchange.authenticate). The gateway propagates W3C traceparent and baggage headers to backends, so a backend that also exports traces joins the same trace.

Scrape Prometheus metrics​

To expose a /metrics endpoint instead of, or in addition to, pushing metrics:

values.yaml
connector-gateway:
  telemetry:
    prometheus:
      enabled: true
    metrics:
      port: 9464

The endpoint runs on its own listener at the metrics.port value (default 9464), separate from the control-plane port. The chart adds a metrics container port, a NetworkPolicy ingress rule for it, and these pod annotations:

prometheus.io/scrape: 'true'
prometheus.io/port: '9464'
prometheus.io/path: /metrics

A Prometheus instance configured for annotation-based pod discovery picks up the gateway automatically. The chart creates no Service port or ServiceMonitor for metrics; if you use the Prometheus Operator, create a PodMonitor that selects the gateway pods and targets the metrics port.

The endpoint exposes Go runtime and process metrics for the gateway. For per-tool call counts, use traces, audit events, or the Tool Usage screen.

Next steps​

Troubleshooting​

The chart fails to render with an OTLP error

otlp.enabled, tracingEnabled, or loggingEnabled is true without an endpoint, either under connector-gateway.telemetry.otlp or inherited from global.stacklok.telemetry.otlp. Set endpoint, or turn off the signals you don't want.

The collector receives nothing

Check that endpoint points at the collector's OTLP/HTTP receiver rather than its gRPC receiver (usually 4317). If the collector serves plain HTTP, set insecure: true. Then check the gateway's logs for export errors:

kubectl logs deployment/stacklok-enterprise-connector-gateway -n stacklok-system