Version: main 🚧

Install the observability gateway

The observability gateway runs in the vCluster Platform namespace. It contains two containers:

otel-gateway, the Write Gateway that receives OTLP.
query-proxy, the read proxy that serves a Prometheus-compatible query API for PromQL reads.

Platform deploys the gateway from an observability connector Secret. The connector stores the metrics backend endpoints and optional TLS settings.

Prerequisites

Before you create the connector, prepare:

A metrics backend that ingests OTLP metrics and serves the Prometheus HTTP API for queries. Deploy the bundled Prometheus out of the box with the Argo CD integration, or bring your own.
Network access from the Platform namespace to both backend endpoints.
A vCluster Platform license that includes Fleet Observability.
Optional custom gateway serving certificate or backend mTLS Secrets.

note

The observability gateway doesn't install or size the metrics backend for you. Bring your own, or deploy the bundled Prometheus backend with Argo CD as described below.

Deploy the metrics backend with Argo CD

If you use the Argo CD integration, deploy the bundled Prometheus backend with the fleet-observability-prometheus ArgoCDApplicationTemplate instead of installing and sizing a backend by hand. The backend runs on the local cluster, the cluster where Platform itself is installed. Deploy it once, before you create the connector below.

Deploy an ArgoCDApplication that references fleet-observability-prometheus. This deploys Prometheus as a pure OTLP receiver, sized for fleet metrics, into the Platform namespace on the local cluster.

Platform UI
YAML

Create the ArgoCDApplication object in your project namespace:

fleet-metrics-backend.yaml
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
  name: fleet-metrics-backend
  namespace: p-my-project          # replace with your project namespace
spec:
  displayName: "Fleet metrics backend"
  destination:
    cluster:
      name: loft-cluster           # the reserved name for Platform's local cluster; don't change this
  templateRef:
    name: fleet-observability-prometheus
  parameters:
    destinationNamespace: vcluster-platform
    prometheusStorageSize: 20Gi
    project: default

Click Control Plane Clusters in the left sidebar under the infrastructure section, then click the local cluster (named loft-cluster by default) to open its detail page.
Select the ArgoCD Apps tab and click .
In the Basics section, enter a Display Name such as Fleet metrics backend.
From the Argo CD Application Template dropdown, select the Fleet Observability: Prometheus template.
Fill the Template Parameters. Keep destinationNamespace as vcluster-platform, and adjust the storage size or chart version if needed.
Click .

Parameter	Default	Description
`destinationNamespace`	`vcluster-platform`	Namespace for the Prometheus backend.
`project`	`default`	Argo CD project that owns the application.
`repoURL`	Prometheus community Helm repo	Helm repository for the Prometheus chart.
`version`	`29.14.0`	Prometheus Helm chart version.
`prometheusStorageSize`	`20Gi`	Persistent volume size for Prometheus storage.
`metricsBackendTLSSecret`	(empty)	Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named `metrics-backend-mtls`.

The connector you create below must use these endpoints when the template is deployed in its default namespace:

stringData:
  metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp
  metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local
  metricsBackendCertSecretName: metrics-backend-mtls

The /api/v1/otlp path is required for OTLP ingestion. If you change destinationNamespace, replace vcluster-platform in both endpoints. If you supply metricsBackendTLSSecret, use that Secret name instead of metrics-backend-mtls.

Create the connector Secret

Create the Fleet Observability connector in the Platform namespace. The connector ID can be any valid Secret name. Platform derives the gateway resource name as gateway-<connector-id>. The examples use fleet-observability, which produces gateway-fleet-observability.

The loft.sh/connector-type: observability label is required. Platform only recognizes and reconciles a Secret as an observability connector when this label is present.

Mark the connector as the fleet default with the platform.vcluster.com/fleet-observability-connector label set to "true". Only one connector can be the fleet default.

Platform UI
YAML

fleet-observability-connector.yaml
apiVersion: v1
kind: Secret
metadata:
  name: fleet-observability
  namespace: vcluster-platform
  labels:
    loft.sh/connector-type: observability
    platform.vcluster.com/fleet-observability-connector: "true"
type: Opaque
stringData:
  metricsBackendOtlpEndpoint: https://metrics.example.com/otlp
  metricsBackendPromQLEndpoint: https://metrics.example.com

Apply the Secret:

kubectl apply -f fleet-observability-connector.yaml

Click Connectors and select the Observability tab.
Click .
In the Display name field, enter a human-readable name for the connector. The Observability Connector ID is auto-generated from the display name. The connector ID determines the gateway resource names. The first observability connector is automatically marked as the fleet default.
In the Metrics Backend OTLP Endpoint field, enter the OTLP endpoint that receives metrics from the Write Gateway.
In the Metrics Backend PromQL Endpoint field, enter the Prometheus HTTP API endpoint queried by the Query Proxy.
Optionally configure TLS for the metrics backend and the gateway. See Connector keys for the available settings.
Click .

Platform reconciles the following resources:

Deployment/gateway-fleet-observability
Service/gateway-fleet-observability
Grafana datasource ConfigMaps for fleet, control plane cluster, and tenant cluster views when Grafana datasource reconciliation is enabled.

Connector keys

Key	Required	Description
`metricsBackendOtlpEndpoint`	Yes	OTLP endpoint that receives metrics from the Write Gateway. Provide the base URL only; the gateway appends `/v1/metrics` (for example, `https://prom-prometheus-server.vcluster-platform/api/v1/otlp` receives writes at `/api/v1/otlp/v1/metrics`).
`metricsBackendPromQLEndpoint`	Yes	Prometheus HTTP API endpoint queried by the Query Proxy.
`metricsBackendInsecureSkipVerify`	No	Set to `"true"` only for test backends with unverifiable TLS.
`metricsBackendServerName`	No	TLS server name override for the metrics backend.
`observabilityGatewayCertSecretName`	No	Custom Secret with `tls.crt` and `tls.key` for gateway serving TLS. Include `ca.crt` when clients must trust a private CA. Platform creates a self-signed serving certificate when this key is omitted.
`metricsBackendCertSecretName`	No	Secret with `ca.crt`, `tls.crt`, and `tls.key` for mTLS to the backend. The gateway trusts the backend with `ca.crt` and presents the certificate and key as its client identity.
`gatewayResources`	No	YAML `resources` block for the Write Gateway container.
`queryProxyResources`	No	YAML `resources` block for the Query Proxy container.
`serviceType`	No	Gateway Service type. Valid values are `ClusterIP`, the default, and `LoadBalancer`.
`serviceAnnotations`	No	YAML map of annotations applied to a `LoadBalancer` gateway Service.
`grafanaUrl`	No	Grafana upstream URL used by the Platform `/grafana/` reverse proxy.
`grafanaInsecureSkipVerify`	No	Set to `"true"` only to test a Grafana endpoint with an unverifiable certificate.
`grafanaCertSecretName`	No	Grafana serving certificate Secret. Platform uses its `ca.crt` to trust the Grafana upstream.
`fleetObservabilityDashboardUid`	No	UID of the fleet dashboard embedded by the Fleet Observability page.

Bring-your-own Grafana requirements

When you point grafanaUrl at a Grafana you manage instead of the bundled Grafana, that Grafana must:

Serve from the /grafana subpath. Set server.root_url and server.serve_from_sub_path so Grafana works behind the Platform /grafana/ reverse proxy. See serve_from_sub_path.
Allow embedding. Set security.allow_embedding = true so the Fleet Observability page and each tenant cluster's Observability tab can embed the dashboards in an iframe. See allow_embedding.

Customize the gateway serving certificate

The gateway always serves OTLP and query traffic over TLS. When you don't specify a certificate Secret, Platform creates and manages a self-signed Secret named gateway-<connector-id>-tls.

To use your own certificate, create a Secret in the Platform namespace:

gateway-tls.yaml
apiVersion: v1
kind: Secret
metadata:
  name: fleet-gateway-tls
  namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

Reference it from the connector:

stringData:
  observabilityGatewayCertSecretName: fleet-gateway-tls

Edge collectors use HTTPS for OTLP HTTP or TLS for OTLP gRPC in both the managed and custom certificate cases. Distribute ca.crt to collectors that don't already trust the issuing CA. Grafana datasources generated by Platform trust the ca.crt from the gateway serving certificate Secret. If a custom certificate is signed by a public CA, ca.crt can be omitted and clients use their system trust stores.

Expose the gateway with a LoadBalancer

The gateway Service defaults to ClusterIP. To make the gateway reachable from collectors and query clients outside the Platform cluster, set serviceType to LoadBalancer. You can optionally apply provider-specific Service annotations:

stringData:
  serviceType: LoadBalancer
  serviceAnnotations: |
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal

For a Platform-managed certificate, the controller waits for the LoadBalancer address and adds it to the certificate as a subject alternative name before deploying the gateway. This means clients that verify TLS trust the address without extra configuration. For a self-signed certificate, distribute ca.crt as described in Customize the gateway serving certificate. A certificate signed by a public CA needs no extra trust configuration.

Retrieve the load balancer address

The cloud provider assigns the address asynchronously, so it stays empty until provisioning finishes. Read it from the gateway Service:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" \
  -o jsonpath='{range .status.loadBalancer.ingress[*]}{.hostname}{.ip}{"\n"}{end}'

The provider reports either a hostname (for example, AWS) or an IP address (for example, GCP and Azure). Build the reachable endpoints from that address:

Endpoint	Port	Purpose
`https://<loadbalancer-address>:4318`	`4318`	OTLP HTTP writes. Use this for collectors.
`https://<loadbalancer-address>:4317`	`4317`	OTLP gRPC writes. The gateway accepts gRPC, but the bundled collector sends over HTTP, so use `4318` for collectors.
`https://<loadbalancer-address>:8081`	`8081`	Prometheus HTTP API through the Query Proxy. Use this for external query clients.

Use this address wherever a collector or query client connects from outside the Platform cluster.

Use backend mTLS

To authenticate the gateway to your backend with mTLS, create a Secret:

backend-client-tls.yaml
apiVersion: v1
kind: Secret
metadata:
  name: metrics-backend-client-tls
  namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----

Reference it from the connector:

stringData:
  metricsBackendCertSecretName: metrics-backend-client-tls

Verify the gateway

Check that the gateway pod is running:

CONNECTOR_ID=fleet-observability
kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID"

Check the service ports:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID"

Expected service ports:

Port	Name	Purpose
`4317`	`otlp`	OTLP gRPC writes.
`4318`	`otlp-http`	OTLP HTTP writes.
`8081`	`query`	Prometheus HTTP API through the Query Proxy.

Next, create metrics access keys in Configure metrics access.

Prerequisites​

Deploy the metrics backend with Argo CD​

Create the connector Secret​

Connector keys​

Customize the gateway serving certificate​

Expose the gateway with a LoadBalancer​

Retrieve the load balancer address​

Use backend mTLS​

Verify the gateway​