Datadog

Spice can be monitored with Datadog using the Spice Metrics Endpoint and pre-built dashboards available in the Spice repository.

Datadog Agent Configuration

Prerequisite: Datadog Agent version 6.5.0 or later is installed.

Configure the Datadog Agent to scrape the Spice metrics endpoint:

1. Edit the openmetrics.d/conf.yaml file in the conf.d/ folder at the root of your Agent’s configuration directory:

init_config:

instances:
  - openmetrics_endpoint: http://localhost:9090/metrics # Spice metrics endpoint
    namespace: spiceai
    metrics:
      - .*
    collect_histogram_buckets: true # Required for percentile queries, see below
    histogram_buckets_as_distributions: true # Required for percentile queries, see below
    max_returned_metrics: 3000 # Default is 2000, which Spice can exceed
    tags:
      - service_instance_id:<unique-id> # e.g. hostname; required by the Spice dashboard filter

1. Restart the Agent to start collecting Spice metrics.
2. Refer to Prometheus and OpenMetrics metrics collection from a host for all available configuration options and supported parameters.
3. Open Datadog Metrics Explorer and type spiceai to confirm Spice telemetry information is successfully collected.

The namespace sets the metric prefix

Every scraped metric is prefixed with the configured namespace, so query_duration_ms is stored as spiceai.query_duration_ms. Keep this value consistent with the prefix used by the queries in the Spice dashboard JSON, and with runtime.telemetry.metric_prefix if the same deployment also pushes metrics over OTLP.

Histogram Percentiles (Distributions)

Spice exports latency and size metrics — query_duration_ms, flight_request_duration_ms, http_requests_duration_ms, dataset_acceleration_refresh_duration_ms, and others — as Prometheus histograms. Configuring the Agent to submit those buckets as Datadog distributions makes percentiles queryable over the selected time window:

p99:spiceai.flight_request_duration_ms{method:do_get} by {command}

Option	Default	Why Spice needs it
collect_histogram_buckets	true	Sends the _bucket series that percentiles are computed from.
histogram_buckets_as_distributions	false	Submits those buckets as a Datadog distribution rather than a set of counters. Required for p50:, p90:, p95:, and p99: queries.

Percentile aggregations must be enabled on the metric

Submitting a distribution is not sufficient on its own. Datadog computes p50/p90/p95/p99 for a distribution metric only once percentile aggregations are enabled for that metric on the Metrics Summary page — see Enabling advanced query functionality. Until then a p99: query returns no data and the widget renders empty rather than reporting an error.

Why not the _summary metrics

The metrics endpoint also exposes a <metric>_summary family carrying quantile tags, derived from each histogram at scrape time. The underlying histogram is cumulative, so those quantiles describe the entire lifetime of the process rather than the queried time window: they flatten the longer a pod stays up and will not surface a latency regression. Prefer distributions for any percentile that needs to track a time window.

Kubernetes (Operator / Autodiscovery)

With the Datadog Agent in the cluster, annotate the Spice container (spiceai in the Helm chart) so OpenMetrics scrapes include the service_instance_id tag used by the dashboard instance filter:

ad.datadoghq.com/spiceai.checks: |
  {
    "openmetrics": {
      "instances": [
        {
          "openmetrics_endpoint": "http://%%host%%:9090/metrics",
          "namespace": "spiceai",
          "metrics": [".*"],
          "collect_histogram_buckets": true,
          "histogram_buckets_as_distributions": true,
          "max_returned_metrics": 3000,
          "tags": ["service_instance_id:%%kube_pod_name%%"]
        }
      ]
    }
  }

collect_histogram_buckets and histogram_buckets_as_distributions serve the same purpose here as in the host configuration — see Histogram Percentiles (Distributions). max_returned_metrics raises the check's default limit of 2000, which Spice can exceed.

Instance Identity

The Spice dashboard identifies each running instance by a service_instance_id tag: the instance filter selects on it, and every per-instance panel groups by it. The Agent supplies no instance-level tag of its own — its unified service tags are env, service, and version — so set it explicitly, as in both examples above:

Deployment	Value
Kubernetes	%%kube_pod_name%%
Host, VM, or Docker	Hostname, or another stable per-process identifier

Panels collapse silently without this tag

Datadog does not error on a missing tag. Without service_instance_id, each panel renders a single N/A series summing every instance — a 12-replica deployment reports 12 times its real dataset count rather than showing nothing.

Panels from the Kubernetes integration (CPU, memory, and PVC utilization) group by pod_name instead, as those metrics never carry service_instance_id.

Import the Spice Datadog Dashboard

1. Create New Datadog Dashboard

2. Click Import dashboard JSON and drag and drop monitoring/datadog-dashboard.json file

3. Dashboard is now configured to display Spice.ai OSS key performance metrics

OpenTelemetry OTLP Export

As an alternative to scraping the Prometheus endpoint with the Datadog Agent, Spice can push metrics directly to Datadog's OTLP Metrics Intake Endpoint over HTTP. This is the recommended approach for agentless deployments (e.g. serverless, ephemeral containers) and for environments where the Datadog API key is managed through Spice's secret stores.

Minimal Configuration

Replace us3 with the Datadog site for the target account (us3, us5, eu, ap1, etc.) and store the Datadog API key in a secret:

runtime:
  telemetry:
    otel_exporter:
      endpoint: https://otlp.us3.datadoghq.com/v1/metrics
      headers:
        DD-API-KEY: ${secrets:DD_API_KEY}

Metrics begin appearing in the Datadog Metrics Explorer within a minute or two.

Namespace Spice Metrics with a Prefix

Use runtime.telemetry.metric_prefix to prepend a string to every exported metric name. This avoids collisions with metrics from other services in the same Datadog account:

runtime:
  telemetry:
    metric_prefix: 'spiceai.'

The runtime metric query_duration_ms is then exported as spiceai.query_duration_ms.

Combining metric_prefix with metric filtering

If you also set runtime.telemetry.otel_exporter.metrics to whitelist specific metrics, the entries must include the prefix. The filter runs after the prefix is applied, so e.g. query_duration_ms will not match when metric_prefix: 'spiceai.' is set — use spiceai.query_duration_ms instead.

Add Custom Tags via Resource Attributes

Attach custom key/value pairs to every metric using runtime.telemetry.properties. Spice sends these as OpenTelemetry resource attributes:

runtime:
  telemetry:
    properties:
      environment: prod
      region: us-west-2
      team: data-platform

For these resource attributes to surface as tags in Datadog, the Datadog OTLP intake also requires the dd-otel-metric-config header with resource_attributes_as_tags enabled (see Datadog OTLP Metrics Intake Endpoint):

runtime:
  telemetry:
    otel_exporter:
      endpoint: https://otlp.us3.datadoghq.com/v1/metrics
      headers:
        DD-API-KEY: ${secrets:DD_API_KEY}
        dd-otel-metric-config: '{"resource_attributes_as_tags": true}'

Tags can lag behind metrics

Datadog typically ingests OTLP metrics within seconds, but the associated tags (from resource attributes) can take noticeably longer to appear in the UI — sometimes several minutes after the first datapoints. The metrics and tags do eventually converge.

Manage tag cardinality in Datadog

Datadog bills on custom metric cardinality, driven by the number of unique tag-value combinations per metric. The custom tags added via runtime.telemetry.properties are typically low-cardinality (environment, region, team), but Spice metrics also carry a number of automatically populated dimensions — for example dataset, protocol, client, client_version, client_system, user_agent, runtime, runtime_version, runtime_system (see Available Metrics) — some of which can grow with the size of the deployment.

Datadog's Metrics without Limits™ decouples ingestion from indexing for exactly this case. With Metrics without Limits™, every tag Spice emits is still ingested, but each metric is configured with one of:

• an allowlist that keeps only the tags actually used in dashboards, monitors, and queries (e.g. keep dataset and environment, drop the rest), or
• a blocklist that drops specific auto-populated tags that are not useful for a given metric (e.g. exclude user_agent or client_version).

Only the indexed (queryable) tag combinations count toward custom metric billing. Configuration is done per metric in the Metrics Summary page or via the Metrics API, and the in-app UI surfaces an estimated indexed-metric volume before saving and can pre-populate an allowlist from tags actively queried in dashboards, monitors, and notebooks.

Full Example

A complete runtime.telemetry block combining metric prefixing, custom tags, and Datadog OTLP export:

runtime:
  telemetry:
    metric_prefix: 'spiceai.'
    properties:
      environment: prod
      region: us-west-2
      team: data-platform
    otel_exporter:
      endpoint: https://otlp.us3.datadoghq.com/v1/metrics
      headers:
        DD-API-KEY: ${secrets:DD_API_KEY}
        dd-otel-metric-config: '{"resource_attributes_as_tags": true}'

With this configuration, every Spice metric (e.g. spiceai.query_duration_ms, spiceai.query_executions) arrives in Datadog tagged with environment:prod, region:us-west-2, and team:data-platform.

For general OTLP exporter options (push interval, metric filtering, gRPC vs HTTP), see OpenTelemetry Metrics Exporter.