Observability & Monitoring
Spice can be monitored using the Spice Prometheus-compatible Metrics Endpoint.
Monitoring clients configuration:
Spice Metrics Endpoint Configuration​
The metrics endpoint uses port 9090 by default. The metrics endpoint configuration is logged at startup.
2024-11-28T19:48:10.942003Z INFO runtime::metrics_server: Spice Runtime Metrics listening on 127.0.0.1:9090
Pass the --metrics parameter to bind to a specific port. For example, to bind to port 9091:
spiced --metrics 0.0.0.0:9091
or when using Docker:
FROM spiceai/spiceai:latest
# Docker configuration ...
# Configure the metrics endpoint on port 9090
CMD ["--metrics", "0.0.0.0:9090"]
EXPOSE 9090
Configuration of the metrics endpoint can be verified using a HTTP GET request, for example:
curl http://localhost:9090/metrics
# HELP runtime_flight_server_started Indicates the runtime Flight server has started.
# TYPE runtime_flight_server_started counter
runtime_flight_server_started 1
# HELP runtime_http_server_started Indicates the runtime HTTP server has started.
# TYPE runtime_http_server_started counter
runtime_http_server_started 1
# HELP dataset_load_state Status of the dataset. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown.
# TYPE dataset_load_state gauge
dataset_load_state{dataset="taxi_trips"} 2
dataset_load_state{dataset="taxi_trips_accelerated"} 2
# HELP dataset_active_count Number of currently loaded datasets.
# TYPE dataset_active_count gauge
dataset_active_count{engine="None"} 1
dataset_active_count{engine="duckdb"} 1
...
Metrics​
| Metric | Description |
|---|---|
accelerated_ready_state_federated_fallback(count) | Number of times the federated table was queried due to the accelerated table loading the initial data. |
catalog_load_errors(count) | Number of errors loading the catalog provider. |
catalog_load_state(gauge) | Status of the catalog provider. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
dataset_acceleration_last_refresh_time_ms(gauge) | Unix timestamp in seconds when the last refresh completed. |
dataset_acceleration_refresh_duration_ms(histogram) | Duration in milliseconds to load a full or appended refresh data. |
dataset_acceleration_refresh_errors(count) | Number of errors refreshing the dataset. |
dataset_active_count(gauge) | Number of currently loaded datasets. |
dataset_load_state(gauge) | Status of the dataset. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
dataset_unavailable_time_ms(gauge) | Time dataset went offline in milliseconds. |
embeddings_active_count(gauge) | Number of currently loaded embeddings. |
embeddings_load_errors(count) | Number of errors loading the embedding. |
embeddings_load_state(gauge) | Status of the embedding. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
flight_request_duration_ms(histogram) | Measures the duration of Flight requests in milliseconds. |
flight_requests(count) | Total number of Flight requests. |
http_requests_duration_ms(histogram) | Measures the duration of HTTP requests in milliseconds. |
http_requests(count) | Number of HTTP requests. |
llm_load_state(gauge) | Status of the LLM model. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
model_active_count(gauge) | Number of currently loaded models. |
model_load_duration_ms(histogram) | Duration in milliseconds to load the model. |
model_load_errors(count) | Number of errors loading the model. |
model_load_state(gauge) | Status of the model. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
query_duration_ms(histogram) | The total amount of time spent planning and executing queries in milliseconds. |
query_execution_duration_ms(histogram) | The total amount of time spent only executing queries (0 for cached queries). |
query_executions(count) | Number of query executions. |
query_failures(count) | Number of query failures. |
query_processed_bytes(count) | Number of bytes processed by the runtime. |
query_returned_bytes(count) | Number of bytes returned to query clients. |
results_cache_max_size_bytes(gauge) | Maximum allowed size of the cache in bytes. |
results_cache_requests(count) | Number of requests to get a key from the cache. |
results_cache_hits(count) | Cache hit count. |
results_cache_items_count(gauge) | Number of items currently in the cache. |
results_cache_size_bytes(gauge) | Size of the cache in bytes. |
search_results_cache_max_size_bytes(gauge) | Maximum allowed size of the search cache in bytes. |
search_results_cache_requests(count) | Number of requests to get a key from the search cache. |
search_results_cache_hits(count) | Search cache hit count. |
search_results_cache_items_count(gauge) | Number of items currently in the search cache. |
search_results_cache_size_bytes(gauge) | Size of the search cache in bytes. |
runtime_flight_server_started(count) | Indicates the runtime Flight server has started. |
runtime_http_server_started(count) | Indicates the runtime HTTP server has started. |
secrets_store_load_duration_ms(histogram) | Duration in milliseconds to load the secret stores. |
tool_active_count(gauge) | Number of currently loaded LLM tools. |
tool_load_errors(count) | Number of errors loading the LLM tool. |
tool_load_state(gauge) | Status of the LLM tools. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
view_load_errors(count) | Number of errors loading the view. |
view_load_state(gauge) | Status of the views. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
Component Metrics
In addition to these core metrics, individual components can expose their own metrics. For example, the MySQL data connector exposes connection pool metrics. See Component Metrics for more information.
