Skip to main content

3 posts tagged with "opentelemetry"

OpenTelemetry topics and usage

View All Tags

Spice v2.4.0-rc.2 (Oct 15, 2026)

ยท 28 min read
Jack Eadie
Member of Technical Staff at Spice AI

Spice v2.4.0-rc.2 is now available! ๐Ÿ”ฅ

Spice v2.4.0-rc.2 is the second release candidate of v2.4.0 for early testing and feedback. It introduces a new OpenAI-compatible POST /v1/decisions API backed by an OpenAI decision model, TypeSafe Jev, or any other LLM in the Spicepod that can answer decisions. Five new SQL functions bring the same AI classification, scoring, and decisions into SQL.

This release also adds Iceberg reads without a catalog and improves acceleration snapshots. Cayenne gains runtime-wide adaptive tuning and applies acceleration configuration changes on restart.

Highlights in this release:

  • AI Decisions API and decisions in SQL: An OpenAI-compatible /v1/decisions API, plus five SQL functions for filtering, classifying, and scoring data with models.
  • Iceberg without a catalog: Read a table directly from its root directory or pin a metadata file to read a fixed version.
  • Shared runtime state: Startup validation and a common location for cluster state, rate control, cache warmup, and acceleration snapshots.
  • SQL results cache: Cache entries expire by TTL and remain available across data refreshes.
  • Cayenne acceleration: Improvements to primary-key loading, secondary indexes, and configuration updates.
  • OpenTelemetry log export: Send runtime logs to any OTLP endpoint as structured log records, and choose which failed queries are logged.

These notes cover changes since v2.4.0-rc.1. Review Breaking Changes before upgrading; upgrades from earlier versions also require the rc.1 migration guidance.

What's New in v2.4.0-rc.2โ€‹

AI Decisions API and decisions in SQLโ€‹

Spice now supports POST /v1/decisions, which is compatible with the OpenAI Decisions API. A decision request asks typed questions about an input. Each answer comes back with its probabilities and a confidence.

Any model in the Spicepod can answer a decision request:

ModelExample fromProbabilities
OpenAI decision modelopenai:gpt-6-lunaThe estimates of the model.
TypeSafe decision modeltypesafe:jevCalibrated by the model.
Any other LLMopenai:gpt-4o-mini, anthropic:claude-haiku-4-5The estimates of the chat model.

OpenAI decision models use the native OpenAI Decisions API. gpt-6-luna also supports chat and /v1/nsql. TypeSafe Jev returns calibrated probabilities and answers decisions only. Other LLMs answer through chat completions and return their own uncalibrated estimates.

models:
- from: openai:gpt-6-luna
name: luna
params:
openai_api_key: ${secrets:OPENAI_API_KEY}
- from: typesafe:jev
name: jev
params:
typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Five new SQL functions make the same decisions available in queries:

FunctionResult
ai_if(input, condition)BOOLEAN: true when the probability of "yes" exceeds 0.5.
ai_probability(input, condition)DOUBLE: a probability from 0 to 1.
ai_classify(input, labels)VARCHAR: one of the supplied labels.
ai_score(input, instructions, levels)DOUBLE: a probability-weighted score.
ai_decide(input, questions)STRUCT: answers, probabilities, and confidence for a set of questions.
SELECT
id,
ai_classify(body, ['billing', 'technical', 'account'], model => 'support_model' ) AS team,
ai_probability(body, 'The customer is requesting a refund', model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI functions for more examples.

Iceberg Without a Catalogโ€‹

Read an Iceberg table directly from object storage or the local filesystem, without configuring a catalog. Set from to the table root using file://, s3://, s3a://, gs://, or gcs://:

datasets:
- from: iceberg:s3://my-bucket/warehouse/db/taxis
name: taxis
params:
iceberg_s3_region: us-east-1

Spice resolves the metadata file on each query and refresh. It supports metadata/version-hint.text as well as a fixed version (a path relative to the table root or a full URL):

datasets:
- from: iceberg:s3://my-bucket/warehouse/db/taxis
name: taxis
params:
iceberg_s3_region: us-east-1
iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Fixed tables are read-only. The earlier metadata_path parameter remains supported with a deprecation warning; see Breaking Changes.

Shared Runtime Stateโ€‹

runtime.state provides a shared, object-store location for cluster state, rate control, SQL results-cache warmup, and acceleration snapshots. Each feature uses its own subdirectory within the provided location.

Acceleration snapshots now default to the snapshots/ subdirectory when runtime.state is configured.

runtime:
state:
location: s3://my-bucket/spice/

datasets:
- from: postgres:orders
name: orders
acceleration:
enabled: true
engine: cayenne
mode: file
snapshots: enabled

A dataset can override the snapshot location, its settings, and the bootstrap behavior on its own acceleration:

datasets:
- from: postgres:events
name: events
acceleration:
snapshots: enabled
snapshots_location: s3://archive-bucket/snapshots/
snapshots_params:
s3_region: eu-west-1
snapshots_bootstrap_on_failure_behavior: fallback

Other changes to shared state:

  • Snapshot queues: An SQS queue that covers the whole runtime.state prefix no longer causes an ERROR log for each scheduler heartbeat. The snapshot reader now deletes messages for the other state folders without a log.

SQL Results Cacheโ€‹

The SQL results cache now expires entries by TTL, independent of acceleration refreshes, CDC batches, and retention passes no longer invalidate cached results.

Review item_ttl and stale_while_revalidate_ttl when upgrading: a successful refresh no longer makes cached results reflect the refreshed data immediately. See Breaking Changes.

Cayenne Accelerationโ€‹

Adaptive tuning is configured once for the runtime. Enable it with runtime.adaptive_tuning and set targets under runtime.params:

runtime:
adaptive_tuning: enabled
params:
target_freshness: 30s

Supported targets are target_replication_lag, target_freshness, target_query_latency, target_convergence_window, and target_qph. Adaptive tuning defaults to disabled. See Breaking Changes and the runtime YAML documentation.

Other Cayenne improvements include:

  • Primary-key loading: Keyed tables build their primary-key filter during the initial load and bulk-load without per-row routing or a whole-table fold.
  • Position deletes: cayenne_deletion_mode: auto can select position deletes for small primary-key tables on local storage on Linux systems with at least eight physical cores.
  • Secondary indexes: Index runs are persisted, indexes remain complete across acceleration snapshots, and CDC maintains indexes for each operation type. Hash joins read each file once when narrowing a scan through an index.
  • Partition handling: Cayenne rejects stored data with an incompatible partition_by, protects keyed appends to partitioned tables during publication, and prepares every partition after a refresh.
  • Storage and query correctness: Cayenne reclaims snapshot directories superseded by a reload, corrects pruning for NaN values, and fixes deletion-index bounds and reload fencing.
  • Snapshot storage growth: A writer with snapshots_trigger: refresh_complete kept one more copy of the table for each refresh. Each snapshot held file deletions, so Cayenne skipped each cleanup pass. Cayenne now runs the cleanup when a snapshot releases its hold. In a 10-minute test with a refresh every 60 seconds, each dataset kept 1 snapshot directory instead of 10.
  • Configuration changes on restart: A change to write settings only, such as cayenne_sort_columns or cayenne_compression_strategy, applies to the existing acceleration. A change to primary_key or partition_by rebuilds the acceleration from the source only under specific conditions. See Cayenne Configuration Updates.

OpenTelemetry Log Exportโ€‹

Spice can now send its logs to an OpenTelemetry Collector, or any OTLP endpoint, as structured OTLP log records. Logs use the same otel_exporter that already sends metrics. A new runtime.query.log_errors setting chooses which failed queries are logged.

runtime:
telemetry:
otel_exporter:
endpoint: otel-collector:4317
signals: [metrics, logs] # new; default [metrics]
query:
log_errors: execution # new; none (default) | execution | all
  • Structured records: Each log line becomes an OTLP log record with its severity, the message as the body, typed fields as attributes, and the logging module as the instrumentation scope. Records share the resource attributes of exported metrics.
  • Trace correlation: A record logged during a query carries the query's trace ID, the same ID returned in spice-trace-id and stored in runtime.task_history.
  • Delivery: Records are batched on a dedicated export runtime, separate from query and HTTP traffic, and flushed at shutdown so the final error and Goodbye! reach the endpoint. For OTLP/HTTP endpoints, logs are sent to <base>/v1/logs.
  • Cluster executors: Executors export their logs with the scheduler's telemetry settings, including headers resolved from the scheduler's secrets.

runtime.query.log_errors controls which failed queries are logged:

ValueWhat is logged
none (default)Existing behavior: a query refused for memory at WARN, and every other failure at DEBUG
executionAlso queries that failed while running, at WARN
allAlso queries rejected before running (syntax and planning errors), at INFO

Each record reads Query failed: <error>, with error_code and datasets fields and the query's trace ID. The SQL text is never logged.

Unknown-column errors are now reported with the correct error_code instead of InternalError, in logs, runtime.task_history, and the query_failures metric.

See OpenTelemetry Logs Export, Exporting logs, and runtime.query.log_errors.

Fixes and Improvementsโ€‹

AI APIs and Modelsโ€‹

  • Responses API: /v1/responses executes Spice tools and honors tool_recursion_limit when streaming. It returns Spice tool calls as mcp_call items in output, allowing clients that manage their own conversation history to replay them.
  • Model controls: /v1/nsql accepts reasoning_effort, and rerank() uses evaluation models with a strategy selected for each query.
  • Anthropic: Streaming handles responses that begin with a thinking block, empty text blocks are dropped, and anthropic_workspace_id is available as a model parameter.
  • OpenAI: Models with an unknown service tier can load and serve requests.
  • Luna: gpt-6-luna and its snapshots support chat, /v1/nsql, and native decisions.

Accelerationโ€‹

  • Caching point lookups: Lookups on a refresh_mode: caching dataset failed with HTTP 400. This occurred when Cayenne accelerated the dataset, the primary key covered the request columns, and the table held two protected snapshots. The point-lookup plan did not combine the partitions of the Cayenne scan. Lookups now return the cached row or fetch it from the source.

Connectorsโ€‹

  • MySQL timestamps: Timestamp literals in queries that the runtime sends to MySQL now keep their fractional seconds.

Dependency Updatesโ€‹

DependencyUpdate
Tursov0.8.2

Breaking Changesโ€‹

Shared State Directory Layout and Startup Checksโ€‹

State written at the runtime.state root by rc.1 is no longer read from that location. Move existing state objects into the appropriate subdirectories before upgrading to retain them:

StateSubdirectory
Cluster schedulercluster/
Rate controlrate-control/
SQL results-cache warmupresults-cache/

Azure locations (abfs:// and abfss://) need one more step. In rc.1, the runtime ignored the path in an Azure URL and wrote state at the container root. For example, with abfs://container/spice/, rc.1 wrote cluster.json at the container root. This release writes spice/cluster/cluster.json. Move Azure state from the container root into the configured path and then into the feature subdirectory.

An explicit runtime.scheduler.state_location retains its earlier layout. Without migration, Spice starts without the state previously stored at the shared root.

The configured store must accept writes and conditional writes. Spice does not become ready until startup validation succeeds and refuses stores with a newer state format. Check storage permissions and compatibility before upgrading.

SQL Results Cache Freshnessโ€‹

A refresh no longer invalidates SQL results-cache entries. Queries can return data from before the latest refresh until item_ttl expires, plus stale_while_revalidate_ttl when configured. Reduce item_ttl if results need to follow source refreshes more closely.

Cayenne Tuning Configurationโ€‹

Replace the Cayenne tuning parameters with runtime settings that apply across datasets and catalogs:

Previous parameterReplacement
cayenne_tuning: adaptiveruntime.adaptive_tuning: enabled
cayenne_tuning: autoruntime.adaptive_tuning: disabled
cayenne_goal_replication_lagruntime.params.target_replication_lag
cayenne_goal_freshnessruntime.params.target_freshness
cayenne_goal_query_latencyruntime.params.target_query_latency
cayenne_goal_convergence_windowruntime.params.target_convergence_window
cayenne_goal_qphruntime.params.target_qph

runtime.adaptive_tuning accepts only enabled or disabled. Other values, including true, fail the Spicepod load.

Pinned Iceberg Tablesโ€‹

Replace the deprecated metadata_path parameter with iceberg_metadata_path. The two settings resolve paths differently. metadata_path uses its value as given. iceberg_metadata_path resolves a value without a URL scheme relative to the table root in from. Convert an absolute local path to a full file:// URL or to a path relative to the table root.

Datasets pinned to a .json metadata file are read-only and fail to load with access: read_write. Remove that setting from pinned datasets before upgrading.

Cayenne Configuration Updatesโ€‹

A restart now applies changed settings to an existing Cayenne acceleration. The result depends on the setting and the dataset:

  • Write settings only: A change to only cayenne_sort_columns, cayenne_cluster_by, cayenne_compression_strategy, cayenne_delta_encoding, or cayenne_shard_key_columns applies to the existing acceleration. Stored rows keep their layout until a refresh or compaction rewrites them.
  • Identity settings: A change to primary_key, partition_by, cayenne_pk_conflict_detection from none to auto, or the data location rebuilds the acceleration from the source. The runtime rebuilds only a dataset with refresh_mode: full, mode: file or file_update, and write_mode: write_through. The source must not be sink:, and must be readable with the schema of the dataset.
  • All other datasets: The acceleration keeps all of its stored settings. This includes write settings that changed in the same restart. The runtime logs a warning that names the change and the fix.

While the runtime cannot read the source, a dataset that qualifies for a rebuild keeps its existing acceleration. A partition_by change or a conflict-detection change cannot use the stored rows, so it fails the dataset instead. Revert the setting, or restart when the source is readable.

A restart alone does not migrate a dataset that does not meet these conditions. Read the startup warning for the dataset before you delete stored data. Allow time for a rebuild when the conditions apply.

Upgradingโ€‹

Once release artifacts are available, select v2.4.0-rc.2 explicitly with the CLI, Docker tag, or Helm chart version.

CLIโ€‹

spice upgrade v2.4.0-rc.2

Dockerโ€‹

docker pull spiceai/spiceai:2.4.0-rc.2

See Docker Hub for available tags.

Helmโ€‹

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.2

Homebrew and AWS Marketplaceโ€‹

Homebrew and AWS Marketplace follow their published versions. To select this release candidate, use an explicit version through the CLI or Docker.

Spice is also available in the AWS Marketplace.

Contributorsโ€‹

Changelogโ€‹

See the full changelog from v2.4.0-rc.1 to v2.4.0-rc.2.

  • feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
  • fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588
  • fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart in #14851
  • fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice in #14792
  • fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-duckdb[file]-2gib.yaml by @github-actions in #14850
  • Update openapi.json by @github-actions in #14858
  • test: strengthen assertions and enforce sign-off coverage by @lukekim in #14740
  • Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart in #14888
  • fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice in #14884
  • fix(anthropic): drop empty text blocks from assistant messages by @Jeadie in #14834
  • fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim in #14882
  • test(sqlite): guard the Date32 literal cast through the SQLite acceleration's own scan (fixes #14491) by @grokspice in #14703
  • fix(cayenne): read each file once when a hash join narrows an indexed table's scan by @lukekim in #14934
  • fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc in #14946
  • fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc in #14939
  • fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim in #14944
  • fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim in #14955
  • fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc in #14942
  • fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc in #14960
  • feat(spicepod): publish pull request reviews by @lukekim in #14964
  • fix(ci): bound the REPL expect-script waits on silence, not total response time (fixes #13711) by @grokspice in #14963
  • fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc in #14957
  • fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc in #14956
  • fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim in #14798
  • fix(ci): provision DuckLake for the ducklake[catalog] schema test, which has never passed by @grokspice in #14952
  • fix(llms): stream Anthropic responses that open with a thinking block (fixes #14909) by @krinart in #14920
  • fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov in #14968
  • feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim in #14852
  • fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart in #14972
  • fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart in #14927
  • feat(cayenne): always persist secondary index runs by @sgrebnov in #14979
  • fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers in #14919
  • feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim in #14838
  • fix(rrf): keep named arguments on nested search calls by @phillipleblanc in #14975
  • fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart in #14932
  • fix(vortex): keep temporal casts Vortex can't evaluate out of the scan by @sgrebnov in #14985
  • fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers in #14928
  • test(huggingface): unblock the merge queue: read imdb's parquet conversion through @~parquet and fail fast on a vanished Hub pin by @grokspice in #14998
  • test(cayenne): link the integration tests into one test binary by @bjchambers in #14976
  • test(vortex): compile the integration tests into a binary the sign-off gate runs by @bjchambers in #14987
  • fix(http): reject a request_path that URL parsing would rewrite by @bjchambers in #14965
  • fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc in #14953
  • Add anthropic_workspace_id param for Anthropic models by @Jeadie in #14829
  • fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim in #14783
  • test: run federated source tests on every supported LTS release line by @lukekim in #14936
  • fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius in #14791
  • fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim in #14806
  • fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim in #15009
  • Improve types methods in runtime-rate-control by @Jeadie in #14573
  • fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart in #14930
  • Update openapi.json by @github-actions in #14908
  • test(cayenne): run the heap-measuring tests in one isolated test binary by @bjchambers in #15010
  • test(vortex): run vortex-datafusion's integration tests in the sign-off gate by @bjchambers in #15001
  • fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc in #15019
  • Update openapi.json by @github-actions in #15018
  • fix(ci): keep the OpenAPI regeneration PR current instead of skipping while one is open by @phillipleblanc in #15023
  • fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim in #14993
  • fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers in #14891
  • chore: Update Turso crate to 0.8.2 by @spiceemma in #15022
  • feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee in #14844
  • ci: apply whole-binary nextest overrides to the module test sets they select by @bjchambers in #15011
  • fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc in #15028
  • test(data_components): compile the integration tests into one test binary by @bjchambers in #14991
  • fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc in #15053
  • fix(runtime): give each runtime.state feature its own folder by @phillipleblanc in #15036
  • test(data_components): make the HTTP provider snapshots independent of the machine by @bjchambers in #15000
  • fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc in #15054
  • test(runtime): drive the cdc_cayenne tests through Cayenne's change sink by @bjchambers in #15003
  • ci(object_store_occ): run the WAL E2E suite on self-hosted runners against disk, MinIO, RustFS and spiceio by @lukekim in #15031
  • test(llms): compile the credential-free tests into the integration binary by @bjchambers in #14990
  • fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov in #15068
  • ci(setup-minio): name the missing secret instead of failing in mc's argument parser by @grokspice in #15073
  • ci: run Substrait Mode B on the Linux runners that can reach the trunk spiced builds by @grokspice in #15074
  • feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie in #14782
  • build(deps): bump the github-actions-dependencies group across 5 directories with 7 updates by @dependabot in #15027
  • fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov in #15069
  • feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov in #15081
  • Fixes for MCP server by @Jeadie in #15041
  • fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice in #15056
  • fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice in #15037
  • perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers in #15072
  • fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart in #14931
  • fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart in #14937
  • fix(decisions): Fixes for /v1/decisions by @Jeadie in #15040
  • perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers in #15076
  • fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers in #15077
  • test(runtime,spiced): compile integration tests into declared binaries, run the ones no CI job ran, and remove the uncalled supersede_initial_load by @bjchambers in #15005
  • fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim in #15083
  • ci(e2e): don't run rust-cache on the macOS build, whose save deletes cargo from sibling runners by @grokspice in #15100
  • feat(testoperator): record per-query search metrics by @Jeadie in #15090
  • fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc in #15096
  • feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie in #14340
  • ci: run merge-queue Mode A TPC-H and LLM integration tests on spiceai-macos-large by @lukekim in #15116
  • docs(cayenne): the per-file fold merges no source files, but may roll by @bjchambers in #15101
  • build(deps): bump async-openai to the fork's merge of upstream main by @krinart in #14980
  • feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov in #15095
  • fix(runtime): fail recovery of an unsaved running query by @phillipleblanc in #15092
  • perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc in #15099
  • test(cayenne): guard per-file statistics sharing for listings outside a scan view by @bjchambers in #15103
  • feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers in #15033
  • fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers in #14140
  • Change codeql runner from spiceai-macos to spiceai-macos-large by @lukekim in #15120
  • fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim in #15106
  • fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim in #15108
  • fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers in #15111
  • fix: Update Search integration test snapshots by @github-actions in #15050
  • test(iceberg): read the Glue fixture unpinned and check the picked metadata file by @sgrebnov in #15115
  • test(cayenne): differential and plan coverage for secondary indexes on partitioned accelerations by @bjchambers in #15113
  • test: answer results-cache misses from the acceleration while the source is down by @bjchambers in #15112
  • ci(lint): fail on test files no target compiles, and document the integration test layout by @bjchambers in #15109
  • feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov in #15117
  • fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov in #15110
  • fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart in #14977
  • feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim in #15042
  • fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim in #15105
  • fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc in #15128
  • fix(test): wait for pkd_ctl to load before reading its bootstrap row in the write-back pk_conflict test by @grokspice in #15102
  • fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart in #15126
  • test(mysql): skip the SQL results cache in the binlog replication end-to-end tests by @grokspice in #15141
  • feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc in #15138
  • feat(acceleration): apply changed acceleration settings on restart by @lukekim in #15094
  • fix(models): enable Luna chat and NSQL support by @krinart in #15166
  • fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc in #15177
  • fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov in #15179
  • feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim in #15176

Spice v2.0-rc.3 (Apr 21, 2026)

ยท 13 min read
Evgenii Khramkov
Member of Technical Staff at Spice AI

Announcing the release of Spice v2.0-rc.3! โšก

v2.0.0-rc.3 is the third release candidate for advanced testing of v2.0, building on v2.0.0-rc.2.

Highlights in this release candidate include:

  • HTTP Connector Enhancements with OAuth2 refresh-token authentication, query-parameter pagination, and map-to-array conversion for broader API compatibility
  • Databricks and Unity Catalog Reliability Improvements with resilience controls, improved UC-awareness, permission checks, and structured error reporting
  • Snowflake and ADBC Registration Performance Improvements with better observability during dataset registration
  • OpenTelemetry Exporter Improvements with exporter fixes and support for authenticated metrics export
  • Kafka, GitHub, and HTTP Connector Fixes including Kafka reliability improvements, GitHub GraphQL resilience updates, and HTTP JSON union/reload fixes

What's New in v2.0.0-rc.3โ€‹

HTTP Connector Enhancementsโ€‹

The HTTP connector now supports more authentication and API response patterns, making it easier to integrate with modern REST APIs.

Key improvements:

  • OAuth2 Refresh-Token Authentication: Added support for OAuth2 refresh-token flows for APIs that issue short-lived access tokens.
  • Query-Parameter Pagination: Added pagination support using query parameters for APIs that expose page or cursor controls in the URL.
  • Map-to-Array Conversion: Added response transformation support for APIs that return map-shaped payloads that need to be normalized into arrays.
  • Improved JSON Union Handling: Better handling for heterogeneous JSON payloads during ingestion.
  • More Reliable Reloads: Fixed runtime behavior for HTTP-backed datasets during spicepod reloads.

Example configuration of an HTTP connector using the OAuth2 refresh token flow:

datasets:
- from: https://api.example.com
name: secure_data
params:
file_format: json
allowed_request_paths: '/v1/**'
auth_token_url: https://auth.example.com/oauth/token
http_auth_refresh_token: ${secrets:my_refresh_token}
http_auth_client_id: ${secrets:my_client_id}
http_auth_client_secret: ${secrets:my_client_secret}
auth_scopes: 'read:data offline_access'

Databricks and Unity Catalog Reliability Improvementsโ€‹

Databricks and Unity Catalog integrations are now more resilient and provide clearer behavior in permission-constrained environments.

Key improvements:

  • Resilience Controls: Added controls to improve reliability when interacting with Databricks services.
  • Unity Catalog Awareness: Improved handling for Unity Catalog-specific behaviors and mixed deployment configurations.
  • Permission Prechecks: Databricks UC permission checks now distinguish explicit denials from ambiguous cases.
  • Structured Error Reporting: Advisory permission failures now surface with more actionable structured errors.
  • Classic SQL Warehouse Compatibility: Improved handling for foreign table scenarios with Classic SQL Warehouse combinations.
  • Task History Instrumentation: Added instrumentation to improve observability for Databricks-related operations.

Snowflake and ADBC Improvementsโ€‹

Snowflake and ADBC-backed dataset registration is now faster and easier to observe.

Key improvements:

  • Faster Dataset Registration: Improved registration performance for Snowflake and ADBC datasets.
  • Better Observability: Added better instrumentation and visibility into registration workflows.
  • ADBC Alignment: Updated ADBC dependencies and integration points for improved compatibility.
  • Search Schema Fix: Fixed a full-text search schema mismatch issue with the ADBC connector.

OpenTelemetry and Observability Improvementsโ€‹

Spice improves telemetry export reliability and authenticated metrics delivery for the OpenTelemetry integration.

Key improvements:

  • OTEL Exporter Fixes: Fixed issues in the OpenTelemetry exporter.
  • Authenticated Metrics Export: Added support for authorization headers in the OTEL metrics exporter.
  • Reduced Startup Noise: Suppressed unnecessary AWS SDK noise and improved OTEL-related initialization behavior.
  • Connector Initialization Reliability: Fixed issues that could block connector initialization in telemetry-related code paths.

Dependency and Toolchain Updatesโ€‹

Dependency / ComponentVersion / Update
Rust toolchainv1.94.1 (from v1.93.1)
DataFusionv52.5.0-rc1
mistral.rs / candlemistral.rs v0.8.x, candle v0.10.1
ADBC Corev0.23

Other Improvementsโ€‹

  • Improved Query Pushdown: Expanded sort and limit pushdown, including improved pushdown behavior for Oracle and MSSQL connectors.
  • Partitioned Query Planning Improvements: Improved PartitionedTableScanRewrite handling for ORDER BY, partition expressions, and fully qualified table references, while preventing incorrect bucketing partition pushdown to executors.
  • MongoDB SRV Support: Upgraded datafusion-table-providers with MongoDB SRV support.
  • Tantivy Logging: Search logging now defaults to warn unless very verbose logging is enabled.
  • Kafka Connector Fixes: Improved reliability for the Kafka connector behavior.
  • GitHub Connector Resilience: Improved commit fetching for dynamic and slash refs, and reduced GraphQL page sizes on gateway errors for the GitHub connector.
  • GitHub API Efficiency: Lowered default comment fetch sizes to reduce pressure on GitHub GraphQL APIs.
  • Embedding Validation: Added validation for embedding row_id columns during dataset initialization.
  • View Cache Invalidation: Cached plans are now cleared when views are updated.
  • Refresh SQL Dedup Fix: Fixed append refresh deduplication when refresh_sql selects a subset of columns.

Contributorsโ€‹

Breaking Changesโ€‹

No breaking changes.

Cookbook Updatesโ€‹

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgradingโ€‹

To upgrade to v2.0.0-rc.3, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.3

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.3 image:

docker pull spiceai/spiceai:2.0.0-rc.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.3

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changedโ€‹

Changelogโ€‹

  • fix: Full Text Search schema mismatch with ADBC connector by @lukekim in #10235
  • docs: Update v2.0.0-rc.2 release notes with latest changes by @lukekim in #10238
  • Fix append refresh dedup failure when refresh_sql selects column subset by @sgrebnov in #10225
  • Revert "Properly mark dataset as Ready on Scheduler (#10215)" by @sgrebnov in #10242
  • Fix failing merge conflicts for benchmarks by @krinart in #10247
  • fix(github): fetch commits for dynamic and slash refs by @lukekim in #10233
  • Upgrade DataFusion to v52.5.0-rc1 by @lukekim in #10249
  • Merge develop to trunk (2026-04-09) by @claudespice in #10248
  • fix: Validate embedding row_id columns during dataset init (fixes #8226) by @claudespice in #10208
  • fix: Update tpch benchmark snapshots for federated/glue[csv].yaml by @app/github-actions in #10244
  • feat(databricks): add resilience controls, UC awareness, and task history instrumentation by @lukekim in #10246
  • fix: Make PartitionManager resilient to bare vs fully qualified table references by @sgrebnov in #10257
  • fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-cayenne[file].yaml by @app/github-actions in #10256
  • Merge develop to trunk (2026-04-10) by @claudespice in #10251
  • Improve Snowflake/ADBC dataset registration performance and observability by @lukekim in #10266
  • Fixes for kafka connector by @krinart in #10263
  • fix(runtime): gate otel code tags, suppress aws sdk noise, and unblock connector init by @lukekim in #10260
  • fix(runtime): avoid regionless AWS SDK loads by @lukekim in #10271
  • Add versioned release install workflow coverage by @lukekim in #10276
  • fix(runtime): handle HTTP JSON unions and spicepod reloads by @lukekim in #10277
  • Databricks UC permission prechecks: explicit denial as permanent error, ambiguous cases advisory by @lukekim in #10274
  • Revert component status changes re-introduced by develop merge (#10248) by @sgrebnov in #10293
  • Fix broken CI workflows by @ewgenius in #10294
  • Group dependabot updates by ecosystem by @lukekim in #10296
  • fix(tests): Replace flaky S3 Vectors snapshot tests with structural validation by @lukekim in #10301
  • Update test_github_workflows snapshot by @lukekim in #10304
  • fix(ci): fix Bedrock runner mismatch and snapshot auto-merge failure by @ewgenius in #10306
  • feat(http): Add map-to-array conversion and query-parameter pagination by @lukekim in #10295
  • New crate: datafusion-ddl by @Jeadie in #10205
  • Make Databricks UC permission checks advisory with structured error reporting by @lukekim in #10283
  • build(deps): bump the github-actions-dependencies group with 4 updates by @app/dependabot in #10298
  • fix: Clear cached plans on view updates by @peasee in #10312
  • build(deps): bump the aws-sdk group with 7 updates by @app/dependabot in #10299
  • Code out of runtime. by @Jeadie in #10178
  • fix: Respect function registry denies for accelerated table filter pushdown by @peasee in #10311
  • fix: Don't block heartbeat when all slots acquired by @peasee in #10322
  • fix: strip only outer parens in get_table_partition_expr_from_ctx by @Jeadie in #10323
  • Upgrade datafusion-table-providers with MongoDB SRV support by @lukekim in #10317
  • fix: Avoid pushing down bucketing partition expressions into executors by @peasee in #10324
  • Upgrade datafusion-table-providers to d1b911a5 and bump adbc to 0.23 by @lukekim in #10329
  • fix: Update Search integration test snapshots by @app/github-actions in #10308
  • Handle foreign table + Classic sql warehouse combination gracefully by @krinart in #10318
  • New crate datafusion-flightsql by @Jeadie in #10201
  • Set tantivy=warn unless very verbose logging by @Jeadie in #10338
  • Remove image registry and image name options from spidapter by @ewgenius in #10241
  • build(deps): bump sysinfo from 0.37.2 to 0.38.4 by @app/dependabot in #10291
  • build(deps): bump futures from 0.3.31 to 0.3.32 by @app/dependabot in #10289
  • New crate 'datafusion-dml' by @Jeadie in #10334
  • Jeadie/26 04 16/spice sql by @Jeadie in #10343
  • Add Teraswitch/Pittsburgh apt mirrors + retry config for CI runners by @lukekim in #10349
  • Implement sort pushdown and fix pushdown gaps across providers by @lukekim in #10337
  • Merge develop to trunk (2026-04-16) by @claudespice in #10345
  • Update candle and mistral.rs lock-step pins by @lukekim in #10278
  • docs: fix status badges in README by @lukekim in #10350
  • Migrate secrets to vars by @krinart in #10354
  • Add limit pushdown and improve sort pushdown for Oracle and MSSQL by @sgrebnov in #10351
  • Fix ubuntu mirror configuration by @ewgenius in #10359
  • fix: Increase throughput test default ready_wait from 30s to 300s (fixes #8207) by @claudespice in #10344
  • Add auth headers support to OTEL metrics exporter by @lukekim in #10347
  • fix(github): shrink GraphQL page size on gateway errors; lower comment defaults by @lukekim in #10355
  • Relax apt mirror substitution failure to warning in CI action by @ewgenius in #10361
  • feat(http): Add OAuth2 refresh-token auth to HTTP connector by @lukekim in #10348
  • Upgrade Rust toolchain to 1.94.1 by @lukekim in #10353
  • Handle order by and sort in PartitionedTableScanRewrite by @Jeadie in #9656
  • Fix OTEL Exporter by @krinart in #10363
  • Pin spiceai candle / TEI forks to merged revs; drop local [patch] overrides by @lukekim in #10362

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.2...v2.0.0-rc.3

Spice v1.10.0 (Dec 9, 2025)

ยท 18 min read
William Croxson
Member of Technical Staff at Spice AI

Announcing the release of Spice v1.10.0! โšก

Spice v1.10.0 introduces a new Caching Acceleration Mode with stale-while-revalidate (SWR) semantics for disk-persisted, low-latency queries with background refresh. This release also adds the TinyLFU eviction policy for the SQL results cache, a preview of the DynamoDB Streams connector for real-time CDC, S3 location predicate pruning for faster partitioned queries, improved distributed query execution, and multiple security hardening improvements.

What's New in v1.10.0โ€‹

Caching Acceleration Modeโ€‹

Low-Latency Queries with Background Refresh: This release introduces a new caching acceleration mode that implements the stale-while-revalidate (SWR) pattern. Queries return cached results immediately while data refreshes asynchronously in the background, eliminating query latency spikes during refresh cycles. Cached data persists to disk using DuckDB, SQLite, or Cayenne file modes.

Key Features:

  • Stale-While-Revalidate (SWR): Returns cached data immediately while refreshing in the background, reducing query latency
  • Disk Persistence: Cached results persist across restarts using DuckDB, SQLite, or Cayenne file modes
  • Configurable Refresh: Control refresh intervals with refresh_check_interval to balance freshness and source load

Recommendation: Use retention configuration with caching acceleration to ensure stale data is cleaned up over time.

Example spicepod.yaml configuration:

datasets:
- from: http://localhost:7400
name: cached_data
time_column: fetched_at
acceleration:
enabled: true
engine: duckdb
mode: file # Persist cache to disk
refresh_mode: caching
refresh_check_interval: 10m
retention_check_enabled: true
retention_period: 24h
retention_check_interval: 1h

For more details, refer to the Data Acceleration Documentation.

TinyLFU Cache Eviction Policyโ€‹

Higher Cache Hit Rates for SQL Results Cache: A new TinyLFU cache eviction policy is now available for the SQL results cache. TinyLFU is a probabilistic cache admission policy that maintains higher hit rates than LRU while keeping memory usage predictable, making it ideal for workloads with varying query frequency patterns.

Example spicepod.yaml configuration:

runtime:
caching:
sql_results:
enabled: true
eviction_policy: tiny_lfu # default: lru

For more details, refer to the Caching Documentation and the Moka TinyLFU Documentation for details of the algorithm.

DynamoDB Streams Data Connector (Preview)โ€‹

Real-Time Change Data Capture for DynamoDB: The DynamoDB connector now integrates with DynamoDB Streams for real-time change data capture (CDC). This enables continuous synchronization of DynamoDB table changes into Spice for real-time query, search, and LLM-inference.

Key Features:

  • Real-Time CDC: Automatically captures inserts, updates, and deletes from DynamoDB tables as they occur
  • Table Bootstrapping: Performs an initial full table scan before streaming changes, ensuring complete data consistency
  • Acceleration Integration: Works with refresh_mode: changes to incrementally update accelerated datasets

Note: DynamoDB Streams must be enabled on your DynamoDB table. This feature is in preview.

Example spicepod.yaml configuration:

datasets:
- from: dynamodb:my_table
name: orders_stream
acceleration:
enabled: true
refresh_mode: changes # Enable Streams capture

For more details, refer to the DynamoDB Connector Documentation.

OpenTelemetry Metrics Exporterโ€‹

Spice can now push metrics to an OpenTelemetry collector, enabling integration with platforms such as Jaeger, New Relic, Honeycomb, and other OpenTelemetry-compatible backends.

Key Features:

  • Protocol Support: Supports the gRPC (default port 4317) protocol
  • Configurable Push Interval: Control how frequently metrics are pushed to the collector

Example spicepod.yaml configuration for gRPC:

runtime:
telemetry:
enabled: true
otel_exporter:
endpoint: 'localhost:4317'
push_interval: '30s'

For more details, refer to the Observability & Monitoring Documentation.

S3 Connector Improvementsโ€‹

S3 Location Predicate Pruning: The S3 data connector now supports location-based predicate pruning, dramatically reducing data scanned by pushing down location filter predicates to S3 listing operations. For partitioned datasets (e.g., year=2025/month=12/), Spice now skips listing irrelevant partitions entirely, significantly reducing query latency and S3 API costs.

AWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables, enabling direct integration with AWS's managed table format for S3. Use standard SQL INSERT INTO to write data.

For more details, refer to the S3 Data Connector Documentation and Glue Data Connector Documentation.

Faster Distributed Query Executionโ€‹

Distributed query planning and execution have been significantly improved:

  • Fixed executor registration in cluster mode for more reliable distributed deployments
  • Improved hostname resolution for Flight server binding, enabling better executor discovery
  • Distributed accelerator registration: Data accelerators now properly register in distributed mode
  • Optimized query planning: DistributeFileScanOptimizer improvements for faster planning with large datasets

For more details, refer to the Distributed Query Documentation.

Search Improvementsโ€‹

Search capabilities have been improved with several performance and reliability enhancements:

  • Fixed FTS query blocking: Full-text search queries no longer block unnecessarily, improving query responsiveness
  • Optimized vector index operations: Eliminated unnecessary list_vectors calls for better performance
  • Improved limit pushdown: IndexerExec now properly handles limit pushdown for more efficient searches

For more details, refer to the Search Documentation.

Security Hardeningโ€‹

Multiple security improvements have been implemented:

  • SQL Identifier Quoting: Hardened SQL identifier quoting across all database connectors (PostgreSQL, MySQL, DuckDB, etc.) to prevent SQL injection attacks through table or column names
  • Token Redaction: Sensitive authentication tokens are now fully redacted in debug and error output, preventing accidental credential exposure in logs
  • Path Traversal Prevention: Fixed tar extraction operations to prevent directory traversal vulnerabilities when processing archived files
  • Input Sanitization: Added strict validation for top_n_sample order_by clause parsing to prevent injection attacks
  • Glue Credential Handling: Prevented automatic loading of AWS credentials from environment in Glue connector, ensuring explicit credential configuration

Developer Experience Improvementsโ€‹

  • Health probe metrics: Added health probe latency metrics for better observability
  • CLI improvements: Fixed .clear history command in the REPL to fully clear persisted history

Contributorsโ€‹

Breaking Changesโ€‹

No breaking changes.

Cookbook Updatesโ€‹

No major cookbook updates.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgradingโ€‹

To upgrade to v1.10.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.0 image:

docker pull spiceai/spiceai:1.10.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

๐ŸŽ‰ Spice is now available in the AWS Marketplace!

What's Changedโ€‹

Changelogโ€‹