13 posts tagged with "iceberg"

Spice v2.4.0-rc.3 is now available! 🛠️

Spice v2.4.0-rc.3 is the third release candidate for v2.4.0, with mostly fixes since rc.2. By default, full and append refreshes now write on their own threads, not on the threads that serve queries. In a Cayenne test, P99 query latency at each refresh commit fell from 130–690 ms to the steady P99 of 44–50 ms. On Unix, the runtime also raises its soft open file limit at startup, up to the hard limit. Before, a few hundred client connections in a container could make Cayenne files fail to open.

Twelve other changes apply to Iceberg append refresh and table reads, Spice Cayenne, Bedrock decisions, OTLP logs, distributed queries, and TLS verification. Some apply only in specific setups. Two fix regressions from earlier v2.4.0 release candidates: Failed to open Vortex file errors after a Cayenne load, and failed MySQL CDC snapshots into Cayenne.

Testers of rc.2 should retest Iceberg append refresh, Cayenne refresh_mode: caching, and ai_* functions on Bedrock models. If you are new to v2.4.0, start with the rc.1 and rc.2 notes for its main features and breaking changes.

Highlights in this release candidate include:

• Lower query latency during refreshes: By default, full and append refreshes write to the accelerator on dedicated refresh threads. In a Cayenne test, P99 latency at each refresh commit fell from 130–690 ms to the steady P99 of 44–50 ms.
• Higher open file limit on Unix: The runtime raises its soft open file limit at startup, up to the hard limit. In a container with a soft limit of 1024 and a hard limit of 524288, the runtime raised its soft limit to 524288.

What's New in v2.4.0-rc.3​

Lower Query Latency During Refreshes​

Full and append refreshes now write to the accelerator on the dedicated refresh threads. Before this release, the write ran on the threads that also serve HTTP requests. For Cayenne, the write sorts, encodes, checks for duplicate keys, and publishes the data. A large write held those threads, and query latency rose until the write ended.

The change applies to these refresh writes, for all accelerators:

• refresh_mode: full, and the initial load
• refresh_mode: append
• refresh_mode: changes, only when it rebuilds the table from a snapshot

Steady CDC apply, the caching and snapshot refresh modes, and retention deletes do not change. With runtime.params.dedicated_thread_pool: disabled, there are no refresh threads, and refresh writes run as before.

In a test, Cayenne ran a full refresh every 5 minutes on a table of 2.2 million rows and about 8 GB. A client sent 500 point-lookup joins per second over HTTP. Before the change, P99 latency in the 15-second window of each refresh commit was 130–690 ms. After the change, it matched the steady P99 of 44–50 ms. The test did not cover append refreshes or other accelerators. See #15262.

Higher Open File Limit on Unix​

On Unix systems such as Linux and macOS, the runtime now raises its soft open file limit (RLIMIT_NOFILE) to the hard limit at startup. Client connections, acceleration files, and pipes all count against this one limit. Containers often start the runtime with a soft limit of 1024 and a hard limit of 524288. Before this release, a few hundred client connections could make Cayenne files fail to open. Queries then failed with Unable to open Cayenne acceleration file, or the client saw Connection reset by peer.

• Measured: On macOS with a soft limit of 256, v2.3.1 failed queries with 176 to 220 idle client connections open. A build with the fix raised the limit to 245760, the kern.maxfilesperproc value of that Mac. It answered queries with 600 connections open.
• Containers: In a Docker container with --ulimit nofile=1024:524288, the runtime raised its soft limit from 1024 to 524288.
• macOS: The runtime raises the limit to at most kern.maxfilesperproc.
• Logs: The runtime logs the result at DEBUG, for example Raised the open file limit from 1024 to 524288. The runtime also logs a failed raise only at DEBUG. Start the runtime with -v or set SPICED_LOG to see it.
• Windows: Nothing changes.

If the hard limit is also low, the runtime cannot raise the soft limit above it. For this case, the Cayenne error now names the effective limit and says that client connections count against it. It also keeps the Too many open files (os error 24) text from the operating system:

Failed to open Cayenne acceleration file '…/01a15b2f5c697c01b2a8acdede2d637d_p000_00011.vortex' because the process reached its open file limit of 256, which client connections and all other open files also count against. Raise the limit where spiced is started (`ulimit -n`, or the container's `nofile` limit), or write fewer, larger acceleration files with `cayenne_target_file_size_mb`. See: https://spiceai.org/docs/components/data-accelerators/cayenne#params. Cause: … Too many open files (os error 24)

See #15339.

Iceberg Append Refresh Follows Table Snapshots​

An Iceberg dataset with refresh_mode: append and a time_column now loads the rows that each new commit adds, whatever their time values. This applies to Cayenne datasets and to datasets without a primary key. It covers Iceberg tables read through the Iceberg or Glue data connectors. Glue tables in Parquet, CSV, or ORC format still read by time.

Before this release, an append refresh read only the rows with a time_column value later than the stored maximum minus refresh_append_overlap. A writer that commits out of time order can put rows outside that window. A sink with several shards, one of them behind, is an example. Those rows did not load until a full refresh.

The refresh now saves the Iceberg snapshot that it loaded last. The next refresh reads the data files that later snapshots added. An unchanged table scans no data. This change adds no setting:

datasets:
  - from: iceberg:https://iceberg-catalog.example.com/v1/namespaces/sales/tables/orders
    name: orders
    time_column: updated_at
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      primary_key: order_id
      refresh_mode: append

• Newest version by time: With a primary key, Cayenne keeps the newest version of each key by time_column. A late row with an older time does not replace a newer stored row.
• Skipped commits: The refresh skips compactions and commits that add no data file. An append refresh never removes a row.
• Fallback to reads by time: If the runtime cannot follow the snapshots, it reads by time for one refresh. This happens after an overwrite on a dataset without a primary key, or after an expired snapshot in the interval. A rollback, a branch switch, or a recreated table has the same effect.
• After a fallback: A refresh that reads by time does not load rows older than refresh_append_overlap. A full refresh loads them. The runtime logs a warning when the reason changes. The new dataset_acceleration_refresh_fallbacks metric counts each fallback by reason.
• Saved position: File-mode accelerations store the last loaded snapshot with the acceleration data. Other accelerations keep it in memory. A full refresh saves the current snapshot, so the next append refresh continues from it.
• Computed columns: A dataset that computes columns over the Iceberg table, such as embedding columns, still reads by time.
• Overlap: refresh_append_overlap now applies only to the first load and to refreshes that read by time. Datasets that follow snapshots no longer log the startup warning that asks for refresh_append_overlap.

With a primary key, a dataset on DuckDB, SQLite, or another accelerator that keeps the last row to arrive still reads by time. This matches rc.2. A primary key that includes the time_column is the exception: such a dataset follows snapshots. On those accelerators, a late row with an older time would replace the newer stored row. If such a dataset has no refresh_append_overlap, the runtime logs this warning when it loads the dataset:

Dataset 'acc' (`engine: duckdb`) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. Use `engine: cayenne` to load every new row, or set `refresh_append_overlap` to how late rows can arrive. See: https://spiceai.org/docs/features/data-acceleration/data-refresh

To load every late row on such a dataset, use engine: cayenne, or set refresh_append_overlap to how late rows can arrive. Append refresh on other data sources does not change. See append refresh, #15147, and #15346.

Spice Cayenne Bug Fixes​

• Stale file lists (rare): In rare cases, Cayenne kept a stale file list for a dataset in mode: file. Queries then missed the rows of an appended file with no error. After retention removed a file, every query failed with Failed to open Vortex file … not found instead. Later appends did not repair the list, but a restart did.

  The fault needed a secondary-index build or some queries to list the files while retention or an append changed them. Tests reproduced it only with a delayed file listing. No spiced run has shown it so far. Cayenne now lists files under the same lock that its writers take. See #15304.

• Vortex file errors after a load: Some keyed Cayenne datasets on network block storage no longer fail every query and refresh with Failed to open Vortex file … not found. This was a regression in earlier v2.4.0 release candidates. The fault needed all of these conditions:

  • mode: file, a primary_key, and key deletion mode (cayenne_deletion_mode: key, or auto when it uses key deletes)
  • a first load or a full refresh with more than one version of some keys
  • network block storage: storage_profile: ebs, or auto when it resolves to EBS or NAS

  Cayenne rewrote the files that held older versions and deleted the originals, but a cached file list still named the deleted files. The data on disk was correct, and a restart cleared the error. Tests reproduced the bug on rc.2. Cayenne now clears the cached list after it deletes the files. See #15303.

• MySQL CDC snapshots (rc.1 regression): A MySQL dataset with refresh_mode: changes no longer fails its initial snapshot on large tables when a Cayenne file acceleration has maintained aggregates. The error was Bootstrap snapshot error: snapshot row read: … bytes remaining on stream, and the dataset stayed in the Error state.

  Since rc.1, Cayenne ran a checkpoint before each snapshot batch. The checkpoint could rebuild the maintained aggregates with a scan of the whole table. The MySQL reader then waited longer than net_write_timeout, and MySQL closed the connection. Cayenne now runs this checkpoint only when it can hold CDC rows in memory, as before rc.1. In a test with 40 million rows and net_write_timeout=2, the snapshot completed in 36–44 seconds. See #15315.

Cayenne Caching Mode Fixes​

These fixes apply to Cayenne datasets with refresh_mode: caching.

• Removed rows served from the cache: An empty origin response now replaces the cached entry of a GET request. Before this release, Cayenne refused that write when the origin started to answer [] for a cached request. Each refusal logged An empty cache replacement requires the complete logical request key. The query that received [] still returned no rows. Later queries could return the rows that the origin had removed:

  • With caching_stale_if_error: enabled, the next origin failure served the removed rows.
  • With caching_stale_while_revalidate_ttl, lookups served the removed rows for the rest of that window. Each lookup started another refresh that Cayenne refused.

  An empty response now replaces only the entry of its own request. Cayenne still refuses an empty response when the lookup filters do not name one request, and it logs An empty cache replacement requires filters that name one request. Entries for a POST request or another query string under the same path stay. DuckDB, SQLite, and Arrow caching datasets still have this fault, because their writer drops empty responses. To get this fix, use engine: cayenne. See #15365.

• Dropped cache writes under load: Cayenne no longer drops most cache writes above about 100 cache misses per second. Before this release, Cayenne applied each replacement of a cached entry as a separate write. Under load, Cayenne fell behind. The runtime then dropped most writes with Cache write admission timed out. The cache held only part of what it served, so caching_stale_if_error had little data to fall back on. Cayenne now combines replacements and new entries into one write when their requests cannot overlap.

  In a test, a client sent 500 lookups per second over 2.5 million random keys for 300 seconds. Cayenne used mode: file, caching_ttl: 60s, and caching_stale_if_error: enabled. Two runs of each build gave these results:

  Measure	Before	After
  Cache writes dropped	121,957–122,508	0
  Keys stored	15,915–16,624	143,356–143,488
  Request P99	361–363 ms	9 ms

  In a second test, the client cached 60,000 keys, and then the origin stopped. Before the change, 2,932 of 5,001 repeated lookups failed. After the change, 0 of 5,000 failed.

  Cayenne also deletes a replaced entry by its primary key when the request names every primary-key column and the new response has rows. Before, each delete read every stored row, so its cost grew with the cache. In a 60-minute test, the share of CPU for these deletes during origin recovery fell from 3.6% to 0.2%.

  In that phase, P99 latency rose from 303 ms to 433 ms, and throughput fell from 487 to 459 requests per second. The extra writes added compaction work. The test limited the runtime to 8 CPUs and 4 GiB. See #15253 and #15312.

Iceberg Bug Fixes​

• Old or uncommitted versions of Hadoop tables: Spice now reads the committed version of a Hadoop-layout Iceberg table. Before this release, a stale version-hint.text kept SQL queries and the Iceberg REST API on an older snapshot. Without a hint, Spice could read the uncommitted metadata file of a writer as the current table. The fix applies to a table read from its root directory, a dataset pinned to version-hint.text, and an Iceberg catalog over a Hadoop warehouse.

  A Hadoop writer commits a version when it renames <uuid>.metadata.json to v<N>.metadata.json. It updates version-hint.text after that, on a best-effort basis. Spice now treats the hint as a starting point and checks v<N+1>, v<N+2>, and later versions, as Iceberg Java does. A metadata file that a bare UUID names no longer counts as a version. See #15142.

• Tables whose last commit changed only the schema: Queries on an Iceberg table no longer fail when its last commit was ADD COLUMN or RENAME COLUMN with no write after it. Such queries failed with DataInvalid => Column value not found in table. A renamed column did not resolve, an added column was hidden, and a dropped column stayed visible. An Arrow acceleration of the table never loaded, so /v1/ready returned 503.

  A schema-only commit creates no snapshot, and the scan used the schema of the last snapshot. Scans now use the current schema of the table. A scan pinned with snapshot_id keeps the schema of its snapshot. v2.3.2, rc.1, and rc.2 had this bug. A column that someone adds while the runtime runs stays hidden until a restart. Distributed scans pinned to a snapshot still use the schema of that snapshot. See #15334.

AI and Observability Bug Fixes​

• Decisions on Bedrock models: ai_if, ai_probability, ai_classify, ai_score, and /v1/decisions now return answers from Bedrock models. Before this release, each such request failed with Evaluation model '<name>' failed: the model refused to answer: . rc.1 and rc.2 had this bug.

  The Bedrock adapter set an empty refusal on every non-streaming reply, and the evaluator read it as a refusal. The adapter no longer sets it, so /v1/chat/completions replies from Bedrock models no longer carry "refusal": "". The evaluator now ignores an empty refusal beside an answer.

  This also covers an OpenAI-compatible model whose upstream is an rc.1 or rc.2 runtime that serves Bedrock. A refusal with text still fails the request. ai_decide uses the same evaluator, but the tests for this fix did not cover it. See #15270.

• OTLP log trace IDs for pinned queries: A query that a client pins to a trace ID with spice-trace-id or traceparent now exports all of its OTLP log records under that ID. Before this release, records that the runtime logged while the query plan ran carried another trace ID. That ID matched neither the spice-trace-id response header nor the runtime.task_history row. A backend filter on the trace ID missed those records.

  Chat, NSQL, Responses, and decisions requests with a pinned trace ID use the same code path. The tests for this fix covered SQL queries only. This bug did not affect queries without a pinned trace ID. OTLP log export first shipped in rc.2. See observability and #15363.

Distributed Queries Lend Task Slots During HTTP Waits​

In a distributed query, a task that waits on a slow HTTP response no longer holds its executor task slot for the whole wait. The task lends the slot to other queries until the response arrives. It then takes a slot back before it continues. Before this release, a few HTTP requests to a slow origin could hold every slot, and other queries waited behind them.

This change needs no configuration. Other data sources and non-distributed queries do not change. In a test, slow HTTP requests filled every executor slot, and a fast query still finished while those requests waited. See #15123.

TLS Certificate Verification​

The runtime now uses rustls-webpki 0.103.16 and version 1.0.10 of the webpki-roots and webpki-root-certs root certificate bundles. The bundles follow the current root store of Mozilla. rustls-webpki 0.103.16 rejects a certificate chain that passes through a CA key that upstream no longer trusts. This also applies to cross-signed copies of that key.

The check applies wherever rustls verifies a chain: the bundled roots, rustls-native-certs, and rustls-platform-verifier on Linux. A connection to a server with such a chain now fails certificate verification. Connections through OpenSSL or native-tls use the trust store of the operating system and do not change. See #15374.

Dependency Updates​

The upstream versions of DataFusion, Arrow, Iceberg, and Ballista do not change.

Upgrading​

An upgrade from rc.2 needs no configuration change. TLS connections that rustls verifies now fail if the server's certificate chain passes through a CA that Mozilla no longer trusts. See TLS Certificate Verification.

To upgrade to v2.4.0-rc.3 once the release artifacts are available, use one of the following methods:

CLI:

spice upgrade v2.4.0-rc.3

Homebrew:

brew upgrade spiceai/spiceai/spice

Homebrew follows the version published in the tap. Use the explicit CLI version or Docker tag to select this release candidate.

Docker:

Pull the spiceai/spiceai:2.4.0-rc.3 image:

docker pull spiceai/spiceai:2.4.0-rc.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.3

AWS Marketplace:

Spice is available in the AWS Marketplace. Marketplace availability follows its published versions.

Cookbook Updates​

• New Decisions API recipe: call /v1/decisions with TypeSafe Jev and GPT-6 Luna.
• The PostgreSQL CDC recipe now accelerates with Cayenne.

The Spice Cookbook provides recipes to help you get started with Spice.

Contributors​

• @bjchambers
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

What's Changed​

Changelog​

• fix(iceberg): find the committed Hadoop metadata version by @phillipleblanc in #15142
• feat(iceberg): append refresh follows Iceberg snapshots so late-arriving rows are not missed by @sgrebnov in #15147
• Drive accelerator refresh writes on the refresh runtime by @sgrebnov in #15262
• fix(models): answer ai_* decisions from Bedrock models instead of failing them as refusals by @krinart in #15270
• fix(cache): coalesce Cayenne cache replacements so caching mode keeps up with sustained misses by @krinart in #15253
• feat(cluster): lend a distributed task's slot while it waits on an HTTP source by @peasee in #15123
• fix(cayenne): evict a folded snapshot's cached listing once its superseded files are removed by @bjchambers in #15303
• fix(cache): delete replaced Cayenne cache entries by primary key, not by scanning the table by @krinart in #15312
• fix(cayenne): list snapshot directories under the listing fence by @bjchambers in #15304
• fix(cayenne): checkpoint before a durable CDC write only when a source callback is installed by @phillipleblanc in #15315
• fix(iceberg): read tables whose last commit is schema-only (ADD/RENAME COLUMN) by @lukekim in #15334
• fix(runtime): skip the refresh_append_overlap warning for Iceberg append datasets by @sgrebnov in #15340
• fix(spiced): raise the soft open file limit to the hard limit at startup by @bjchambers in #15339
• fix(iceberg): keyed DuckDB/SQLite append refreshes read by time so late rows don't overwrite newer versions by @sgrebnov in #15346
• fix(acceleration): restore the strict high-water mark in append dedupe by @sgrebnov in #15358
• fix(deps): bump rustls-webpki to 0.103.16 and webpki-roots to 1.0.10 by @phillipleblanc in #15374
• fix(observability): export a pinned query's execution log records under its trace id by @krinart in #15363
• fix(caching): let an empty origin response replace a cached HTTP entry by @krinart in #15365

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.4.0-rc.2...v2.4.0-rc.3

Spice v2.4.0-rc.2 is now available! 🔥

Spice v2.4.0-rc.2 is the second release candidate of v2.4.0 for early testing and feedback. It introduces a new OpenAI-compatible POST /v1/decisions API backed by an OpenAI decision model, TypeSafe Jev, or any other LLM in the Spicepod that can answer decisions. Five new SQL functions bring the same AI classification, scoring, and decisions into SQL.

This release also adds Iceberg reads without a catalog and improves acceleration snapshots. Cayenne gains runtime-wide adaptive tuning and applies acceleration configuration changes on restart.

Highlights in this release:

• AI Decisions API and decisions in SQL: An OpenAI-compatible /v1/decisions API, plus five SQL functions for filtering, classifying, and scoring data with models.
• Iceberg without a catalog: Read a table directly from its root directory or pin a metadata file to read a fixed version.
• Shared runtime state: Startup validation and a common location for cluster state, rate control, cache warmup, and acceleration snapshots.
• SQL results cache: Cache entries expire by TTL and remain available across data refreshes.
• Cayenne acceleration: Improvements to primary-key loading, secondary indexes, and configuration updates.
• OpenTelemetry log export: Send runtime logs to any OTLP endpoint as structured log records, and choose which failed queries are logged.

These notes cover changes since v2.4.0-rc.1. Review Breaking Changes before upgrading; upgrades from earlier versions also require the rc.1 migration guidance.

What's New in v2.4.0-rc.2​

AI Decisions API and decisions in SQL​

Spice now supports POST /v1/decisions, which is compatible with the OpenAI Decisions API. A decision request asks typed questions about an input. Each answer comes back with its probabilities and a confidence.

Any model in the Spicepod can answer a decision request:

OpenAI decision models use the native OpenAI Decisions API. gpt-6-luna also supports chat and /v1/nsql. TypeSafe Jev returns calibrated probabilities and answers decisions only. Other LLMs answer through chat completions and return their own uncalibrated estimates.

models:
  - from: openai:gpt-6-luna
    name: luna
    params:
      openai_api_key: ${secrets:OPENAI_API_KEY}
  - from: typesafe:jev
    name: jev
    params:
      typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Five new SQL functions make the same decisions available in queries:

SELECT
    id,
    ai_classify(body, ['billing', 'technical', 'account'], model => 'support_model' ) AS team,
    ai_probability(body, 'The customer is requesting a refund', model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI functions for more examples.

Iceberg Without a Catalog​

Read an Iceberg table directly from object storage or the local filesystem, without configuring a catalog. Set from to the table root using file://, s3://, s3a://, gs://, or gcs://:

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1

Spice resolves the metadata file on each query and refresh. It supports metadata/version-hint.text as well as a fixed version (a path relative to the table root or a full URL):

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1
      iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Fixed tables are read-only. The earlier metadata_path parameter remains supported with a deprecation warning; see Breaking Changes.

Shared Runtime State​

runtime.state provides a shared, object-store location for cluster state, rate control, SQL results-cache warmup, and acceleration snapshots. Each feature uses its own subdirectory within the provided location.

Acceleration snapshots now default to the snapshots/ subdirectory when runtime.state is configured.

runtime:
  state:
    location: s3://my-bucket/spice/

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled

A dataset can override the snapshot location, its settings, and the bootstrap behavior on its own acceleration:

datasets:
  - from: postgres:events
    name: events
    acceleration:
      snapshots: enabled
      snapshots_location: s3://archive-bucket/snapshots/
      snapshots_params:
        s3_region: eu-west-1
      snapshots_bootstrap_on_failure_behavior: fallback

Other changes to shared state:

• Snapshot queues: An SQS queue that covers the whole runtime.state prefix no longer causes an ERROR log for each scheduler heartbeat. The snapshot reader now deletes messages for the other state folders without a log.

SQL Results Cache​

The SQL results cache now expires entries by TTL, independent of acceleration refreshes, CDC batches, and retention passes no longer invalidate cached results.

Review item_ttl and stale_while_revalidate_ttl when upgrading: a successful refresh no longer makes cached results reflect the refreshed data immediately. See Breaking Changes.

Cayenne Acceleration​

Adaptive tuning is configured once for the runtime. Enable it with runtime.adaptive_tuning and set targets under runtime.params:

runtime:
  adaptive_tuning: enabled
  params:
    target_freshness: 30s

Supported targets are target_replication_lag, target_freshness, target_query_latency, target_convergence_window, and target_qph. Adaptive tuning defaults to disabled. See Breaking Changes and the runtime YAML documentation.

Other Cayenne improvements include:

• Primary-key loading: Keyed tables build their primary-key filter during the initial load and bulk-load without per-row routing or a whole-table fold.
• Position deletes: cayenne_deletion_mode: auto can select position deletes for small primary-key tables on local storage on Linux systems with at least eight physical cores.
• Secondary indexes: Index runs are persisted, indexes remain complete across acceleration snapshots, and CDC maintains indexes for each operation type. Hash joins read each file once when narrowing a scan through an index.
• Partition handling: Cayenne rejects stored data with an incompatible partition_by, protects keyed appends to partitioned tables during publication, and prepares every partition after a refresh.
• Storage and query correctness: Cayenne reclaims snapshot directories superseded by a reload, corrects pruning for NaN values, and fixes deletion-index bounds and reload fencing.
• Snapshot storage growth: A writer with snapshots_trigger: refresh_complete kept one more copy of the table for each refresh. Each snapshot held file deletions, so Cayenne skipped each cleanup pass. Cayenne now runs the cleanup when a snapshot releases its hold. In a 10-minute test with a refresh every 60 seconds, each dataset kept 1 snapshot directory instead of 10.
• Configuration changes on restart: A change to write settings only, such as cayenne_sort_columns or cayenne_compression_strategy, applies to the existing acceleration. A change to primary_key or partition_by rebuilds the acceleration from the source only under specific conditions. See Cayenne Configuration Updates.

OpenTelemetry Log Export​

Spice can now send its logs to an OpenTelemetry Collector, or any OTLP endpoint, as structured OTLP log records. Logs use the same otel_exporter that already sends metrics. A new runtime.query.log_errors setting chooses which failed queries are logged.

runtime:
  telemetry:
    otel_exporter:
      endpoint: otel-collector:4317
      signals: [metrics, logs] # new; default [metrics]
  query:
    log_errors: execution # new; none (default) | execution | all

• Structured records: Each log line becomes an OTLP log record with its severity, the message as the body, typed fields as attributes, and the logging module as the instrumentation scope. Records share the resource attributes of exported metrics.
• Trace correlation: A record logged during a query carries the query's trace ID, the same ID returned in spice-trace-id and stored in runtime.task_history.
• Delivery: Records are batched on a dedicated export runtime, separate from query and HTTP traffic, and flushed at shutdown so the final error and Goodbye! reach the endpoint. For OTLP/HTTP endpoints, logs are sent to <base>/v1/logs.
• Cluster executors: Executors export their logs with the scheduler's telemetry settings, including headers resolved from the scheduler's secrets.

runtime.query.log_errors controls which failed queries are logged:

Each record reads Query failed: <error>, with error_code and datasets fields and the query's trace ID. The SQL text is never logged.

Unknown-column errors are now reported with the correct error_code instead of InternalError, in logs, runtime.task_history, and the query_failures metric.

See OpenTelemetry Logs Export, Exporting logs, and runtime.query.log_errors.

Fixes and Improvements​

AI APIs and Models​

• Responses API: /v1/responses executes Spice tools and honors tool_recursion_limit when streaming. It returns Spice tool calls as mcp_call items in output, allowing clients that manage their own conversation history to replay them.
• Model controls: /v1/nsql accepts reasoning_effort, and rerank() uses evaluation models with a strategy selected for each query.
• Anthropic: Streaming handles responses that begin with a thinking block, empty text blocks are dropped, and anthropic_workspace_id is available as a model parameter.
• OpenAI: Models with an unknown service tier can load and serve requests.
• Luna: gpt-6-luna and its snapshots support chat, /v1/nsql, and native decisions.

Acceleration​

• Caching point lookups: Lookups on a refresh_mode: caching dataset failed with HTTP 400. This occurred when Cayenne accelerated the dataset, the primary key covered the request columns, and the table held two protected snapshots. The point-lookup plan did not combine the partitions of the Cayenne scan. Lookups now return the cached row or fetch it from the source.

Connectors​

• MySQL timestamps: Timestamp literals in queries that the runtime sends to MySQL now keep their fractional seconds.

Dependency Updates​

Breaking Changes​

Shared State Directory Layout and Startup Checks​

State written at the runtime.state root by rc.1 is no longer read from that location. Move existing state objects into the appropriate subdirectories before upgrading to retain them:

Azure locations (abfs:// and abfss://) need one more step. In rc.1, the runtime ignored the path in an Azure URL and wrote state at the container root. For example, with abfs://container/spice/, rc.1 wrote cluster.json at the container root. This release writes spice/cluster/cluster.json. Move Azure state from the container root into the configured path and then into the feature subdirectory.

An explicit runtime.scheduler.state_location retains its earlier layout. Without migration, Spice starts without the state previously stored at the shared root.

The configured store must accept writes and conditional writes. Spice does not become ready until startup validation succeeds and refuses stores with a newer state format. Check storage permissions and compatibility before upgrading.

SQL Results Cache Freshness​

A refresh no longer invalidates SQL results-cache entries. Queries can return data from before the latest refresh until item_ttl expires, plus stale_while_revalidate_ttl when configured. Reduce item_ttl if results need to follow source refreshes more closely.

Cayenne Tuning Configuration​

Replace the Cayenne tuning parameters with runtime settings that apply across datasets and catalogs:

runtime.adaptive_tuning accepts only enabled or disabled. Other values, including true, fail the Spicepod load.

Pinned Iceberg Tables​

Replace the deprecated metadata_path parameter with iceberg_metadata_path. The two settings resolve paths differently. metadata_path uses its value as given. iceberg_metadata_path resolves a value without a URL scheme relative to the table root in from. Convert an absolute local path to a full file:// URL or to a path relative to the table root.

Datasets pinned to a .json metadata file are read-only and fail to load with access: read_write. Remove that setting from pinned datasets before upgrading.

Cayenne Configuration Updates​

A restart now applies changed settings to an existing Cayenne acceleration. The result depends on the setting and the dataset:

• Write settings only: A change to only cayenne_sort_columns, cayenne_cluster_by, cayenne_compression_strategy, cayenne_delta_encoding, or cayenne_shard_key_columns applies to the existing acceleration. Stored rows keep their layout until a refresh or compaction rewrites them.
• Identity settings: A change to primary_key, partition_by, cayenne_pk_conflict_detection from none to auto, or the data location rebuilds the acceleration from the source. The runtime rebuilds only a dataset with refresh_mode: full, mode: file or file_update, and write_mode: write_through. The source must not be sink:, and must be readable with the schema of the dataset.
• All other datasets: The acceleration keeps all of its stored settings. This includes write settings that changed in the same restart. The runtime logs a warning that names the change and the fix.

While the runtime cannot read the source, a dataset that qualifies for a rebuild keeps its existing acceleration. A partition_by change or a conflict-detection change cannot use the stored rows, so it fails the dataset instead. Revert the setting, or restart when the source is readable.

A restart alone does not migrate a dataset that does not meet these conditions. Read the startup warning for the dataset before you delete stored data. Allow time for a rebuild when the conditions apply.

Upgrading​

Once release artifacts are available, select v2.4.0-rc.2 explicitly with the CLI, Docker tag, or Helm chart version.

CLI​

spice upgrade v2.4.0-rc.2

Docker​

docker pull spiceai/spiceai:2.4.0-rc.2

See Docker Hub for available tags.

Helm​

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.2

Homebrew and AWS Marketplace​

Homebrew and AWS Marketplace follow their published versions. To select this release candidate, use an explicit version through the CLI or Docker.

Spice is also available in the AWS Marketplace.

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @spiceemma

Changelog​

See the full changelog from v2.4.0-rc.1 to v2.4.0-rc.2.

• feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
• fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588
• fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart in #14851
• fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice in #14792
• fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-duckdb[file]-2gib.yaml by @github-actions in #14850
• Update openapi.json by @github-actions in #14858
• test: strengthen assertions and enforce sign-off coverage by @lukekim in #14740
• Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart in #14888
• fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice in #14884
• fix(anthropic): drop empty text blocks from assistant messages by @Jeadie in #14834
• fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim in #14882
• test(sqlite): guard the Date32 literal cast through the SQLite acceleration's own scan (fixes #14491) by @grokspice in #14703
• fix(cayenne): read each file once when a hash join narrows an indexed table's scan by @lukekim in #14934
• fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc in #14946
• fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc in #14939
• fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim in #14944
• fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim in #14955
• fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc in #14942
• fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc in #14960
• feat(spicepod): publish pull request reviews by @lukekim in #14964
• fix(ci): bound the REPL expect-script waits on silence, not total response time (fixes #13711) by @grokspice in #14963
• fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc in #14957
• fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc in #14956
• fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim in #14798
• fix(ci): provision DuckLake for the ducklake[catalog] schema test, which has never passed by @grokspice in #14952
• fix(llms): stream Anthropic responses that open with a thinking block (fixes #14909) by @krinart in #14920
• fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov in #14968
• feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim in #14852
• fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart in #14972
• fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart in #14927
• feat(cayenne): always persist secondary index runs by @sgrebnov in #14979
• fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers in #14919
• feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim in #14838
• fix(rrf): keep named arguments on nested search calls by @phillipleblanc in #14975
• fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart in #14932
• fix(vortex): keep temporal casts Vortex can't evaluate out of the scan by @sgrebnov in #14985
• fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers in #14928
• test(huggingface): unblock the merge queue: read imdb's parquet conversion through @~parquet and fail fast on a vanished Hub pin by @grokspice in #14998
• test(cayenne): link the integration tests into one test binary by @bjchambers in #14976
• test(vortex): compile the integration tests into a binary the sign-off gate runs by @bjchambers in #14987
• fix(http): reject a request_path that URL parsing would rewrite by @bjchambers in #14965
• fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc in #14953
• Add anthropic_workspace_id param for Anthropic models by @Jeadie in #14829
• fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim in #14783
• test: run federated source tests on every supported LTS release line by @lukekim in #14936
• fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius in #14791
• fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim in #14806
• fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim in #15009
• Improve types methods in runtime-rate-control by @Jeadie in #14573
• fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart in #14930
• Update openapi.json by @github-actions in #14908
• test(cayenne): run the heap-measuring tests in one isolated test binary by @bjchambers in #15010
• test(vortex): run vortex-datafusion's integration tests in the sign-off gate by @bjchambers in #15001
• fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc in #15019
• Update openapi.json by @github-actions in #15018
• fix(ci): keep the OpenAPI regeneration PR current instead of skipping while one is open by @phillipleblanc in #15023
• fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim in #14993
• fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers in #14891
• chore: Update Turso crate to 0.8.2 by @spiceemma in #15022
• feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee in #14844
• ci: apply whole-binary nextest overrides to the module test sets they select by @bjchambers in #15011
• fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc in #15028
• test(data_components): compile the integration tests into one test binary by @bjchambers in #14991
• fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc in #15053
• fix(runtime): give each runtime.state feature its own folder by @phillipleblanc in #15036
• test(data_components): make the HTTP provider snapshots independent of the machine by @bjchambers in #15000
• fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc in #15054
• test(runtime): drive the cdc_cayenne tests through Cayenne's change sink by @bjchambers in #15003
• ci(object_store_occ): run the WAL E2E suite on self-hosted runners against disk, MinIO, RustFS and spiceio by @lukekim in #15031
• test(llms): compile the credential-free tests into the integration binary by @bjchambers in #14990
• fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov in #15068
• ci(setup-minio): name the missing secret instead of failing in mc's argument parser by @grokspice in #15073
• ci: run Substrait Mode B on the Linux runners that can reach the trunk spiced builds by @grokspice in #15074
• feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie in #14782
• build(deps): bump the github-actions-dependencies group across 5 directories with 7 updates by @dependabot in #15027
• fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov in #15069
• feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov in #15081
• Fixes for MCP server by @Jeadie in #15041
• fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice in #15056
• fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice in #15037
• perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers in #15072
• fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart in #14931
• fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart in #14937
• fix(decisions): Fixes for /v1/decisions by @Jeadie in #15040
• perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers in #15076
• fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers in #15077
• test(runtime,spiced): compile integration tests into declared binaries, run the ones no CI job ran, and remove the uncalled supersede_initial_load by @bjchambers in #15005
• fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim in #15083
• ci(e2e): don't run rust-cache on the macOS build, whose save deletes cargo from sibling runners by @grokspice in #15100
• feat(testoperator): record per-query search metrics by @Jeadie in #15090
• fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc in #15096
• feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie in #14340
• ci: run merge-queue Mode A TPC-H and LLM integration tests on spiceai-macos-large by @lukekim in #15116
• docs(cayenne): the per-file fold merges no source files, but may roll by @bjchambers in #15101
• build(deps): bump async-openai to the fork's merge of upstream main by @krinart in #14980
• feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov in #15095
• fix(runtime): fail recovery of an unsaved running query by @phillipleblanc in #15092
• perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc in #15099
• test(cayenne): guard per-file statistics sharing for listings outside a scan view by @bjchambers in #15103
• feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers in #15033
• fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers in #14140
• Change codeql runner from spiceai-macos to spiceai-macos-large by @lukekim in #15120
• fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim in #15106
• fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim in #15108
• fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers in #15111
• fix: Update Search integration test snapshots by @github-actions in #15050
• test(iceberg): read the Glue fixture unpinned and check the picked metadata file by @sgrebnov in #15115
• test(cayenne): differential and plan coverage for secondary indexes on partitioned accelerations by @bjchambers in #15113
• test: answer results-cache misses from the acceleration while the source is down by @bjchambers in #15112
• ci(lint): fail on test files no target compiles, and document the integration test layout by @bjchambers in #15109
• feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov in #15117
• fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov in #15110
• fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart in #14977
• feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim in #15042
• fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim in #15105
• fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc in #15128
• fix(test): wait for pkd_ctl to load before reading its bootstrap row in the write-back pk_conflict test by @grokspice in #15102
• fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart in #15126
• test(mysql): skip the SQL results cache in the binlog replication end-to-end tests by @grokspice in #15141
• feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc in #15138
• feat(acceleration): apply changed acceleration settings on restart by @lukekim in #15094
• fix(models): enable Luna chat and NSQL support by @krinart in #15166
• fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc in #15177
• fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov in #15179
• feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim in #15176

Spice v2.1.4 is now available! 🛠️

Spice v2.1.4 is a patch release that improves DuckDB acceleration and runtime stability: datasets with UTC timestamp columns, such as Iceberg tables, now load successfully when accelerated with DuckDB, retention policies run reliably, and the runtime is more resilient under heavy query load.

What's New in v2.1.4​

DuckDB Acceleration Works with Iceberg Timestamps​

A dataset with a UTC timestamp column — for example, any Iceberg timestamptz column — could previously fail to load when accelerated with DuckDB, leaving the dataset unhealthy and unqueryable. These datasets now load and become ready normally, with no configuration changes needed.

Reliable Retention Policies on DuckDB​

Retention policies now apply cleanly on DuckDB-accelerated datasets, including policies that combine a time window with an additional condition. Expired rows are evicted on every retention interval, keeping accelerated data fresh and storage bounded.

Improved Runtime Stability​

The runtime is now more robust when queries are cancelled — whether by a client disconnecting, a timeout, or a new refresh superseding an in-flight read. A rare crash in this path has been eliminated.

Contributors​

• @phillipleblanc
• @peasee

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.4, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.4 image:

docker pull spiceai/spiceai:2.1.4

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.4

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• chore(deps): bump datafusion and table-providers for the DuckDB timezone and retention fixes (2.1 backport) by @phillipleblanc in #12546
• fix(deps): bump the Vortex pin to pick up the task-cancellation fix by @phillipleblanc in #12547

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.3...v2.1.4

Spice v2.1.3 is now available! 🛠️

Spice v2.1.3 is a patch release focused on resource efficiency: explicit CPU sizing with the new runtime.cpu.cores setting, more query memory for Cayenne deployments, and improved memory and crash diagnostics. It also fixes Cayenne acceleration of Iceberg datasets with timestamptz columns and restores WHERE filters on federated LEFT/RIGHT JOIN queries.

What's New in v2.1.3​

CPU Sizing with runtime.cpu.cores​

The new runtime.cpu.cores setting controls how many cores the runtime targets. Thread pools, query partitioning, and accelerator concurrency are all derived from it.

runtime:
  cpu:
    cores: 4 # `auto` (default) detects. Accepts 4, 3.5, 3500m

Also available as --cpu-cores and SPICE_CPU_CORES (precedence: flag > environment > Spicepod).

This is most useful on large, shared nodes. A pod that sets resources.requests.cpu without a CPU limit exposes no cgroup quota, so the runtime sizes itself for every core on the node rather than its allocated share. Setting the entitlement aligns parallelism and memory footprint with the CPU the pod actually receives.

The effective value, its source, and the derived sizing are logged at startup and exported as the spiced_cpu_budget_cores gauge.

More Query Memory for Cayenne Deployments​

The Cayenne compaction memory pool is now reserved only for accelerations that can compact into it: file mode with a small-write refresh profile. Other deployments, including refresh_mode: full, keep the full memory limit available to queries — up to 6.4 GiB on a 32 GiB limit, with no configuration change.

Memory budgets are now derived from the process's own cgroup limit rather than total host memory.

Diagnostics​

Three new gauges report memory in use: query_memory_pool_used_bytes, cayenne_compaction_memory_pool_used_bytes, and process_resident_memory_bytes.

Memory pool refusals now return ResourcesExhausted and HTTP 503, distinguishing them from query errors.

Fatal native signals (SIGSEGV, SIGBUS, SIGILL, SIGFPE) report the signal, faulting address, and thread before exit, so a crash can be diagnosed from logs.

Fixed a bug where setting runtime.task_history.enabled: false also disabled every query metric — query_duration_ms, query_execution_duration_ms, query_executions, query_failures, query_returned_rows, and query_returned_bytes. These are now reported regardless of the task history setting.

Cayenne Acceleration of Iceberg timestamptz Columns​

Accelerating an Iceberg dataset with a timestamptz column using the Cayenne engine previously failed during the refresh write with an error resolving the time zone +00:00. Iceberg maps every timestamptz column to the fixed-offset Arrow time zone +00:00, which the file writer could not resolve when building column statistics. Fixed-offset time zones (±HH:MM, ±HHMM, and ±HH) are now resolved wherever time zones are handled, so these datasets accelerate correctly.

Federated Outer Join Filters​

A federated query combining a LEFT JOIN with a WHERE filter on the left table previously returned all rows instead of the filtered rows: when the query was pushed down to the data source or accelerator, the filter was folded into the JOIN ON clause, where it no longer filters the left side (RIGHT JOIN was affected symmetrically). Filters now stay on the side of the join they came from, so these queries return the correct rows.

Contributors​

• @lukekim
• @sgrebnov
• @phillipleblanc
• @bjchambers
• @Jeadie

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.3, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.3 image:

docker pull spiceai/spiceai:2.1.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.3

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• feat(runtime): size every CPU-derived pool from the CPU entitlement by @bjchambers in #12276
• fix(runtime): carve the Cayenne compaction memory pool only when a dataset can compact into it by @sgrebnov in #12326
• fix(runtime): size memory budgets from the process's own cgroup limit by @lukekim in #12263
• fix(telemetry): read the cgroup CPU quota along the whole cgroup path by @sgrebnov in #12300
• feat(runtime): expose the memory numbers that explain an OOM as gauges by @lukekim in #12195
• fix(runtime): report a memory-pool refusal as ResourcesExhausted and answer it with 503 by @sgrebnov in #12289
• fix(cayenne): the write-concurrency raise must respect the memory brake by @lukekim in #12317
• feat(spiced): report fatal signals before exit by @sgrebnov in #12334
• fix: report query metrics when task history is disabled by @sgrebnov in #12227
• chore(deps): repoint vortex at the 2.1 fixed-offset timezone fix by @phillipleblanc in #12455
• chore(deps): bump datafusion rev for outer-join unparser fix by @Jeadie in #12460

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.2...v2.1.3

Spice v2.1.0 is now available! 🎉

Spice v2.1.0 is the next minor release of Spice, headlined by high-throughput Cayenne CDC, scaling and resilience improvements to PostgreSQL logical replication, expanded distributed query with Iceberg catalog scans and broadcast joins, and the upgrade to DataFusion v54 (including v53), Arrow v58.3, and Vortex v0.74. The release also adds experimental adaptive self-tuning for the Cayenne accelerator, distributed GLM inference, and a range of security, search, and connector improvements.

Highlights in v2.1.0 include:

• High-Throughput Cayenne CDC — in-memory CDC tier, dedicated compaction runtime, and write-path optimizations that cut replication lag on high-volume CDC workloads
• PostgreSQL Replication at Scale — multiple changes-mode datasets share a single replication slot, unchanged-TOAST recovery, and resilient reconnects across rolling deploys
• Distributed Query — distributed Ballista scans of Iceberg catalog tables, broadcast joins for small dimension tables, and shared scheduler job state with failover
• DataFusion v54 — upgrade to DataFusion v54 (folding in v53), Arrow v58.3, and Vortex v0.74, bringing faster joins, scans, and planning
• Adaptive Self-Tuning (Experimental) — opt-in closed-loop tuning and maintained aggregates that adapt Cayenne to hardware, schema, and live workload

What's New in v2.1.0​

High-Throughput Cayenne CDC​

A major focus of v2.1 is Spice Cayenne write-path throughput for change-data-capture (HTAP) workloads:

• In-Memory CDC Tier: A new in-memory CDC tier and follow-ups cut replication lag on hot upsert tables, with bounded mem-tier checkpointing and O(1) per-scan deletion views, plus a two-phase off-fence checkpoint on the ingest path.
• Dedicated Compaction Runtime: A dedicated compaction runtime with CDC pipelining and protected snapshots isolates compaction from query and ingest paths, with parallelized deletion-vector writes, per-batch directory-barrier coalescing, and size-aware parallel encode for protected-snapshot compaction.
• Incremental Protected Snapshot Compaction: Incremental compaction of protected snapshots (used in Cayenne's merge-on-read deletion index) reduces disk usage and improves query performance.
• Smaller WAL & Metadata-Only Publish: cayenne_insert_record table IDs are stored as 16-byte raw-UUID BLOBs, cutting CDC WAL volume ~34%; upsert commits publish metadata-only, dropping per-key insert records; transient staged CDC deltas are light-encoded.
• Delta-Write Encoding Levels: A new cayenne_delta_encoding setting (default auto) selects delta-write encoding, and cayenne_compression_strategy: zstd is now fully wired.
• In-Memory CDC Sharding: PK-hash intra-apply sharding parallelizes in-memory CDC apply.
• Scan Safety Under Write: In-flight scans are ref-counted so snapshot GC can't delete Vortex files mid-read; in-RAM scan parallelism, query admission control, and sound scan output ordering improve read behavior under sustained CDC.

Delta-write encoding effort and Vortex compression are tunable per accelerator. cayenne_delta_encoding: auto (the default) size-gates fresh CDC/append writes — small deltas use a light scheme and are re-encoded during compaction — or pin an explicit level 0..10 (7 is the full default cascade); cayenne_compression_strategy selects the Vortex compression:

acceleration:
  engine: cayenne
  refresh_mode: changes
  params:
    cayenne_delta_encoding: auto # 'auto' (default), or pin a level 0..10 (7 = full cascade)
    cayenne_compression_strategy: zstd # 'btrblocks' (default) or 'zstd'

Change Data Capture & HTAP​

PostgreSQL logical replication (CDC, refresh_mode: changes, introduced in v2.0) gets significant scaling and resilience work in v2.1:

• Shared Replication Slot: Multiple refresh_mode: changes PostgreSQL datasets on the same connection can name the same pg_replication_slot to share a single replication slot, walsender decoder, and publication, with decoded changes multiplexed by (schema, table) to each dataset. This collapses the slot count from one-per-dataset to one — staying well under Postgres's default max_replication_slots = 10.

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # shared slot name
    acceleration:
      refresh_mode: changes
  - from: postgres:public.customers
    name: customers
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # same name -> one slot, walsender & publication
    acceleration:
      refresh_mode: changes

• Unchanged-TOAST Recovery: Under REPLICA IDENTITY FULL, when an UPDATE leaves a large TOASTed column unchanged, pgoutput sends an "unchanged" marker; Spice now fills that value from the old tuple — its old value is its current value — so updates no longer error or drop columns. Without an old tuple, the error persists with a hint to enable REPLICA IDENTITY FULL.
• Transient Walsender Contention: Slot-contention errors during rolling deploys — SQLSTATE 55006 ("replication slot is active for PID") and 53300 ("requested standby connections exceeds max_wal_senders") — are now classified as transient and retried with backoff instead of fatally terminating the stream. Replication connections are also released at shutdown start (not process exit), freeing walsender seats for replacement instances.
• Strict CDC Param Validation: PostgreSQL CDC parameters are strictly validated rather than silently defaulted.
• Debezium Schema Evolution: Fixes for Debezium schema-evolution support, including tombstone-message handling and sign-extension of minimal-width base64 decimals.

Distributed Query​

Distributed Query gains:

• Distributed Iceberg Catalog Scans: Ballista distributes scans of Iceberg catalog tables across executors.
• Broadcast Joins: Small dimension tables are broadcast to executors for distributed joins.
• Shared Scheduler Job State with Failover: Ballista job state is shared so the scheduler can fail over without losing in-flight work.

Performance & Query Engine​

Apache DataFusion is upgraded to v54, folding in v53, alongside Arrow v58.3 and Vortex v0.74 (with a pin bump adding intra-file decode split and a per-execution kernel cache). Two DataFusion releases land in this upgrade:

• DataFusion v54 (release notes): adds LATERAL joins, SQL lambda functions (x -> expr with array_transform/array_filter/array_any_match), spilling nested-loop joins, and a faster arrow-avro reader. Performance work includes morsel-driven Parquet scans (up to ~2x faster for skewed scans), 20-50x faster sort-merge semi/anti/mark joins, redundant-sort-key pruning, NDV-based cardinality estimation, and inner_product/cosine_distance functions.
• DataFusion v53 (release notes): adds LIMIT-aware Parquet row-group pruning, broader filter pushdown through joins and UNION, nested-field pushdown (get_field into the scan), faster query planning (some plans dropping from ~4-5ms to ~100us), and 42 faster built-in functions.

Federation deny-list enforcement and catalog DDL are restored after the DataFusion upgrades, and a cost-based left-deep join reordering rule is added for Cayenne acceleration.

AI & LLM​

• Native GLM Support with Distributed Inference: Native GLM model support with surfaced reasoning_content, including tensor-parallel GLM inference. Load a GLM model with model_type: glm4 (glm4moe and glm4moelite are also supported):

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4

For large models, GLM inference can be distributed across nodes (tensor parallelism) via the mistral.rs pure-TCP ring all-reduce backend — no NCCL/system dependency. This is a Spice.ai Enterprise feature requiring the distributed build. Run the same model on each node, changing only node_rank:

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4
      distributed_backend: ring
      nodes: 10.0.4.21,10.0.4.22 # ordered host/IP per rank; the ring backend currently requires exactly 2
      node_rank: 0 # rank of THIS node in [0, world_size); rank 0 serves the API. Set node_rank: 1 on 10.0.4.22

• NSQL Context Endpoint: A new GET /v1/nsql/context endpoint returns the SQL dialect, dataset schemas (with optional sample rows), and registered functions that Spice injects into natural-language-to-SQL (POST /v1/nsql) requests — useful for inspecting or caching exactly what the model sees:

# Inspect the context injected into /v1/nsql requests (examples_limit default 3, max 100)
curl "http://localhost:8090/v1/nsql/context?include_examples=true&examples_limit=3"

Returns the dialect, per-dataset schema (keys, indexes, searchable columns), the registered function inventory, and sample rows (abbreviated):

{
  "context": "# Spice.ai NSQL Context",
  "instructions": [
    "Write SQL for the Spice runtime, which uses Apache DataFusion with the SQL parser configured for the PostgreSQL dialect.",
    "Use table and column descriptions, primary keys, foreign keys, unique constraints, and indexes when choosing joins and filters."
  ],
  "sql": {
    "engine": "Apache DataFusion",
    "version": "54.0.0",
    "dialect": "PostgreSQL",
    "parser": "DataFusion SQL parser configured with PostgreSQL dialect"
  },
  "datasets": [
    {
      "name": "sales.orders",
      "table": "orders",
      "description": "Customer orders",
      "columns": [
        { "name": "order_id", "data_type": "Int64", "nullable": false, "primary_key": true, "indexed": true },
        { "name": "customer_id", "data_type": "Utf8", "nullable": false, "vector_search": true, "full_text_search": true }
      ],
      "primary_key": ["order_id"],
      "foreign_keys": [
        { "columns": ["customer_id"], "foreign_table": "spice.sales.customers", "foreign_columns": ["id"] }
      ]
    }
  ],
  "functions": {
    "summary": "Spice SQL runs on Apache DataFusion ... Run SELECT * FROM list_udfs() to inspect the full registered function inventory",
    "search": [
      { "name": "vector_search", "syntax": "vector_search(dataset, 'query text'[, column])" },
      { "name": "text_search", "syntax": "text_search(dataset, 'query text'[, column])" }
    ]
  },
  "samples": [
    { "title": "Example rows for `sales.orders`", "content": "| order_id | customer_id |\n| --- | --- |\n| 42 | CUST-1 |" }
  ]
}

Search & Vectors​

• S3 Vectors Pagination: QueryVectors paginates for top-K up to 10,000.
• Elasticsearch kNN Candidate Pool: The default kNN candidate pool is raised from 10 to 1000 for better recall.

SQL & Query Engine​

• FlightSQL Substrait Plans: CommandStatementSubstraitPlan support.
• Large Result Streaming: Flight streaming is optimized for large result sets.
• Write Authorization: The SQL tool allows writes for ReadWrite API keys.
• Schema Evolution Policies: on_schema_change supports widening-only evolution and a drop_and_recreate policy.

Security & Connectors​

• Kafka mTLS: Mutual TLS configuration is surfaced in the Kafka data connector.
• Secret Resolution at Startup: Secret references are checked and reported at startup.
• DuckDB HNSW: Upgrade to DuckDB v1.5.3 with the statically linked VSS (HNSW) vector extension.

Adaptive Self-Tuning (Experimental)​

The Spice Cayenne accelerator gains experimental opt-in self-tuning. cayenne_tuning: auto derives configuration from the detected hardware and inferred schema, while adaptive additionally runs a per-table closed-feedback controller that adapts flush caps, the in-memory CDC tier, compaction cadence, and write concurrency toward operator SLOs (replication lag, freshness, query latency, queries-per-hour). Cayenne can also maintain aggregates incrementally — with predicate-aware delta serving and incremental retraction — and fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics. These features are experimental and disabled by default.

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      refresh_mode: changes
      params:
        cayenne_tuning: adaptive # 'auto' (static, env- + schema-derived) or 'adaptive' (closed-loop)

Observability​

• Per-Dataset Query Attribution: The query_executions metric gains a datasets dimension.
• HTAP Diagnostics: Improved HTAP replication diagnostics on non-convergence.
• Cayenne Write Observability: Write-phase observability for the in-memory CDC tier.

Notable Bug Fixes​

• Cayenne Utf8View: The Utf8View read schema avoids a hash-join offset overflow.
• Cayenne metastore: cayenne_metastore: turso is honored for partitioned tables and the dataset checkpoint.
• Dual-write detection: Dual-write accelerated tables are detected behind the metadata-enrichment wrapper.
• digest_many collisions: Values are length-prefixed so column boundaries can't collide.
• Turso WAL checkpoint: WAL checkpoints route through the native Turso connection.
• TLS status probe: The status check probes the metrics endpoint over HTTPS when TLS is enabled.
• Search snippet offsets: Character chunk offsets persist so search snippets aren't shifted or garbled.
• Async query chunk offsets: /v1/queries chunk row_offset uses the cumulative offset rather than chunk_index * chunk_size.

Dependency Updates​

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @melks
• @mvanhorn
• @Oxygen56
• @peasee
• @phillipleblanc
• @sgrebnov
• @v1gnesh

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.0 image:

docker pull spiceai/spiceai:2.1.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Make DuckDB schema cast logic more robust by @sgrebnov in #10991
• perf(cayenne): reduce allocation overheads in hot paths by @lukekim in #10950
• improve error message for 'params.spiceai_region' by @Jeadie in #10954
• fix(acceleration): rename WriteMode variants to fix #10960 by @phillipleblanc in #10974
• add Scylla, bigquery and turso to throughput benchmarking by @Jeadie in #11006
• deps(ballista): pull in shuffle-on-object-store correctness fixes (datafusion-ballista PRs #42 + #43) by @phillipleblanc in #10919
• fix(kafka): seek to sidecar offsets via post_rebalance callback on restart by @ewgenius in #11007
• Propagate source comments into schema metadata by @lukekim in #10944
• feat(chbench-driver): better alignment to BenchBase by @sgrebnov in #11012
• fix: handle EXISTS/NOT EXISTS subqueries in federation analyzer by @sgrebnov in #10996
• Enable filter pushdown in spicepod defined UDTFs by @Jeadie in #11004
• Refactor spice dataset configuration command by @Jeadie in #10999
• fix: ensure HashJoinExec partition counts match when join input is statically empty by @phillipleblanc in #11025
• feat(chbench): Improve OLTP throughput and reduce PostgreSQL CDC overhead by @sgrebnov in #11018
• feat(cluster): add distributed query observability metrics by @phillipleblanc in #10990
• fix: delegate truncate in PolyTableProvider to inner write provider by @claudespice in #11036
• Remove default runtime features - enable explicitly in spiced by @phillipleblanc in #11037
• fix: preserve field and schema metadata in Vortex physical schema calculation by @claudespice in #11013
• Expose metadata descriptions via PostgreSQL UDFs by @lukekim in #11032
• fix: route Turso WAL checkpoint through native turso connection (fixes #10657) by @claudespice in #11048
• fix: add missing truncate delegation and guards for wrapper TableProviders by @claudespice in #11014
• Update DataConnector statuses by @krinart in #11052
• remove redundant readonly checks by @Jeadie in #10975
• Fix Unity Catalog connector compatibility with OSS Unity Catalog by @ewgenius in #11026
• refactor(cdc): reduce CDC sub-batch splits for interleaved upsert/delete workloads by @sgrebnov in #11051
• feat(cayenne): allow inline writes with pending deletions (deletes/upserts) by @sgrebnov in #11031
• fix(sql-tool): defer read-only gate to caller's API key role by @phillipleblanc in #11040
• fix: map Gemini Recitation finish reason to ContentFilter by @claudespice in #11046
• feat(cayenne): fast-path CDC deletes by extracting PK values from filters by @sgrebnov in #11049
• fix(cluster): gate scheduler readiness on executor partition loads by @phillipleblanc in #10992
• fix(snowflake): enforce function deny-list in federation pushdown by @claudespice in #11057
• perf(cdc): Last-write-wins dedup in group_into_sub_batches to reduce sub-batch splits by @sgrebnov in #11059
• [Bug] Timing between reconnect and AllocateInitialPartitions leaves connection without flight_sql_client by @Jeadie in #10805
• Define 'trait QueryEngine' to refactor runtime crate by @Jeadie in #11028
• fix(snowflake): apply Spice function deny-list in extracted connector crate by @claudespice in #11071
• perf(cayenne): keep CDC upsert PK keysets resident to avoid per-batch full-table rebuilds by @lukekim in #11074
• fix(postgres-replication): emit recovery log + reduce reconnect-warn volume by @claudespice in #11084
• fix metadata on search indexing by @Jeadie in #11080
• perf(cayenne): scale CDC inline flush caps with memory + storage class by @lukekim in #11087
• feat(cayenne): merge-on-read position deletes for PK upsert tables + memory-pool accounting by @lukekim in #11085
• Support tuple-IN composite PK extraction in Cayenne delete fast-path by @sgrebnov in #11093
• feat(cluster): report per-executor table statistics so distributed JoinSelection can size joins by @phillipleblanc in #11089
• feat(cluster): NDV-aware executor stats so CDC q18 join swap fires by @phillipleblanc in #11098
• Improve HTAP replication diagnostics on non-convergence by @sgrebnov in #11100
• Normalize DataType::Null to Int32 in acceleration schema for duckdb by @krinart in #11062
• Fix debezium schema evolution support by @ewgenius in #11095
• feat(cayenne): incremental write-path executor statistics for distributed join sizing by @phillipleblanc in #11104
• fix(cache): periodic moka maintenance to drain invalidation predicates (#11077) by @phillipleblanc in #11106
• fix: validate Snowflake account identifiers and auth config by @Jeadie in #11024
• fix: trace external mcp server tool calls by @ewgenius in #11058
• Remove unwrap from test code; drop clippy::unwrap_used suppressions by @phillipleblanc in #11108
• Upgrade to DuckDB 1.5.3 + statically link the VSS (HNSW) extension by @sgrebnov in #11107
• Fix fetched_at for HTTP connector by @Jeadie in #11116
• fix(search): propagate LIMIT to base TableScan in VectorScanTableProvider (fixes #8368) by @claudespice in #11124
• feat(runtime): add spicebench feature to register the Cayenne catalog connector by @phillipleblanc in #11122
• fix(cayenne): tombstone inline-checkpointed rows on upsert to prevent duplicate PKs by @sgrebnov in #11129
• Remove possibility of a deadlock in RuntimeStatus by @krinart in #11114
• Fix Windows build: vendored-vss duckdb-rs + adapt to table-providers mongodb API by @phillipleblanc in #11140
• localpod: synchronize child refreshes when parent uses in-memory (arrow) accelerator by @phillipleblanc in #11139
• Add datasets dimension to the query_executions metric by @phillipleblanc in #11138
• fix(spiceai): keep correlated subqueries out of JOIN ON for Spice Cloud federation by @phillipleblanc in #11143
• fix(duckdb): normalize timestamp columns to microsecond precision (fixes #10627) by @claudespice in #11145
• feat(cayenne): sharded parallel Vortex encode with key/time clustering by @lukekim in #11144
• fix(cluster): prevent DoPut write pipeline self-deadlock under ingest backpressure by @phillipleblanc in #11160
• feat(chbench): configurable HTAP concurrency, DuckDB query overrides, and OLTP rate control by @sgrebnov in #11162
• fix(http): preserve non-JSON response rows instead of crashing nested decomposition (fixes #11155) by @claudespice in #11161
• Use declared schema in DynamoDB/MongoDB/Debezium by @krinart in #11066
• fix(cluster): prevent partitioned datasets from staying Refreshing by @phillipleblanc in #11157
• fix(runtime): don't list postgres as a valid accelerator engine when postgres-accel is disabled by @sgrebnov in #11169
• fix(spark): recover stale or broken Spark Connect sessions on failure by @lukekim in #11171
• fix(secrets): don't abort secret lookup precedence walk on a failing store by @phillipleblanc in #11175
• feat(cayenne): bound aggregate write concurrency, conservative defaults, and write/read observability by @lukekim in #11170
• feat(unity_catalog): support Unity Catalog credential vending for Delta Lake tables by @phillipleblanc in #11180
• fix(secrets): keep failed secret stores registered so lookups report the init root cause by @phillipleblanc in #11181
• fix(debezium): sign-extend minimal-width base64 decimals instead of zero-padding by @claudespice in #11184
• fix(deps): update hickory-resolver to 0.26 (evicts hickory-proto 0.25.x) by @phillipleblanc in #11183
• refactor(secrets): derive secret store metadata from a single registry table by @phillipleblanc in #11188
• perf(cayenne): cut CDC replication lag on hot upsert tables by @lukekim in #11191
• feat(cayenne): async inline-fallback (per-tombstone published flag) + 64c/256GB tuning by @lukekim in #11194
• Add HTTP connector mTLS support by @lukekim in #11127
• feat(snowflake): push AT TIME ZONE as CONVERT_TIMEZONE and pin session to UTC by @lukekim in #11190
• feat(cdc): make cdc_max_coalesce_age_ms a real apply-loop linger by @sgrebnov in #11196
• feat(cayenne): delta-write encoding levels (cayenne_delta_encoding, default auto) + make compression_strategy=zstd real by @lukekim in #11199
• Add NSQL context endpoint by @lukekim in #11075
• fix(federation): respect the Spice function deny-list across all SQL connectors; dialect-aware DuckDB pushdown by @claudespice in #11186
• fix: surface unknown/applied cayenne_* runtime.params at startup (fixes #10970) by @claudespice in #11133
• perf(cayenne): plain-fsync ordering tier on the staged-commit hot path by @lukekim in #11198
• fix(kafka): decode JSON payloads to Arrow directly — fixes lossy Decimal128 + removes double-parse (#11192) by @claudespice in #11207
• feat(cayenne): self-tuning accelerator — hardware + schema + closed-loop adaptive (auto/adaptive modes) by @lukekim in #11213
• perf(cayenne): CDC throughput — SF-100 @10K txn/s toward <5s lag + 5K QPH by @lukekim in #11206
• fix(cluster): distribute accelerated tables wrapped by metadata/index providers by @phillipleblanc in #11226
• Improve Cayenne adaptive tuning and schema safety by @lukekim in #11237
• feat: Add cayenne_file_pruning param by @peasee in #11239
• Debezium connector - handle tombstone messages in kafka topic, with schema evolution enabled by @ewgenius in #11197
• feat(cayenne): broadcast small-dimension joins to executors by @phillipleblanc in #11245
• fix: scope request context across the managed query runtime by @phillipleblanc in #11253
• fix: Strip inference columns from table schema on query by @peasee in #11251
• fix(flightsql): don't drop un-pushed FilterExec predicates in distributed pushdown rules by @claudespice in #11256
• feat(postgres): share one replication slot across multiple changes-mode datasets by @phillipleblanc in #11255
• Upgrade to DataFusion v53.1, Arrow v58.3, Vortex v0.74, and dependencies by @lukekim in #11118
• feat(connectors): support file_format: vortex everywhere parquet is supported by @lukekim in #11282
• perf(cayenne): metadata-only publish — drop per-key insert records on upsert commit by @lukekim in #11260
• fix(kafka): harden fetch_latest_message for multi-partition topics by @ewgenius in #11285
• perf(cayenne): bound mem-tier checkpoint churn + O(1) per-scan deletion view by @lukekim in #11249
• fix(udfs): length-prefix digest_many values so column boundaries can't collide (fixes #11272) by @claudespice in #11288
• fix: restore federation deny-list enforcement regressed by the DataFusion 53 upgrade by @claudespice in #11294
• fix(postgres): recover unchanged-TOAST columns from the old tuple; classify walsender contention as transient by @phillipleblanc in #11293
• feat: deepen extended schema inference and wire it into cayenne compaction sharding/sorting by @lukekim in #11284
• perf(cayenne): in-memory CDC tier follow-ups + write-phase observability by @lukekim in #11278
• Support per-dataset CDC tunable overrides by @sgrebnov in #11295
• feat(cayenne): harden adaptive auto-tuner (controller hygiene, mem-tier actuator, delete/burst signals, single opt-in) by @lukekim in #11302
• fix(cayenne): scan inlined-view capture starvation under sustained CDC (analytical QPH) by @lukekim in #11299
• fix(vortex): don't row-evaluate hash-join dynamic filters in the scan by @sgrebnov in #11307
• fix(deps): bump rust-postgres crates (RUSTSEC-2026-0178/0179) by @lukekim in #11313
• fix: strict validation of Postgres CDC params instead of silent defaults (fixes #11274) by @claudespice in #11304
• fix(duckdb): always quote on-refresh sort columns so reserved-word names don't break refresh by @claudespice in #11305
• feat(flightsql): fall back to original connection when endpoint location is unreachable by @melks in #11287
• perf(cayenne): light-encode transient staged CDC deltas by @lukekim in #11311
• feat(acceleration): widening-only schema evolution via on_schema_change by @lukekim in #11261
• deps(vortex): bump pin to spiceai-53 HEAD — intra-file decode split + per-execution kernel cache by @lukekim in #11314
• feat(github): enhance GitHub component validation and error handling by @lukekim in #11259
• feat(cayenne): goal-driven adaptive tuning toward operator SLOs (lag, freshness, query latency, QPH) by @lukekim in #11310
• fix(cayenne): broadcast-join rewrite must bail on ambiguous columns, NULL-equal joins, and residual filters by @claudespice in #11252
• Add Cayenne maintained aggregates by @lukekim in #11235
• perf(cayenne): single-hash composite deletion filter via KeyDeletionIndex::get_batch by @phillipleblanc in #11325
• fix(Vortex): decline only the InList membership conjunct of hash-join dynamic filters by @sgrebnov in #11335
• fix: scope SQL UDF arg inlining to args-table columns (fixes #11273) by @claudespice in #11337
• fix(refresh): restore S3 ETag/Version refresh-skip behind provider wrappers by @phillipleblanc in #11339
• fix(cayenne): ref-count in-flight scans so GC can't delete Vortex files mid-read by @phillipleblanc in #11321
• fix(runtime): retry object-store dataset load when source files are not yet available by @phillipleblanc in #11342
• feat(s3): default to path-style for dotted bucket names on standard AWS by @phillipleblanc in #11347
• fix(runtime): resolve accelerated table through metadata-enrichment wrapper by @phillipleblanc in #11345
• fix: detect dual-write accelerated tables behind the metadata-enrichment wrapper by @claudespice in #11351
• feat(cayenne): incremental seq-prefix bake — shrink the merge-on-read deletion index by @lukekim in #11326
• fix(adbc): prevent Spice-specific UDFs from being pushed down to ADBC sources by @krinart in #11297
• fix: Query Redshift schema details from svv_redshift tables by @peasee in #11362
• perf(cayenne): tune Turso connection PRAGMAs + jitter metastore retries by @lukekim in #11359
• Upgrade to DataFusion 54 by @sgrebnov in #11360
• feat(runtime): dedicated CDC-apply tokio runtime + per-runtime tokio metrics by @lukekim in #11370
• fix(cayenne): spill oversized hash joins via sort-merge to avoid OOM by @lukekim in #11371
• fix(cayenne): honor cayenne_metastore: turso for partitioned tables and the dataset checkpoint by @phillipleblanc in #11365
• perf(cayenne): in-RAM scan parallelism, query admission control, skip no-op deletion encode, sound scan output_ordering by @lukekim in #11332
• fix(catalog): restore DDL after DataFusion 54 broke transparent catalog-provider downcasts by @phillipleblanc in #11375
• feat(flightsql): infer schema via SELECT * LIMIT 1 when GetTables is unimplemented by @melks in #11286
• fix(cayenne): Utf8View read schema avoids hash-join offset overflow by @lukekim in #11379
• feat(cluster): support distributed (Ballista) scans of Iceberg tables by @phillipleblanc in #11378
• feat(optimizer): cost-based left-deep join reordering for Cayenne acceleration by @sgrebnov in #11377
• cli - fix service-account auth in spice cloud * commands by @ewgenius in #11316
• fix: Support external Redshift table schema inference and Hive external type parsing by @peasee in #11399
• feat(llms): native GLM support — opt-in flash-attn + surface reasoning_content by @lukekim in #11400
• feat(s3_vectors): paginate QueryVectors for topK up to 10,000 by @bjchambers in #11405
• fix: surface .env parse errors with line numbers instead of silently skipping by @Oxygen56 in #11306
• fix(status): probe metrics endpoint over https when TLS is enabled by @phillipleblanc in #11393
• fix(mcp): record task_history spans for tool calls proxied through /v1/mcp by @phillipleblanc in #11397
• Properly handle date_trunc in BigQueryDialect by @krinart in #11416
• fix(snowflake): honor column scale in Int64 timestamp arm and cast TIME by @claudespice in #11418
• fix: /v1/queries chunk row_offset uses cumulative offset, not chunk_index * chunk_size (fixes #11271) by @claudespice in #11398
• feat(cayenne): incremental retraction for maintained aggregates + anchor bench by @lukekim in #11389
• feat(cluster): support distributed (Ballista) scans of Iceberg catalog tables by @phillipleblanc in #11419
• Simplify chat/responses models by @Jeadie in #10997
• feat(llms): distributed tensor-parallel GLM inference via mistral.rs ring backend by @lukekim in #11406
• Optimize Flight streaming for large result sets by @lukekim in #11420
• fix(deps): evict rustls 0.21 / rustls-webpki 0.101.7 (GHSA-82j2-j2ch-gfr8) by @phillipleblanc in #11428
• feat(cayenne): in-memory CDC intra-apply sharding (PK-hash shards) by @lukekim in #11421
• fix(cayenne): shard CDC upserts with pending deletions so the N>1 slot-ack advances by @lukekim in #11445
• fix(cluster): keep built-in avg over Spark avg (distributed aggregate state schema mismatch) by @phillipleblanc in #11434
• feat(views): support params.file_format for embedding chunking by @Jeadie in #11424
• fix: offload blocking sync calls off the primary async runtime by @phillipleblanc in #11435
• fix(udfs): rebind dot_product alias to Spice's inner_product on DataFusion 54 by @lukekim in #11443
• feat(secrets): add full-fidelity reference iteration and a resolution-status API by @phillipleblanc in #11195
• fix: Deny unsupported array functions for Postgres pushdown by @peasee in #11450
• fix(cli): spice query honors --http-endpoint instead of failing on a Flight connect by @phillipleblanc in #11452
• fix(cayenne): coordinate query-pool + in-memory CDC tier budgets to prevent adaptive OOM by @lukekim in #11449
• Surface mTLS config in Kafka data connector by @v1gnesh in #11372
• feat(secrets): check and report secret references at startup by @phillipleblanc in #11457
• Default cayenne_force_view_types to false by @sgrebnov in #11459
• fix: resolve table-reference qualification in results-cache invalidation (fixes #11266) by @claudespice in #11460
• feat(cayenne): metadata aggregate pushdown — fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics by @bjchambers in #11414
• fix(search): restore numeric trunc and fix SortPreservingMergeExec planning error (DF54) by @Jeadie in #11415
• fix(cluster): route Ballista shuffle/temp to the data PVC by @phillipleblanc in #11454
• fix(search): default Elasticsearch kNN candidate pool to 1000 instead of 10 (fixes #11264) by @claudespice in #11467
• feat(acceleration): add on_schema_change drop_and_recreate policy by @lukekim in #11462
• feat(cayenne): predicate-aware maintained aggregates serve filtered analytical queries from the CDC delta by @lukekim in #11458
• feat(cluster): shared Ballista job state with scheduler failover by @phillipleblanc in #11436
• feat(cayenne): extend HLL NDV sketching to string and date columns by @bjchambers in #11468
• fix(search): persist character chunk offsets so search snippets aren't shifted/garbled (fixes #11269) by @claudespice in #11479
• feat(cayenne): storage-aware adaptive CDC tuning — calibration probe, IMDS, I/O-cliff fast path, infeasible-SLO feedback by @lukekim in #11463
• fix(cayenne): LIMIT N under-delivers on key-deletion tables by @lukekim in #11490
• fix(cayenne): live/tier-accurate join build-side stats (merge-on-read deletes + never-shrink NDV) by @lukekim in #11496
• perf(cayenne): kernel-space I/O hygiene — compaction fadvise + staged-commit barrier reduction by @lukekim in #11495
• feat(cayenne): feed maintained-aggregate IVM from the staged-disk CDC path by @lukekim in #11491
• feat(cayenne): global adaptive-tuning SLOs with per-dataset overrides; QPH global-only by @lukekim in #11497
• Update search snapshots by @sgrebnov in #11473
• fix: Support reading column types longer than 128 chars in Redshift by @peasee in #11500
• fix(cluster): distributed (Ballista) query-execution config + scheduled SF10 bench by @phillipleblanc in #11478
• fix(http): retry transient response-body read failures; de-flake backoff test by @claudespice in #11482
• Re-land orphaned deletion-vector cleanup during retention deletes by @lukekim in #11501
• fix(cayenne): re-upsert over a pending delete tombstone records an insert-record (overwrite resurrection) by @bjchambers in #11469
• fix: Flight DoPut silently dropped client batches on early sink completion by @claudespice in #11507
• Upgrade OpenTelemetry to 0.32 and reqwest to 0.13 by @phillipleblanc in #11506
• fix(queries): run async /v1/queries jobs under the submitting request context by @phillipleblanc in #11505
• fix(cayenne): restore append-only current-snapshot compaction by @Jeadie in #11439
• perf(cayenne): orphaned deletion-vector cleanup off the write path, behind a knob by @bjchambers in #11517
• fix(datafusion): accurate projected scan byte size so hash joins build the smaller side by @sgrebnov in #11503
• fix(cayenne): user-visible DELETE WHERE pk IN (...) reports the real row count by @lukekim in #11514
• fix(cayenne): seed persisted num_rows for hash-join sizing by @sgrebnov in #11515
• fix(cayenne): correctness & memory_limit fixes from perf audit (2 P0, 3 P1) by @lukekim in #11516
• feat(cayenne): wire orphaned-DV cleanup knob to spicepod params + doc sync by @bjchambers in #11523
• Fix s3 vectors API by @krinart in #11536
• fix: Placeholder table initialization lock swap by @peasee in #11540
• chore(cluster): bump ballista pin for the null-aware anti-join fix by @phillipleblanc in #11544
• fix(runtime-tools): fix memory table identifier validation rejecting valid names by @Jeadie in #11546
• Bump datafusion to include spiceai/datafusion#181 by @Jeadie in #11563
• Update deny.toml by @krinart in #11571
• Bump datafusion (spiceai/datafusion#182) and datafusion-table-providers (#27): fix q16 CollectLeft planning error and SQLite q6 wrong revenue by @Jeadie in #11598

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0...v2.1.0

Spice v2.0.1 is now available! 🛠️

Spice v2.0.1 is a patch release focused on reliability and performance. It speeds up Apache Iceberg reads and fixes bugs across AWS S3 and object-store datasets, data acceleration, distributed query, and authenticated access.

What's New in v2.0.1​

Faster Iceberg Reads with Parallel File Scanning​

The Apache Iceberg reader now scans data files in parallel (#11331), improving read throughput and latency for Iceberg tables that span many files.

AWS S3 & Object-Store Reliability​

Three fixes improve S3 and object-store dataset behavior:

• Refresh-skip restored (#11339): ETag/Version-based refresh-skip works reliably again, so unchanged S3 objects are no longer re-downloaded on every refresh.
• Retry when source files are not yet available (#11342): an object-store dataset whose source files are not present at startup now retries and becomes ready once the data appears, instead of failing permanently.
• Path-style addressing for dotted bucket names (#11347): on standard AWS, buckets whose names contain dots now default to path-style addressing, avoiding TLS wildcard certificate errors under virtual-hosted-style HTTPS.

Data Acceleration & Distributed Query Fixes​

Two fixes ensure accelerated datasets behave correctly in more configurations:

• Acceleration endpoints (#11345): /v1/datasets/{name}/acceleration/refresh (and the related update-refresh-sql, partition-filters, and snapshots endpoints) now work for all accelerated datasets, fixing cases where some incorrectly reported Table is not accelerated.
• Distributed clusters (#11226): the distributed query coordinator now serves accelerated data from executors for all accelerated datasets, instead of falling back to reading from the source for some.

Authenticated Query Fixes​

With authentication enabled, queries now consistently run as the requesting user (#11253), so per-user behavior such as results caching is correctly scoped to each user.

Contributors​

• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.1 image:

docker pull spiceai/spiceai:2.0.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(ci): key testoperator & validator artifacts by checked-out commit by @sgrebnov in #11281
• chore(deps): fix cargo-deny advisory failures on release/2.0 by @phillipleblanc in #11333
• chore(deps): bump iceberg-rust to parallel file scanning fork (release/2.0) by @phillipleblanc in #11331
• fix(refresh): restore S3 ETag/Version refresh-skip behind provider wrappers by @phillipleblanc in #11339
• fix(runtime): retry object-store dataset load when source files are not yet available by @phillipleblanc in #11342
• feat(s3): default to path-style for dotted bucket names on standard AWS by @phillipleblanc in #11347
• fix(runtime): resolve accelerated table through metadata-enrichment wrapper by @phillipleblanc in #11345
• fix(cluster): distribute accelerated tables wrapped by metadata/index providers by @phillipleblanc in #11226
• fix: scope request context across the managed query runtime by @phillipleblanc in #11253

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0...v2.0.1

Announcing the release of Spice v1.11.0-stable! ⚡

In Spice v1.11.0, Spice Cayenne reaches Beta status with acceleration snapshots, Key-based deletion vectors, and Amazon S3 Express One Zone support. DataFusion has been upgraded to v51 along with Arrow v57.2, and iceberg-rust v0.8.0. v1.11 adds several DynamoDB & DynamoDB Streams improvements such as JSON nesting, and adds significant improvements to Distributed Query with active-active schedulers and mTLS for enterprise-grade high-availability and secure cluster communication.

This release also adds new SMB, NFS, and ScyllaDB Data Connectors (Alpha), Prepared Statements with full SDK support (gospice, spice-rs, spice-dotnet, spice-java, spice.js, and spicepy), Google LLM Support for expanded AI inference capabilities, and significant improvements to caching, observability, and Hash Indexing for Arrow Acceleration.

What's New in v1.11.0​

Spice Cayenne Accelerator Reaches Beta​

Spice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous performance and stability improvements.

Key Enhancements:

• Key-based Deletion Vectors: Improved deletion vector support using key-based lookups for more efficient data management and faster delete operations. Key-based deletion vectors are more memory-efficient than positional vectors for sparse deletions.
• S3 Express One Zone Support: Store Cayenne data files in S3 Express One Zone for single-digit millisecond latency, ideal for latency-sensitive query workloads that require persistence.

Improved Reliability:

• Resolved FuturesUnordered reentrant drop crashes
• Fixed memory growth issues related to Vortex metrics allocation
• Metadata catalog now properly respects cayenne_file_path location
• Added warnings for unparseable configuration values

For more details, refer to the Cayenne Documentation.

DataFusion v51 Upgrade​

Apache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.

Performance Improvements:

• Faster CASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patterns
• Better Defaults for Remote Parquet Reads: DataFusion now fetches the last 512KB of Parquet files by default, typically avoiding 2 I/O requests per file
• Faster Parquet Metadata Parsing: Leverages Arrow 57's new thrift metadata parser for up to 4x faster metadata parsing

New SQL Features:

• SQL Pipe Operators: Support for |> syntax for inline transforms
• DESCRIBE <query>: Returns the schema of any query without executing it
• Named Arguments in SQL Functions: PostgreSQL-style param => value syntax for scalar, aggregate, and window functions
• Decimal32/Decimal64 Support: New Arrow types supported including aggregations like SUM, AVG, and MIN/MAX

Example pipe operator:

SELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;

Improved Observability:

• Improved EXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdowns

Arrow 57.2 Upgrade​

Apache Arrow has been upgraded to v57.2, bringing major performance improvements and new capabilities.

Key Features:

• 4x Faster Parquet Metadata Parsing: A rewritten thrift metadata parser delivers up to 4x faster metadata parsing, especially beneficial for low-latency use cases and files with large amounts of metadata
• Parquet Variant Support: Experimental support for reading and writing the new Parquet Variant type for semi-structured data, including shredded variant values
• Parquet Geometry Support: Read and write support for Parquet Geometry types (GEOMETRY and GEOGRAPHY) with GeospatialStatistics
• New arrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution support

DynamoDB Connector Enhancements​

• Added JSON nesting for DynamoDB Streams
• Improved batch deletion handling

Distributed Query Improvements​

High Availability Clusters: Spice now supports running multiple active schedulers in an active/active configuration for production deployments. This eliminates the scheduler as a single point of failure and enables graceful handling of node failures.

• Multiple schedulers run simultaneously, each capable of accepting queries
• Schedulers coordinate via a shared S3-compatible object store
• Executors discover all schedulers automatically
• A load balancer distributes client queries across schedulers

Example HA configuration:

runtime:
  scheduler:
    state_location: s3://my-bucket/spice-cluster
    params:
      region: us-east-1

mTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.

Credential Propagation: S3, ABFS, and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.

Improved Resilience:

• Exponential backoff for scheduler disconnection recovery
• Increased gRPC message size limit from 16MB to 100MB for large query plans
• HTTP health endpoint for cluster executors
• Automatic executor role inference when --scheduler-address is provided

For more details, refer to the Distributed Query Documentation.

iceberg-rust v0.8.0 Upgrade​

Spice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.

Key Features:

• V3 Metadata Support: Full support for Iceberg V3 table metadata format
• INSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tables
• Improved Delete File Handling: Better support for position and equality delete files, including shared delete file loading and caching
• SQL Catalog Updates: Implement update_table and register_table for SQL catalog
• S3 Tables Catalog: Implement update_table for S3 Tables catalog
• Enhanced Arrow Integration: Convert Arrow schema to Iceberg schema with auto-assigned field IDs, _file column support, and Date32 type support

Acceleration Snapshots​

Acceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.

Key Features:

• Flexible Triggers: Configure when snapshots are created based on time intervals or stream batch counts
• Automatic Compaction: Reduce storage overhead by compacting older snapshots (DuckDB only)
• Bootstrap Integration: Snapshots can reset cache expiry on load for seamless recovery (DuckDB with Caching refresh mode)
• Smart Creation Policies: Only create snapshots when data has actually changed

Example configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled
      snapshots_trigger: time_interval
      snapshots_trigger_threshold: 1h
      snapshots_creation_policy: on_changed

Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically.

CLI Commands:

# List all snapshots for a dataset
spice acceleration snapshots taxi_trips

# Get details of a specific snapshot
spice acceleration snapshot taxi_trips 3

# Set the current snapshot for rollback (requires runtime restart)
spice acceleration set-snapshot taxi_trips 2

HTTP API Endpoints:

For more details, refer to the Acceleration Snapshots Documentation.

Caching Acceleration Mode Improvements​

The Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.

Performance Optimizations:

• Non-blocking Cache Writes: Cache misses no longer block query responses. Data is written to the cache asynchronously after the query returns, reducing query latency for cache miss scenarios.
• Batch Cache Writes: Multiple cache entries are now written in batches rather than individually, significantly improving write throughput for high-volume cache operations.

Reliability Fixes:

• Correct SWR Refresh Behavior: The stale-while-revalidate (SWR) pattern now correctly refreshes only the specific entries that were accessed instead of refreshing all stale rows in the dataset. This prevents unnecessary source queries and reduces load on upstream data sources.
• Deduplicated Refresh Requests: Fixed an issue where JSON array responses could trigger multiple redundant refresh operations. Refresh requests are now properly deduplicated.
• Fixed Cache Hit Detection: Resolved an issue where queries that didn't include fetched_at in their projection would always result in cache misses, even when cached data was available.
• Unfiltered Query Optimization: SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.

For more details, refer to the Caching Acceleration Mode Documentation.

Prepared Statements​

Improved Query Performance and Security: Spice now supports prepared statements, enabling parameterized queries that improve both performance through query plan caching and security by preventing SQL injection attacks.

Key Features:

• Query Plan Caching: Prepared statements cache query plans, reducing planning overhead for repeated queries
• SQL Injection Prevention: Parameters are safely bound, preventing SQL injection vulnerabilities
• Arrow Flight SQL Support: Full prepared statement support via Arrow Flight SQL protocol

SDK Support:

Example (Go):

import "github.com/spiceai/gospice/v8"

client, _ := spice.NewClient()
defer client.Close()

// Parameterized query with typed parameters
results, _ := client.SqlWithParams(ctx,
    "SELECT * FROM products WHERE price > $1 AND category = $2",
    spice.Float64Param(10.0),
    spice.StringParam("electronics"),
)

Example (Java):

import ai.spice.SpiceClient;
import ai.spice.Param;
import org.apache.arrow.adbc.core.ArrowReader;

try (SpiceClient client = new SpiceClient()) {
    // With automatic type inference
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM products WHERE price > $1 AND category = $2",
        10.0, "electronics");

    // With explicit typed parameters
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM products WHERE price > $1 AND category = $2",
        Param.float64(10.0),
        Param.string("electronics"));
}

For more details, refer to the Parameterized Queries Documentation.

Spice Java SDK v0.5.0​

Parameterized Query Support for Java: The Spice Java SDK v0.5.0 introduces parameterized queries using ADBC (Arrow Database Connectivity), providing a safer and more efficient way to execute queries with dynamic parameters.

Key Features:

• SQL Injection Prevention: Parameters are safely bound, preventing SQL injection vulnerabilities
• Automatic Type Inference: Java types are automatically mapped to Arrow types (e.g., double → Float64, String → Utf8)
• Explicit Type Control: Use the new Param class with typed factory methods (Param.int64(), Param.string(), Param.decimal128(), etc.) for precise control over Arrow types
• Updated Dependencies: Apache Arrow Flight SQL upgraded to 18.3.0, plus new ADBC driver support

Example:

import ai.spice.SpiceClient;
import ai.spice.Param;

try (SpiceClient client = new SpiceClient()) {
    // With automatic type inference
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM taxi_trips WHERE trip_distance > $1 LIMIT 10",
        5.0);

    // With explicit typed parameters for precise control
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM orders WHERE order_id = $1 AND amount >= $2",
        Param.int64(12345),
        Param.decimal128(new BigDecimal("99.99"), 10, 2));
}

Maven:

<dependency>
  <groupId>ai.spice</groupId>
  <artifactId>spiceai</artifactId>
  <version>0.5.0</version>
</dependency>

For more details, refer to the Spice Java SDK Repository.

Google LLM Support​

Expanded AI Provider Support: Spice now supports Google embedding and chat models via the Google AI provider, expanding the available LLM options for AI inference workloads alongside existing providers like OpenAI, Anthropic, and AWS Bedrock.

Key Features:

• Google Chat Models: Access Google's Gemini models for chat completions
• Google Embeddings: Generate embeddings using Google's text embedding models
• Unified API: Use the same OpenAI-compatible API endpoints for all LLM providers

Example spicepod.yaml configuration:

models:
  - from: google:gemini-2.0-flash
    name: gemini
    params:
      google_api_key: ${secrets:GOOGLE_API_KEY}

embeddings:
  - from: google:text-embedding-004
    name: google_embeddings
    params:
      google_api_key: ${secrets:GOOGLE_API_KEY}

For more details, refer to the Google LLM Documentation (see docs PR #1286).

URL Tables​

Query data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.

Supported Patterns:

• Single files: SELECT * FROM 's3://bucket/data.parquet'
• Directories/prefixes: SELECT * FROM 's3://bucket/data/'
• Glob patterns: SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'

Key Features:

• Automatic file format detection (Parquet, CSV, JSON, etc.)
• Hive-style partition inference with filter pushdown
• Schema inference from files
• Works with both SQL and DataFrame APIs

Example with hive partitioning:

-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'

Enable via spicepod.yml:

runtime:
  params:
    url_tables: enabled

Cluster Mode Async Query APIs (experimental)​

New asynchronous query APIs for long-running queries in cluster mode:

• /v1/queries endpoint: Submit queries and retrieve results asynchronously

OpenTelemetry Improvements​

Unified Telemetry Endpoint: OTel metrics ingestion has been consolidated to the Flight port (50051), simplifying deployment by removing the separate OTel port (50052). The push-based metrics exporter continues to support integration with OpenTelemetry collectors.

Note: This is a breaking change. Update your configurations if you were using the dedicated OTel port 50052. Internal cluster communication now uses port 50052 exclusively.

Observability Improvements​

Enhanced Dashboards: Updated Grafana and Datadog example dashboards with:

• Snapshot monitoring widgets
• Improved accelerated datasets section
• Renamed ingestion lag charts for clarity

Additional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.

For more details, refer to the Monitoring Documentation.

Hash Indexing for Arrow Acceleration (experimental)​

Arrow-based accelerations now support hash indexing for faster point lookups on equality predicates. Hash indexes provide O(1) average-case lookup performance for columns with high cardinality.

Features:

• Primary key hash index support
• Secondary index support for non-primary key columns
• Composite key support with proper null value handling

Example configuration:

datasets:
  - from: postgres:users
    name: users
    acceleration:
      enabled: true
      engine: arrow
      primary_key: user_id
      indexes:
        '(tenant_id, user_id)': unique  # Composite hash index

For more details, refer to the Hash Index Documentation.

SMB and NFS Data Connectors​

Network-Attached Storage Connectors: New data connectors for SMB (Server Message Block) and NFS (Network File System) protocols enable direct federated queries against network-attached storage without requiring data movement to cloud object stores.

Key Features:

• SMB Protocol Support: Connect to Windows file shares and Samba servers with authentication support
• NFS Protocol Support: Connect to Unix/Linux NFS exports for direct data access
• Federated Queries: Query Parquet, CSV, JSON, and other file formats directly from network storage with full SQL support
• Acceleration Support: Accelerate data from SMB/NFS sources using DuckDB, Spice Cayenne, or other accelerators

Example spicepod.yaml configuration:

datasets:
  # SMB share
  - from: smb://fileserver/share/data.parquet
    name: smb_data
    params:
      smb_username: ${secrets:SMB_USER}
      smb_password: ${secrets:SMB_PASS}

  # NFS export
  - from: nfs://nfsserver/export/data.parquet
    name: nfs_data

For more details, refer to the Data Connectors Documentation.

ScyllaDB Data Connector​

A new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.

Example configuration:

datasets:
  - from: scylladb:my_keyspace.my_table
    name: scylla_data
    acceleration:
      enabled: true
      engine: duckdb

For more details, refer to the ScyllaDB Data Connector Documentation.

Flight SQL TLS Connection Fixes​

TLS Connection Support: Fixed TLS connection issues when using grpc+tls:// scheme with Flight SQL endpoints. Added support for custom CA certificate files via the new flightsql_tls_ca_certificate_file parameter.

Developer Experience Improvements​

• Turso v0.3.2 Upgrade: Upgraded Turso accelerator for improved performance and reliability
• Rust 1.91 Upgrade: Updated to Rust 1.91 for latest language features and performance improvements
• Spice Cloud CLI: Added spice cloud CLI commands for cloud deployment management
• Improved Spicepod Schema: Improved JSON schema generation for better IDE support and validation
• Acceleration Snapshots: Added configurable snapshots_create_interval for periodic acceleration snapshots independent of refresh cycles
• Tiered Caching with Localpod: The Localpod connector now supports caching refresh mode, enabling multi-layer acceleration where a persistent cache feeds a fast in-memory cache
• GitHub Data Connector: Added workflows and workflow runs support for GitHub repositories
• NDJSON/LDJSON Support: Added support for Newline Delimited JSON and Line Delimited JSON file formats

Additional Improvements & Bug Fixes​

• Model Listing: New functionality to list available models across multiple AI providers
• DuckDB Partitioned Tables: Primary key constraints now supported in partitioned DuckDB table mode
• Post-refresh Sorting: New on_refresh_sort_columns parameter for DuckDB enables data ordering after writes
• Improved Install Scripts: Removed jq dependency and improved cross-platform compatibility
• Better Error Messages: Improved error messaging for bucket UDF arguments and deprecated OpenAI parameters
• Reliability: Fixed DynamoDB IAM role authentication with new dynamodb_auth: iam_role parameter
• Reliability: Fixed cluster executors to use scheduler's temp_directory parameter for shuffle files
• Reliability: Initialize secrets before object stores in cluster executor mode
• Reliability: Added page-level retry with backoff for transient GitHub GraphQL errors
• Performance: Improved statistics for rewritten DistributeFileScanOptimizer plans
• Developer Experience: Added max_message_size configuration for Flight service

Contributors​

• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @mach-kernel
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

OTel Ingestion Port Change​

OTel ingestion has been moved to the Flight port (50051), removing the separate OTel port 50052. Port 50052 is now used exclusively for internal cluster communication. Update your configurations if you were using the dedicated OTel port.

Distributed Query Cluster Mode Requires mTLS​

Distributed query cluster mode now requires mTLS for secure communication between cluster nodes. This is a security enhancement to prevent unauthorized nodes from joining the cluster and accessing secrets.

Migration Steps:

1. Generate certificates using spice cluster tls init and spice cluster tls add
2. Update scheduler and executor startup commands with --node-mtls-* arguments
3. For development/testing, use --allow-insecure-connections to opt out of mTLS

Renamed CLI Arguments:

Removed CLI Arguments:

• --cluster-api-key: Replaced by mTLS authentication

Cookbook Updates​

New ScyllaDB Data Connector Recipe: New recipe demonstrating how to use the ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.

New SMB Data Connector Recipe: New recipe demonstrating how to use the SMB Data Connector. See SMB Data Connector Recipe for details.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.0 image:

docker pull spiceai/spiceai:1.11.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

Dependencies​

• DataFusion: Upgraded to v51 (release notes)
• Arrow: Upgraded to v57.2 (release notes)
• iceberg-rust: Upgraded to v0.8.0 (release notes)

What's Changed​

Changelog​

• OTel exporter for push metrics by @lukekim in #8442
• fix: Update benchmark snapshots by @app/github-actions in #8448
• Add TPCH append tests to scheduled dispatch workflow by @sgrebnov in #8451
• Add snapshot creation logging by @krinart in #8469
• Fix PeriodicReader panic by @krinart in #8471
• Benchmarks: increase readiness timeout for turso acceleration (TPC-H) by @sgrebnov in #8470
• fix: Pin CUDA build actions to commits by @peasee in #8477
• Add Criterion benchmarking to chunking crate. by @Jeadie in #8431
• DuckDB agg pushdown: gate behind accelerator parameter by @mach-kernel in #8474
• Rename aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485
• Handle throttling exception for DynamoDB streams by @phillipleblanc in #8492
• docs: Add release notes by @peasee in #8478
• Update spicepod.schema.json by @app/github-actions in #8496
• Move 'test_projection_pushdown' to runtime-datafusion by @Jeadie in #8490
• Fix OTEL metrics HTTP exporter client setup by @phillipleblanc in #8489
• Update endgame to include new caching accelerator cookbook by @phillipleblanc in #8487
• DynamoDB tests and fixes by @lukekim in #8491
• Align make lint-rust-fix with make lint-rust by @Jeadie in #8499
• fix: Remove unused Cayenne parameters by @peasee in #8500
• Force task history captured_plan outputs to be captured even if they would be filtered out otherwise by @phillipleblanc in #8501
• release: post-release updates by @peasee in #8503
• CI: Fix E2E models dispatch by @mach-kernel in #8505
• Use an isolated Tokio runtime for refresh tasks that is separate from the main query API by @phillipleblanc in #8504
• Update openapi.json by @app/github-actions in #8512
• Update dependencies by @phillipleblanc in #8513
• fix: Avoid double hashing cache key by @peasee in #8511
• fix: Eagerly drop cached records for results larger than max by @peasee in #8516
• feat: Support vortex zstd compressor by @peasee in #8515
• warning if column is defined in spicepod but is non-existant by @Jeadie in #8498
• Return summarized spicepods from /v1/spicepods by @phillipleblanc in #8404
• Fix for idle DynamoDB Stream by @krinart in #8506
• Use Datafusion::Plan over Datafusion::Internal for user-facing search errors. by @Jeadie in #8484
• DDB Streams Integration Test + Memory Acceleration + Improved Warning by @krinart in #8520
• Upgrade to gospice v8 by @lukekim in #8524
• Vortex file format for object store by @Jeadie in #8525
• docs: Add missing cookbooks to endgame, focus area section by @peasee in #8527
• ListingTableConnector: Drop partition columns that reoccur in file schema by @mach-kernel in #8519
• fix(cluster): initialize secrets before object stores in executor by @sgrebnov in #8532
• Use separate Tokio runtime for SWR refreshes by @phillipleblanc in #8530
• fix: SQL Results Cache SWR triggers incorrect cache miss metric by @phillipleblanc in #8529
• feat: testoperator: OTLP streaming metrics / connect to existing instance / infinite mode / query timeout by @phillipleblanc in #8537
• Update openapi.json by @app/github-actions in #8518
• Add better attributes for search testing by @Jeadie in #8531
• fix: Improve Cayenne errors, ID selection for table/partition creation by @peasee in #8523
• Percent-encode Kubernetes secret name path segment by @phillipleblanc in #8522
• Initial parameterisation of Search integration tests by @Jeadie in #8066
• fix: Add warning when multiple partitions are defined for the same table by @peasee in #8540
• Add DuckDB file-mode support for search test parameterization. by @Jeadie in #8541
• fix: Add recursion depth limits to prevent DoS via deeply nested data (DynamoDB + S3 Vectors) by @phillipleblanc in #8544
• Remove the clippy::too_many_lines lint by @phillipleblanc in #8549
• feat: Add spice cluster tls commands by @phillipleblanc in #8550
• Move OTel ingestion to Flight port, remove separate OTel port 50052 by @phillipleblanc in #8551
• Move verbose tool init messages to trace by @phillipleblanc in #8552
• Add SANs to spice cluster tls add certificates by @phillipleblanc in #8554
• Add cpu, gpu, and memory to telemetry by @lukekim in #8483
• feat: Add workflows and workflow runs to GitHub Data Connector by @peasee in #8548
• Add S3Vectors option for paramterised search tests by @Jeadie in #8555
• Distributed query: TLS + API key by @mach-kernel in #8468
• Refactor RRF SQL to use LogicalPlanBuilder by @Jeadie in #7968
• Require mTLS for distributed query cluster mode by @phillipleblanc in #8580
• Fix stats for rewritten DistributeFileScanOptimizer plans by @mach-kernel in #8581
• Bump actions/cache from 4.3.0 to 5.0.1 by @app/dependabot in #8573
• Show user-friendly error on empty DDB table by @krinart in #8586
• Add (Deprecated) labels to deprecated spice sql params by @krinart in #8588
• Fix kafka warning when security.protocol is set to PLAINTEXT by @krinart in #8587
• Upgrade dependencies by @phillipleblanc in #8593
• Release notes for v1.10.1 by @Jeadie in #8568
• Run search benchmarks twice a week. by @Jeadie in #8592
• Add cayenne data accelerator by @Jeadie in #8553
• SQL allowlist for tools: sql, list_datasets, search, table_schema by @Jeadie in #8449
• use rstest for llms integration tests by @Jeadie in #8566
• Post v1.10.1 housekeeping by @Jeadie in #8600
• Add checklist for SDK publication in end_game.md by @Jeadie in #8602
• Fix IMAGE_TAG assignment for Docker compatibility by @Jeadie in #8599
• Modify cluster arguments to spiced for UX review by @phillipleblanc in #8603
• Update QA analytics with new release data by @Jeadie in #8601
• Remove 'tract-core' dependency. by @Jeadie in #8605
• Google embedding and chat models. by @Jeadie in #8423
• Fix test_github_workflows integration test by @sgrebnov in #8607
• Update openapi.json by @app/github-actions in #8604
• ci: Upload artifacts to MinIO eagerly after each build step by @phillipleblanc in #8615
• fix: SQLite accelerator decimal/date handling by @phillipleblanc in #8606
• Configure mTLS for executor-to-executor gRPC connections by @sgrebnov in #8617
• feat: Enable localpod with caching mode accelerator for tiered caching by @phillipleblanc in #8621
• Add Cayenne S3 Express One Zone support for data files by @lukekim in #8502
• Add snapshot interval for acceleration snapshots by @phillipleblanc in #8627
• Add dataset_load_parallelism parameter to spicepod.yml by @peasee in #8630
• Json Nesting for DynamoDB by @krinart in #8623
• Restore deprecated open-telemetry flag in spiced by @phillipleblanc in #8629
• Implement batching for Kafka/Debezium + null Decimal handling by @krinart in #8622
• fix: Status field in /v1/datasets & /v1/models by @lukekim in #8633
• Add Spice test operator improvements by @sgrebnov in #8625
• Add v1.10.2 release notes by @sgrebnov in #8640
• Align object-store vortex with runtime feature flagging by @Jeadie in #8620
• Extended LLM & search tests on cron by @Jeadie in #8624
• Test-operator: emit main metrics as part of load tests by @sgrebnov in #8639
• Build a local docker image from an existing Spice binary by @phillipleblanc in #8619
• fix: Use runtime-rate-control for GitHub Data Connector by @peasee in #8638
• Upgrade dependencies by @phillipleblanc in #8655
• Bump headers-accept from 0.1.4 to 0.3.0 by @app/dependabot in #8644
• Update openapi.json by @app/github-actions in #8637
• Update SECURITY.md - Include v1.10.2 by @sgrebnov in #8661
• Serialize acceleration snapshots with refresh writes by @phillipleblanc in #8652
• Update AI Installation test to use minilm_l6_v2 by @sgrebnov in #8659
• Fixes for search integration test CI by @Jeadie in #8656
• fix: Use a GitHub rate controller per auth context by @peasee in #8662
• fix: Update Search integration test snapshots by @app/github-actions in #8654
• Make E2E Test Release Installation (AI, Local HF model) test more robust by @sgrebnov in #8666
• Fix issue with location predicate for custom S3 endpoints + regression integration test by @phillipleblanc in #8668
• fix: Validate schema match before projection pushdown in UnionProjectionPushdownOptimizer by @phillipleblanc in #8669
• Proper batch commit for kafka/debezium by @krinart in #8671
• Improve spicepod json schema generation by @ewgenius in #8547
• Start the anonymous telemetry exporter asynchronously by @phillipleblanc in #8679
• fix: Move enforce-pulls to hosted runner by @phillipleblanc in #8686
• Update QA analytics with 1.10.2 release data by @sgrebnov in #8667
• fix: Azure does not support suffix range requests by @phillipleblanc in #8685
• Remove spicepod-validator cargo build from build-dev target by @Jeadie in #8684
• fix: Update test snapshots by @app/github-actions in #8680
• fix: Update Search integration test snapshots by @app/github-actions in #8681
• SMB and NFS Data Connectors by @lukekim in #8674
• Upgrade to openai-async v0.32 by @lukekim in #8635
• v1.10.3 release notes by @phillipleblanc in #8693
• Upgrade dependencies by @phillipleblanc in #8704
• fix: Support NDJSON and LDJSON by @lukekim in #8649
• move OpenAI overrides to non-prefixed by @Jeadie in #8678
• Update Google LLM param: google_dimensions -> dimensions by @Jeadie in #8677
• Make cluster mTLS optional with insecure flag by @phillipleblanc in #8703
• Revert "fix: Move enforce-pulls to hosted runner (#8686)" by @phillipleblanc in #8709
• Initial 'testoperator run text-to-sql' by @Jeadie in #8618
• Add support for abfss by @krinart in #8706
• Add testoperator TPCH dispatch for ABFS with hierarchical namespace disabled + versioning enabled by @phillipleblanc in #8711
• Update openapi.json by @app/github-actions in #8692
• cluster: validate --role argument by @phillipleblanc in #8717
• Upgrade to Turso v0.3.2 by @lukekim in #8716
• Rename --insecure to --allow-insecure-connections to be consistent with existing naming by @lukekim in #8720
• Remove 'testoperator run http-consistency/http-overhead' by @Jeadie in #8708
• refactor: Remove cluster feature flag by @phillipleblanc in #8718
• Docs: Distributed query ADR by @mach-kernel in #8608
• Use model.datasets to allowlist on tools by @Jeadie in #8714
• cluster: quality of life improvements to starting cluster mode locally by @phillipleblanc in #8719
• Docs: Ballista extension ADR by @mach-kernel in #8616
• Improve deprecation messages when going from prefixed -> non-prefixed. by @Jeadie in #8724
• Remove tools from auto-defaults by @Jeadie in #8725
• Make distinct providers for vector spilling, vector partitioning. by @Jeadie in #8546
• cluster: default scheduler address port by @phillipleblanc in #8728
• Add Makefile targets for testoperator by @Jeadie in #8729
• text-to-sql dispatch in testoperator by @Jeadie in #8705
• DR-006: High Availability Distributed Query with Stateless Schedulers by @lukekim in #8721
• DR-007: mTLS for Distributed Query Cluster Communication by @lukekim in #8722
• SMB and NFS improvements by @lukekim in #8710
• fix: Cluster executors use scheduler's temp_directory for shuffle files by @phillipleblanc in #8733
• use 'max_message_size' in flight service too by @Jeadie in #8730
• Add page-level retry for transient GraphQL errors with backoff and increase GitHub rate limit buffer up to 100 by @ewgenius in #8726
• Make testoperator Dockerfile; CI to build docker image to ghcr.io. by @Jeadie in #8732
• cluster: UnionProjectionPushdownOptimizer: Add projection pushdown diagnostics for union children by @phillipleblanc in #8734
• Fix column projection order mismatch with location metadata columns by @phillipleblanc in #8738
• Fixes for testoperator. by @Jeadie in #8737
• Improve Cayenne Deletion Vectors with KeyBased support by @lukekim in #8713
• Fix testoperator_dispatch.yaml by @Jeadie in #8740
• Add spice cloud CLI commands by @lukekim in #8528
• Add FTP, NFS, & SMB TPCH SF1 spicepods by @lukekim in #8739
• Prepared Statements by @lukekim in #7588
• Schedule dispatch of testoperator run text-to-sql. by @Jeadie in #8745
• Fix minio for ai benchmark CI by @Jeadie in #8743
• Upgrade to Rust 1.91 by @phillipleblanc in #8749
• fix: Update benchmark snapshots by @app/github-actions in #8763
• Benchmarks: make row count validation skip logic configurable by scale factor, query set, and overrides by @sgrebnov in #8756
• Make benchmark tests more robust by @sgrebnov in #8766
• Add parameter to force using iam_role for DynamoDB by @krinart in #8767
• fix: Update Search integration test snapshots by @app/github-actions in #8735
• v1.10.4 release notes by @phillipleblanc in #8790
• Trace metrics export errors by @sgrebnov in #8791
• v1.10.4 SECURITY.md update by @phillipleblanc in #8800
• Add timezone database to Docker image to fix Cayenne acceleration panic by @sgrebnov in #8799
• Upgrade dependencies by @phillipleblanc in #8801
• Fix table_allowlist for table sampling and NSQL by @Jeadie in #8789
• Cayenne primary key on-conflict handling by @lukekim in #8788
• fix: Update benchmark snapshots by @app/github-actions in #8773
• fix: correctly identify deprecated openai_* parameters by @phillipleblanc in #8809
• fix: Update benchmark snapshots by @app/github-actions in #8812
• Use workspace version for cayenne crate by @phillipleblanc in #8811
• Don't CAST strings which breaks push down optimizer by @lukekim in #8810
• fix: Update benchmark snapshots by @app/github-actions in #8815
• Update async-openai to latest revision 4dcd633aad6f - brings fix for openai compatible model providers by @ewgenius in #8816
• Add auth/iam_role_source to DynamoDB connector by @krinart in #8808
• DynamoDB fixes: JSON nesting for Streams, proper batch deletions by @krinart in #8821
• Rough roadmap for 2026-2027 by @lukekim in #8805
• Release notes for v1.11.0-rc1 by @ewgenius in #8786
• Make S3V integration tests prepare_for_aws_tests more robust by @sgrebnov in #8820
• Bump rsa from 0.9.9 to 0.9.10 in the cargo group across 1 directory by @app/dependabot in #8819
• Add timezone database to Release and CUDA Docker images to fix Cayene panic by @sgrebnov in #8832
• fix: UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831
• Update grafana/datadog example dashboards by @krinart in #8833
• Add Dev bird bench as text-to-sql queryset in CI. by @Jeadie in #8753
• Update testoperator scheduler to use release/1.11 branch by @ewgenius in #8829
• Spice Cayenne fixes and test spicepods for Beta & RC by @lukekim in #8787
• testoperator dispatch all bird-bench database variants by @Jeadie in #8835
• feat: Improve column statistics handling with safe access and defaults by @phillipleblanc in #8836
• cluster: mTLS verification by @phillipleblanc in #8837
• fix: 8770: Unsupported ScalarFunctionExpr in ORDER BY by @lukekim in #8838
• Workflow tweaks by @lukekim in #8845
• Cayenne: metadata catalog should respect cayenne_file_path location by @sgrebnov in #8844
• Expand Cayenne feature coverage by @lukekim in #8848
• docs: HA distributed query decisions by @phillipleblanc in #8817
• fix(optimizer): Fix correctness issues in UnionProjectionPushdownOptimizer by @phillipleblanc in #8851
• Pin reqwest to 0.12.24 to fix HuggingFace embedding model download by @ewgenius in #8853
• Fix builds and pin to Ubuntu 22.04 by @lukekim in #8856
• Revert "Fix builds and pin to Ubuntu 22.04" by @lukekim in #8861
• Ensure setup Rust is run by @lukekim in #8862
• fix: Ubuntu 24.04+ renamed libaio1 to libaio1t64 by @lukekim in #8865
• Upgrade to Pulls with Spice v2 by @lukekim in #8866
• Add limit and configuration name to 'testoperator run text-to-sql' by @Jeadie in #8839
• PR check and test optimization by @lukekim in #8868
• Upgrade S3 Vectors SDK and improve test robustness by @lukekim in #8867
• [Testoperator] Query level and improved aggregate level for NSQL by @Jeadie in #8840
• Add docker build for private branches for ghcr.io/spiceai/spiceai-dev by @phillipleblanc in #8873
• Expand the data acceleration round-trip test coverage by @lukekim in #8855
• fix: Provide a better error for improper bucket UDF arguments by @peasee in #8849
• ScyllaDB Data Connector by @lukekim in #8827
• Use tokio-rusqlite for Spice Cayenne SQLite by @lukekim in #8857
• Cayenne: fix FuturesUnordered reentrant drop crash by @sgrebnov in #8863
• Bump github/codeql-action from 4.31.9 to 4.31.10 by @app/dependabot in #8884
• Bump golang.org/x/sys from 0.39.0 to 0.40.0 by @app/dependabot in #8881
• Bump github.com/spiceai/gospice/v8 from 8.0.0 to 8.0.1 by @app/dependabot in #8883
• Bump roaring from 0.11.2 to 0.11.3 by @app/dependabot in #8885
• Bump golang.org/x/mod from 0.31.0 to 0.32.0 by @app/dependabot in #8882
• Bump aws-sdk-s3 from 1.115.0 to 1.119.0 by @app/dependabot in #8887
• Bump libc from 0.2.177 to 0.2.180 by @app/dependabot in #8886
• Bump tokio-util from 0.7.17 to 0.7.18 by @app/dependabot in #8889
• Bump governor from 0.10.2 to 0.10.4 by @app/dependabot in #8888
• fix: flaky test test_concurrent_partition_creation by @phillipleblanc in #8898
• Update Cayenne snapshots for TPC-DS by @lukekim in #8890
• Add more buckets to histogram metrics by @krinart in #8850
• feat: Add HTTP health endpoint for cluster executors by @phillipleblanc in #8899
• feat: Implement model listing functionality for multiple providers by @lukekim in #8901
• feat: Initial HA schedulers distributed query implementation by @phillipleblanc in #8852
• fix: infer executor role from --scheduler-address when --role is omitted by @phillipleblanc in #8903
• Improve install scripts and remove jq dependency by @lukekim in #8847
• Benchmarks: sort PartitionedUnionExec children for deterministic snapshot comparison by @sgrebnov in #8877
• Cayenne: share VortexFileCache across partitions via CayenneContext by @sgrebnov in #8880
• Update ballista to add exponential backoff for scheduler disconnection by @phillipleblanc in #8905
• Configurably add BirdBench evidence to testoperator text-to-SQL. by @Jeadie in #8904
• Helm: Allow command override via values.yaml by @sgrebnov in #8906
• Fix distributed query gRPC message size limit (16MB -> 100MB) by @phillipleblanc in #8900
• OS specific setup actions by @lukekim in #8909
• Cayenne should warn if unable to parse configuration value by @sgrebnov in #8907
• Add snapshots widgets to example dashboard by @krinart in #8910
• Add quality criteria for the features by @krinart in #8897
• Improve Accelerated Datasets section for Grafana/Datadog dashboards by @krinart in #8915
• Use HTTP traceparent in NSQL to support concurrency in 'testoperator run text-to-SQL' by @Jeadie in #8912
• Remove setup for cc from integration_models.yml by @Jeadie in #8917
• Propagate Azure and GCS credentials to executors in cluster mode by @phillipleblanc in #8918
• Cayenne: fix memory growth due to vortex metrics allocation by @sgrebnov in #8908
• fix(caching): Deduplicate refresh requests for JSON array responses by @sgrebnov in #8921
• fix(caching): Return cached data directly for unfiltered queries (SELECT *) by @sgrebnov in #8919
• Correct MinIO path syntax for spiced download by @Jeadie in #8916
• Acceleration snapshots compaction + Improved Snapshots UX by @krinart in #8858
• Change base image from bookworm-slim to trixie-slim by @Jeadie in #8923
• Add testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895
• Fix spicepod dependencies in testoperator by @Jeadie in #8875
• Update copilot instructions for data correctness by @lukekim in #8922
• Add BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926
• fix: add missing feature-gate for AWS Secrets Manager error variant by @phillipleblanc in #8928
• refactor: make ConnectorParams fields public for external connectors by @phillipleblanc in #8929
• fix(caching): SWR refreshes only accessed entry instead of all stale rows by @sgrebnov in #8931
• Cayenne: include cayenne_metadata_dir to known params by @sgrebnov in #8933
• Rename Ingestion Lag chart in example dashboards by @krinart in #8932
• fix(caching): Fix HTTP caching always MISS when projection excludes fetched_at by @sgrebnov in #8930
• Reset expiry after snapshot bootstraping for Caching by @krinart in #8925
• Set use_ssl=false for sccache by @lukekim in #8945
• Hash indexing for Arrow Acceleration by @lukekim in #8924
• [Cayenne] Acceleration snapshots support by @lukekim in #7973
• perf(caching): Non-blocking cache writes on cache miss by @sgrebnov in #8948
• Update NSQL models by @lukekim in #8951
• Hash Index Key verification by @lukekim in #8949
• Add snapshots_creation_policy param by @krinart in #8954
• Remove candle & cudarc from non-models build by @lukekim in #8955
• Acceleration Snapshots API and CLI by @lukekim in #8934
• Ignore test for data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962
• fix: Update benchmark snapshots by @app/github-actions in #8965
• Hash Index secondary index support by @lukekim in #8958
• fix: Support primary key constraints in partitioned DuckDB tables mode by @sgrebnov in #8966
• perf(caching): Batch cache writes by @sgrebnov in #8959
• CI perf optimizations by @lukekim in #8968
• Fix Makefile linting by @Jeadie in #8970
• Fixes in testoperator run text-to-sql. by @Jeadie in #8927
• implement Chat::as_sql for xAI anthropic by @Jeadie in #8957
• Fix duckdb_file_path in search integration test by @Jeadie in #8972
• fix: Update benchmark snapshots by @app/github-actions in #8971
• Maintenance updates to Anthropic API by @Jeadie in #8956
• Add CacheBackend Trait, implement pingora-lru, and add throughput tests by @lukekim in #8080
• fix: Update benchmark snapshots by @app/github-actions in #8974
• fix: Update benchmark snapshots by @app/github-actions in #8975
• Make accelerator shutdown more robust by @lukekim in #8969
• feat(duckdb): Add on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964
• Proper handling for initial snapshot by @krinart in #8911
• fix: Remove --no-default-features from cargo-hack command in features workflow by @phillipleblanc in #8977
• build(deps): bump actions/cache from 5.0.1 to 5.0.2 by @app/dependabot in #8983
• build(deps): bump actions/checkout from 4 to 6 by @app/dependabot in #8982
• build(deps): bump actions/setup-go from 6.1.0 to 6.2.0 by @app/dependabot in #8984
• build(deps): bump github.com/olekukonko/tablewriter from 1.1.2 to 1.1.3 by @app/dependabot in #8979
• build(deps): bump github.com/klauspost/compress from 1.18.2 to 1.18.3 by @app/dependabot in #8980
• Add /v1/queries and Arrow Flight async APIs by @lukekim in #8946
• build(deps): bump Vampire/setup-wsl from 5 to 6 by @app/dependabot in #8981
• fix: Update Search integration test snapshots by @app/github-actions in #8973
• build(deps): bump insta from 1.46.0 to 1.46.1 by @app/dependabot in #8988
• build(deps): bump schemars from 1.1.0 to 1.2.0 by @app/dependabot in #8985
• fix: Update benchmark snapshots by @app/github-actions in #8978
• fix: Data correctness edge cases by @lukekim in #8953
• Correct MinIO path syntax for spiced download (Part 2) by @Jeadie in #8995
• Make .spice/data in search integration tests by @Jeadie in #8992
• fix: Hash index composite keys null values by @lukekim in #9001
• Update Cayenne status to Beta by @lukekim in #9002
• fix: Disable TPC-DS result validation (not yet supported) by @sgrebnov in #9004
• feat: Upgrade to DataFusion v51 and dependencies by @lukekim in #8864
• Improvements for snapshots_creation_policy by @krinart in #9003
• fix(ci): restore cached spicepod-validator binary instead of lookup-only by @phillipleblanc in #9007
• Update version by @krinart in #9010
• Update lock file - https://github.com/spiceai/spiceai/commit/53babbf07ca8c1c7b2e1da42ce58c465d9bc9276/
• fix: Enable Cayenne acceleration snapshots by @lukekim in #9020
• Add TPC-DS integration tests with S3 source and PostgreSQL acceleration by @phillipleblanc in #9006
• fix(tests): fix flaky/slow/failing unit tests by @phillipleblanc in #9009
• fix: Update benchmark snapshots for DF51 upgrade by @app/github-actions in #9008
• fix: add feature gate to rrf TEST_EMBEDDING_MODEL by @phillipleblanc in #9017
• fix: features check by @phillipleblanc in #9014
• URL table support by @lukekim in #9018
• ScyllaDB key filter by @lukekim in #8997
• fix: Schema mismatch when using column projection with HTTP caching by @phillipleblanc in #9021
• Add more tests for HTTP caching with columns selection by @sgrebnov in #9025
• HTTP cache snapshots: default to time_interval and fix snapshots_creation_policy: on_change by @sgrebnov in #9026
• Fix duplicate snapshot creation on startup by @sgrebnov in #9029
• Remove waiting for runtime to be ready before creating snapshot by @krinart in #9033
• Fix snapshot on_change policy to skip when no writes occurred by @sgrebnov in #9028
• Release notes for release release/1.11.0-rc.2 by @krinart in #9016
• ci: use arduino/setup-protoc for official protobuf compiler by @phillipleblanc in #9036
• ci: install unzip on aarch64 runner for arduino/setup-protoc by @phillipleblanc in #9038
• fix: don't fail release if upload to minio fails by @phillipleblanc in #9039
• Improve validation and logging for hash indexes by @lukekim in #9047
• Pin to ubuntu-22.04 by @lukekim in #9068
• Fix broken telemetry for testoperator by @krinart in #9054
• Fix release builds by @lukekim in #9069
• Spice 1.11.0-rc3 release notes by @krinart in #9070
• Update spicepod.schema.json by @app/github-actions in #9071
• Add missing protoc step to setup-cc action by @krinart in #9041
• Fix TLS connection for grpc+tls:// Flight SQL endpoints and add custom CA certificate support by @phillipleblanc in #9073
• Update 1.11.0-rc.3 release notes by @krinart in #9082
• Fix formula_1 and codebase_community in bird-bench by @Jeadie in #9000
• Cayenne S3 Express One Zone improvements by @lukekim in #9015
• Add zlib1g-dev to CI by @lukekim in #9052
• Upgrade Vortex with CASE-WHEN by @lukekim in #9051
• fix: Cayenne CatalogError handling for constraint violations by @lukekim in #9050
• Fix Docker build failing to copy shared libraries due to ldd output parsing by @phillipleblanc in #9058
• feat: Change /v1/sql and FlightSQL to use local execution in cluster mode by @phillipleblanc in #9055
• Remove unmaintained dependencies by @lukekim in #9045
• Enable cayenne + changes stream by @Jeadie in #9053
• feat(cli): add spice query command for async queries REPL by @phillipleblanc in #9057
• Remove unncessary allocations by @lukekim in #9059
• Add dataset_acceleration_size_bytes metric by @krinart in #9062
• Fix tracing of sql_query beneath tool_use::sample_data. by @Jeadie in #9043
• Basic script to run distributed spice by @Jeadie in #9049
• Add integration tests for Acceleration Snapshots by @krinart in #9067
• Upgrade CUDA toolkit to 12.6.0 by @sgrebnov in #9079
• Install required protoc dependency for CUDA build by @sgrebnov in #9080
• feat(cluster): add executor control stream heartbeat by @phillipleblanc in #9072
• feat: Fix async queries API and integrate Ballista shuffle improvements by @lukekim in #9075
• Remove models variant (now default) & Windows builds (use WSL) by @lukekim in #9063
• Cayenne: share upload semaphore across partitions to bound memory growth and optimize I/O by @sgrebnov in #9078
• Snowflake data connector - add snowflake_private_key parameter by @ewgenius in #9085
• Rewrite Go CLI in Rust by @phillipleblanc in #9061
• GCS Data Connector (Alpha) by @lukekim in #9084
• Skip 'latest' Docker tag for pre-release versions by @sgrebnov in #9077
• Add HTTP endpoints for acceleration snapshots API by @phillipleblanc in #9065
• Cayenne: Allow append mode with both primary_key and time_column by @sgrebnov in #9090
• Add Cluster Observability (Metrics+Dashboard) by @phillipleblanc in #9066
• proto for 'CayenneAccelerationExec' by @Jeadie in #9094
• Add 'anthropic-beta' header for structured outputs by @Jeadie in #9093
• Cayenne: refactor write path to use insert_into() as single entry point (part 1) by @sgrebnov in #9088
• Fix testoperator dispatch by @sgrebnov in #9097
• Fix setup-spiced GH action (_models suffix does not exist anymore) by @sgrebnov in #9102
• build(deps): bump github/codeql-action from 4.31.10 to 4.31.11 by @app/dependabot in #9108
• Remove DistributeFileScanOptimizer and UnionProjectionPushdownOptimizer & set target_partitions dynamically based on cluster capacity by @phillipleblanc in #9100
• build(deps): bump zip from 2.4.2 to 6.0.0 by @app/dependabot in #9111
• fix: Preserve query parameter order in HTTP connector to match filter values by @sgrebnov in #9114
• Add PollNow interrupt for Ballista executors to reduce task scheduling latency by @phillipleblanc in #9098
• Revert "GCS Data Connector (Alpha) " by @lukekim in #9084
• Fix stack overflow for CDC batching by @krinart in #9115
• fix: update Ballista fork to include executor timeout fix by @phillipleblanc in #9124
• Properly propagate SIGINT/SIGTERM from CLI to runtime by @krinart in #9127
• fix: Use the same vortex dependency as ballista by @peasee in #9123
• release: Bump version to 1.11.0 for stable - https://github.com/spiceai/spiceai/commit/14d09f8e262008df69ded898ed3bebee08471508/
• Cayenne snapshots with shared metadata by @lukekim in #9118
• Improve error handling for URL tables with Azure URLs by @phillipleblanc in #9129
• Add missing Windows build step for spice CLI in build_and_release workflow by @phillipleblanc in #9143
• Fix install-dev to use debug build path for spice binary by @phillipleblanc in #9142
• fix: CLI builds by @peasee in #9145
• Always create initial snapshots (unless bootstrapped) + when no snapshots exist by @krinart in #9119
• fix(cayenne): Fix upsert with pending deletions causing duplicate PKs by @sgrebnov in #9152
• fix(flightrepl): Add chrono-tz feature to flightrepl for timezone formatting by @sgrebnov in #9153
• fix(delta_lake): Preserve container name in ABFSS URLs for Azure Delta Lake tables by @sgrebnov in #9155
• fix: Make CLI system and asset type detection more robust by @peasee in #9148
• fix: Set query set properly on benchmarks telemetry metrics attributes by @peasee in #9162
• fix: Download _models variant - https://github.com/spiceai/spiceai/commit/27f3058d0007595b02c198755c3b22319032ff30/
• fix: Helm chart image tag - https://github.com/spiceai/spiceai/commit/7405c8df0db4ecce0ed6d4a4d424553d604b9036/
• Revert "Remove models variant (now default) & Windows builds (use WSL) " by @lukekim in #9063
• fix(cli): Several CLI fixes from the Go to Rust migration by @lukekim in #9157

Announcing the release of Spice v1.11.0-rc.2! ⭐

v1.11.0-rc.2 is the second release candidate for advanced test of v1.11. It brings Spice Cayenne to Beta status with acceleration snapshots support, a new ScyllaDB Data Connector, upgrades to DataFusion v51, Arrow 57.2, and iceberg-rust v0.8.0. It includes significant improvements to distributed query, caching, and observability.

What's New in v1.11.0-rc.2​

Spice Cayenne Accelerator Reaches Beta​

Spice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous stability improvements.

Improved Reliability:

• Fixed timezone database issues in Docker images that caused acceleration panics
• Resolved FuturesUnordered reentrant drop crashes
• Fixed memory growth issues related to Vortex metrics allocation
• Metadata catalog now properly respects cayenne_file_path location
• Added warnings for unparseable configuration values

Example configuration with snapshots:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file

DataFusion v51 Upgrade​

Apache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.

Performance Improvements:

• Faster CASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patterns
• Better Defaults for Remote Parquet Reads: DataFusion now fetches the last 512KB of Parquet files by default, typically avoiding 2 I/O requests per file
• Faster Parquet Metadata Parsing: Leverages Arrow 57's new thrift metadata parser for up to 4x faster metadata parsing

New SQL Features:

• SQL Pipe Operators: Support for |> syntax for inline transforms
• DESCRIBE <query>: Returns the schema of any query without executing it
• Named Arguments in SQL Functions: PostgreSQL-style param => value syntax for scalar, aggregate, and window functions
• Decimal32/Decimal64 Support: New Arrow types supported including aggregations like SUM, AVG, and MIN/MAX

Example pipe operator:

SELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;

Improved Observability:

• Improved EXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdowns

Arrow 57.2 Upgrade​

Spice has been upgraded to Apache Arrow Rust 57.2.0, bringing major performance improvements and new capabilities.

Key Features:

• 4x Faster Parquet Metadata Parsing: A rewritten thrift metadata parser delivers up to 4x faster metadata parsing, especially beneficial for low-latency use cases and files with large amounts of metadata
• Parquet Variant Support: Experimental support for reading and writing the new Parquet Variant type for semi-structured data, including shredded variant values
• Parquet Geometry Support: Read and write support for Parquet Geometry types (GEOMETRY and GEOGRAPHY) with GeospatialStatistics
• New arrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution support

iceberg-rust v0.8.0 Upgrade​

Spice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.

Key Features:

• V3 Metadata Support: Full support for Iceberg V3 table metadata format
• INSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tables
• Improved Delete File Handling: Better support for position and equality delete files, including shared delete file loading and caching
• SQL Catalog Updates: Implement update_table and register_table for SQL catalog
• S3 Tables Catalog: Implement update_table for S3 Tables catalog
• Enhanced Arrow Integration: Convert Arrow schema to Iceberg schema with auto-assigned field IDs, _file column support, and Date32 type support

Acceleration Snapshots​

Acceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.

Key Feature Improvements in v1.11:

• Flexible Triggers: Configure when snapshots are created based on time intervals or stream batch counts
• Automatic Compaction: Reduce storage overhead by compacting older snapshots (DuckDB only)
• Bootstrap Integration: Snapshots can reset cache expiry on load for seamless recovery (DuckDB with Caching refresh mode)
• Smart Creation Policies: Only create snapshots when data has actually changed

Example configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled
      snapshots_trigger: time_interval
      snapshots_trigger_threshold: 1h
      snapshots_creation_policy: on_changed

Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically. List, create, and restore snapshots directly from the command line or via HTTP.

For more details, refer to the Acceleration Snapshots Documentation.

ScyllaDB Data Connector​

A new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.

Example configuration:

datasets:
  - from: scylladb:my_keyspace.my_table
    name: scylla_data
    acceleration:
      enabled: true
      engine: duckdb

For more details, refer to the ScyllaDB Data Connector Documentation.

Distributed Query Improvements​

mTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.

Credential Propagation: Azure and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.

Improved Resilience:

• Exponential backoff for scheduler disconnection recovery
• Increased gRPC message size limit from 16MB to 100MB for large query plans
• HTTP health endpoint for cluster executors
• Automatic executor role inference when --scheduler-address is provided

For more details, refer to the Distributed Query Documentation.

Caching Acceleration Mode Improvements​

The Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.

Performance Optimizations:

• Non-blocking Cache Writes: Cache misses no longer block query responses. Data is written to the cache asynchronously after the query returns, reducing query latency for cache miss scenarios.
• Batch Cache Writes: Multiple cache entries are now written in batches rather than individually, significantly improving write throughput for high-volume cache operations.

Reliability Fixes:

• Correct SWR Refresh Behavior: The stale-while-revalidate (SWR) pattern now correctly refreshes only the specific entries that were accessed instead of refreshing all stale rows in the dataset. This prevents unnecessary source queries and reduces load on upstream data sources.
• Deduplicated Refresh Requests: Fixed an issue where JSON array responses could trigger multiple redundant refresh operations. Refresh requests are now properly deduplicated.
• Fixed Cache Hit Detection: Resolved an issue where queries that didn't include fetched_at in their projection would always result in cache misses, even when cached data was available.
• Unfiltered Query Optimization: SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.

For more details, refer to the Caching Acceleration Mode Documentation.

DynamoDB Connector Enhancements​

• Added JSON nesting for DynamoDB Streams
• Proper batch deletion handling

URL Tables​

Query data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.

Supported Patterns:

• Single files: SELECT * FROM 's3://bucket/data.parquet'
• Directories/prefixes: SELECT * FROM 's3://bucket/data/'
• Glob patterns: SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'

Key Features:

• Automatic file format detection (Parquet, CSV, JSON, etc.)
• Hive-style partition inference with filter pushdown
• Schema inference from files
• Works with both SQL and DataFrame APIs

Example with hive partitioning:

-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'

Enable via spicepod.yml:

runtime:
  params:
    url_tables: enabled

Cluster Mode Async Query APIs (experimental)​

New asynchronous query APIs for long-running queries in cluster mode:

• /v1/queries endpoint: Submit queries and retrieve results asynchronously
• Arrow Flight async support: Non-blocking query execution via Arrow Flight protocol

Observability Improvements​

Enhanced Dashboards: Updated Grafana and Datadog example dashboards with:

• Snapshot monitoring widgets
• Improved accelerated datasets section
• Renamed ingestion lag charts for clarity

Additional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.

For more details, refer to the Monitoring Documentation.

Additional Improvements​

• Model Listing: New functionality to list available models across multiple AI providers
• DuckDB Partitioned Tables: Primary key constraints now supported in partitioned DuckDB table mode
• Post-refresh Sorting: New on_refresh_sort_columns parameter for DuckDB enables data ordering after writes
• Improved Install Scripts: Removed jq dependency and improved cross-platform compatibility
• Better Error Messages: Improved error messaging for bucket UDF arguments and deprecated OpenAI parameters

Contributors​

• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

New ScyllaDB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.

New SMB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See SMB Data Connector Recipe for details.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.0-rc.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:v1.11.0-rc.2 image:

docker pull spiceai/spiceai:v1.11.0-rc.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

Spice is available in the AWS Marketplace.

Dependencies​

• DataFusion: Upgraded to v51 (release notes)
• Arrow: Upgraded to v57 (release notes)
• iceberg-rust: Upgraded to v0.8.0 (release notes)

Changelog​

• Add timezone database to Docker image to fix Cayenne acceleration panic by @sgrebnov in #8799
• Upgrade dependencies by @phillipleblanc in #8801
• Fix table_allowlist for table sampling and NSQL by @Jeadie in #8789
• Cayenne primary key on-conflict handling by @lukekim in #8788
• fix: Update benchmark snapshots by @app/github-actions in #8773
• fix: correctly identify deprecated openai_* parameters by @phillipleblanc in #8809
• fix: Update benchmark snapshots by @app/github-actions in #8812
• Use workspace version for cayenne crate by @phillipleblanc in #8811
• Don't CAST strings which breaks push down optimizer by @lukekim in #8810
• fix: Update benchmark snapshots by @app/github-actions in #8815
• Update async-openai to latest revision 4dcd633aad6f - brings fix for openai compatible model providers by @ewgenius in #8816
• Add auth/iam_role_source to DynamoDB connector by @krinart in #8808
• DynamoDB fixes: JSON nesting for Streams, proper batch deletions by @krinart in #8821
• Rough roadmap for 2026-2027 by @lukekim in #8805
• Release notes for v1.11.0-rc1 by @ewgenius in #8786
• Make S3V integration tests prepare_for_aws_tests more robust by @sgrebnov in #8820
• Bump rsa from 0.9.9 to 0.9.10 in the cargo group across 1 directory by @app/dependabot in #8819
• Add timezone database to Release and CUDA Docker images to fix Cayene panic by @sgrebnov in #8832
• fix: UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831
• Update grafana/datadog example dashboards by @krinart in #8833
• Add Dev bird bench as text-to-sql queryset in CI. by @Jeadie in #8753
• Update testoperator scheduler to use release/1.11 branch by @ewgenius in #8829
• Spice Cayenne fixes and test spicepods for Beta & RC by @lukekim in #8787
• testoperator dispatch all bird-bench database variants by @Jeadie in #8835
• feat: Improve column statistics handling with safe access and defaults by @phillipleblanc in #8836
• cluster: mTLS verification by @phillipleblanc in #8837
• fix: 8770: Unsupported ScalarFunctionExpr in ORDER BY by @lukekim in #8838
• Workflow tweaks by @lukekim in #8845
• Cayenne: metadata catalog should respect cayenne_file_path location by @sgrebnov in #8844
• Expand Cayenne feature coverage by @lukekim in #8848
• docs: HA distributed query decisions by @phillipleblanc in #8817
• fix(optimizer): Fix correctness issues in UnionProjectionPushdownOptimizer by @phillipleblanc in #8851
• Pin reqwest to 0.12.24 to fix HuggingFace embedding model download by @ewgenius in #8853
• Fix builds and pin to Ubuntu 22.04 by @lukekim in #8856
• Revert "Fix builds and pin to Ubuntu 22.04" by @lukekim in #8861
• Ensure setup Rust is run by @lukekim in #8862
• fix: Ubuntu 24.04+ renamed libaio1 to libaio1t64 by @lukekim in #8865
• Upgrade to Pulls with Spice v2 by @lukekim in #8866
• Add limit and configuration name to 'testoperator run text-to-sql' by @Jeadie in #8839
• PR check and test optimization by @lukekim in #8868
• Upgrade S3 Vectors SDK and improve test robustness by @lukekim in #8867
• [Testoperator] Query level and improved aggregate level for NSQL by @Jeadie in #8840
• Add docker build for private branches for ghcr.io/spiceai/spiceai-dev by @phillipleblanc in #8873
• Expand the data acceleration round-trip test coverage by @lukekim in #8855
• fix: Provide a better error for improper bucket UDF arguments by @peasee in #8849
• ScyllaDB Data Connector by @lukekim in #8827
• Use tokio-rusqlite for Spice Cayenne SQLite by @lukekim in #8857
• Cayenne: fix FuturesUnordered reentrant drop crash by @sgrebnov in #8863
• Bump github/codeql-action from 4.31.9 to 4.31.10 by @app/dependabot in #8884
• Bump golang.org/x/sys from 0.39.0 to 0.40.0 by @app/dependabot in #8881
• Bump github.com/spiceai/gospice/v8 from 8.0.0 to 8.0.1 by @app/dependabot in #8883
• Bump roaring from 0.11.2 to 0.11.3 by @app/dependabot in #8885
• Bump golang.org/x/mod from 0.31.0 to 0.32.0 by @app/dependabot in #8882
• Bump aws-sdk-s3 from 1.115.0 to 1.119.0 by @app/dependabot in #8887
• Bump libc from 0.2.177 to 0.2.180 by @app/dependabot in #8886
• Bump tokio-util from 0.7.17 to 0.7.18 by @app/dependabot in #8889
• Bump governor from 0.10.2 to 0.10.4 by @app/dependabot in #8888
• fix: flaky test test_concurrent_partition_creation by @phillipleblanc in #8898
• Update Cayenne snapshots for TPC-DS by @lukekim in #8890
• Add more buckets to histogram metrics by @krinart in #8850
• feat: Add HTTP health endpoint for cluster executors by @phillipleblanc in #8899
• feat: Implement model listing functionality for multiple providers by @lukekim in #8901
• feat: Initial HA schedulers distributed query implementation by @phillipleblanc in #8852
• fix: infer executor role from --scheduler-address when --role is omitted by @phillipleblanc in #8903
• Improve install scripts and remove jq dependency by @lukekim in #8847
• Benchmarks: sort PartitionedUnionExec children for deterministic snapshot comparison by @sgrebnov in #8877
• Cayenne: share VortexFileCache across partitions via CayenneContext by @sgrebnov in #8880
• Update ballista to add exponential backoff for scheduler disconnection by @phillipleblanc in #8905
• Configurably add BirdBench evidence to testoperator text-to-SQL. by @Jeadie in #8904
• Helm: Allow command override via values.yaml by @sgrebnov in #8906
• Fix distributed query gRPC message size limit (16MB -> 100MB) by @phillipleblanc in #8900
• OS specific setup actions by @lukekim in #8909
• Cayenne should warn if unable to parse configuration value by @sgrebnov in #8907
• Add snapshots widgets to example dashboard by @krinart in #8910
• Add quality criteria for the features by @krinart in #8897
• Improve Accelerated Datasets section for Grafana/Datadog dashboards by @krinart in #8915
• Use HTTP traceparent in NSQL to support concurrency in 'testoperator run text-to-SQL' by @Jeadie in #8912
• Remove setup for cc from integration_models.yml by @Jeadie in #8917
• Propagate Azure and GCS credentials to executors in cluster mode by @phillipleblanc in #8918
• Cayenne: fix memory growth due to vortex metrics allocation by @sgrebnov in #8908
• fix(caching): Deduplicate refresh requests for JSON array responses by @sgrebnov in #8921
• fix(caching): Return cached data directly for unfiltered queries (SELECT *) by @sgrebnov in #8919
• Correct MinIO path syntax for spiced download by @Jeadie in #8916
• Acceleration snapshots compaction + Improved Snapshots UX by @krinart in #8858
• Change base image from bookworm-slim to trixie-slim by @Jeadie in #8923
• Add testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895
• Fix spicepod dependencies in testoperator by @Jeadie in #8875
• Update copilot instructions for data correctness by @lukekim in #8922
• Add BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926
• fix: add missing feature-gate for AWS Secrets Manager error variant by @phillipleblanc in #8928
• refactor: make ConnectorParams fields public for external connectors by @phillipleblanc in #8929
• fix(caching): SWR refreshes only accessed entry instead of all stale rows by @sgrebnov in #8931
• Cayenne: include cayenne_metadata_dir to known params by @sgrebnov in #8933
• Rename Ingestion Lag chart in example dashboards by @krinart in #8932
• fix(caching): Fix HTTP caching always MISS when projection excludes fetched_at by @sgrebnov in #8930
• Reset expiry after snapshot bootstraping for Caching by @krinart in #8925
• Set use_ssl=false for sccache by @lukekim in #8945
• Hash indexing for Arrow Acceleration by @lukekim in #8924
• [Cayenne] Acceleration snapshots support by @lukekim in #7973
• perf(caching): Non-blocking cache writes on cache miss by @sgrebnov in #8948
• Update NSQL models by @lukekim in #8951
• Hash Index Key verification by @lukekim in #8949
• Add snapshots_creation_policy param by @krinart in #8954
• Remove candle & cudarc from non-models build by @lukekim in #8955
• Acceleration Snapshots API and CLI by @lukekim in #8934
• Ignore test for data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962
• fix: Update benchmark snapshots by @app/github-actions in #8965
• Hash Index secondary index support by @lukekim in #8958
• fix: Support primary key constraints in partitioned DuckDB tables mode by @sgrebnov in #8966
• perf(caching): Batch cache writes by @sgrebnov in #8959
• CI perf optimizations by @lukekim in #8968
• Fix Makefile linting by @Jeadie in #8970
• Fixes in testoperator run text-to-sql. by @Jeadie in #8927
• implement Chat::as_sql for xAI anthropic by @Jeadie in #8957
• Fix duckdb_file_path in search integration test by @Jeadie in #8972
• fix: Update benchmark snapshots by @app/github-actions in #8971
• Maintenance updates to Anthropic API by @Jeadie in #8956
• Add CacheBackend Trait, implement pingora-lru, and add throughput tests by @lukekim in #8080
• fix: Update benchmark snapshots by @app/github-actions in #8974
• fix: Update benchmark snapshots by @app/github-actions in #8975
• Make accelerator shutdown more robust by @lukekim in #8969
• feat(duckdb): Add on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964
• Proper handling for initial snapshot by @krinart in #8911
• fix: Remove --no-default-features from cargo-hack command in features workflow by @phillipleblanc in #8977
• build(deps): bump actions/cache from 5.0.1 to 5.0.2 by @app/dependabot in #8983
• build(deps): bump actions/checkout from 4 to 6 by @app/dependabot in #8982
• build(deps): bump actions/setup-go from 6.1.0 to 6.2.0 by @app/dependabot in #8984
• build(deps): bump github.com/olekukonko/tablewriter from 1.1.2 to 1.1.3 by @app/dependabot in #8979
• build(deps): bump github.com/klauspost/compress from 1.18.2 to 1.18.3 by @app/dependabot in #8980
• Add /v1/queries and Arrow Flight async APIs by @lukekim in #8946
• build(deps): bump Vampire/setup-wsl from 5 to 6 by @app/dependabot in #8981
• fix: Update Search integration test snapshots by @app/github-actions in #8973
• build(deps): bump insta from 1.46.0 to 1.46.1 by @app/dependabot in #8988
• build(deps): bump schemars from 1.1.0 to 1.2.0 by @app/dependabot in #8985
• fix: Update benchmark snapshots by @app/github-actions in #8978
• fix: Data correctness edge cases by @lukekim in #8953
• Correct MinIO path syntax for spiced download (Part 2) by @Jeadie in #8995
• Make .spice/data in search integration tests by @Jeadie in #8992
• fix: Hash index composite keys null values by @lukekim in #9001
• Update Cayenne status to Beta by @lukekim in #9002
• fix: Disable TPC-DS result validation (not yet supported) by @sgrebnov in #9004
• feat: Upgrade to DataFusion v51 and dependencies by @lukekim in #8864
• Improvements for snapshots_creation_policy by @krinart in #9003
• fix(ci): restore cached spicepod-validator binary instead of lookup-only by @phillipleblanc in #9007
• Update version by @krinart in #9010