Skip to main content

Spice v2.0-rc.5 (May 27, 2026)

· 30 min read
Jack Eadie
Jack Eadie
Token Plumber at Spice AI

Spice v2.0-rc.5 is now available! 🔥

v2.0.0-rc.5 is the fifth release candidate for advanced testing of v2.0, building on v2.0.0-rc.4.

This release completes the mTLS implementation across server endpoints and outbound connectors, adds MongoDB Change Streams and durable Kafka offset persistence as new CDC sources, expands DML write-back to PostgreSQL, Snowflake, and Arrow, promotes DuckLake to Beta, introduces user-defined functions, on-demand dataset loading, unified query cancellation, dynamic HTTP request headers and subquery-driven request parameters, provider-aware LLM prompt caching, and a long list of Cayenne performance improvements.

Highlights in this release candidate include:

  • Spice Cayenne — CDC throughput, compaction and scan caching, synchronized partition commits, join filter propagation, parallel Vortex writes, lock-free deletion caches
  • Mutual TLS (mTLS) — TLS cert hot-reload, public mTLS for HTTP and Flight (channel + identity modes), mTLS client certs for FlightSQL and Spice.ai connectors
  • MongoDB Change Streams — native real-time CDC for MongoDB, no Debezium or Kafka required
  • Kafka CDC offsets — offsets persisted in sidecar tables for durable, resumable Kafka CDC
  • PostgreSQL DML — INSERT, UPDATE, DELETE write-back on PostgreSQL datasets
  • Snowflake DML — INSERT, UPDATE, DELETE write-back on Snowflake datasets
  • Arrow Primary Key Upserts — native upsert path using primary key matching
  • DuckLake promoted to Beta — with INSERT support on catalog tables
  • User-Defined Functions — define SQL UDFs in spicepods, plus remote UDFs over HTTP (Spice.ai Enterprise)
  • Spatial SQL UDFs — optional geospatial UDFs (ST_*) for geometry workloads
  • On-Demand Dataset Loading — datasets can be deferred and loaded on first reference
  • Unified Query Cancellation — Ctrl-C and HTTP request cancellation propagate across all execution paths
  • Dynamic HTTP Connector — pass-through request headers, subquery-driven params, and JSON schema decomposition
  • HTTP Rate-Control persistence — rate-limit state persisted in object storage across restarts
  • refresh_mode: snapshot — point-in-time snapshot acceleration with SQLite/Turso WAL flushing
  • Storage-profile accelerator tuning — accelerators auto-tune defaults based on local SSD, EBS-class disk, or tmpfs
  • Provider-Aware LLM Prompt Caching — automatic prompt caching for OpenAI-compatible providers that support it
  • Responses API — support across all model providers with streaming response.output_text.delta, plus Authorization: Bearer header support

What's New in v2.0.0-rc.5​

Cayenne Improvements​

Significant performance work across Spice Cayenne-backed catalogs and accelerators.

  • Ingest throughput: End-to-end improvements to CDC ingest, background compaction, and a new scan-result cache for hot reads; parallel Vortex partition writes; lock-free deletion caches with bloom-prefiltered probes; background retention with CDC pipelining; SQLite metastore pool scaled to 32 for high-concurrency mutation workloads.
  • Data inlining: Small writes are serialized as Arrow IPC and committed directly into the Cayenne metastore (cayenne_inlined_data), bypassing the staged Vortex write path for low-latency ingest. Inline upserts atomically rewrite existing inline rows instead of emitting side delete markers, and inline data remains query-visible via an in-memory union scan with a generation-keyed decode cache. Inline rows are checkpointed to Vortex when row, segment, or byte thresholds are reached. Defaults are refresh-mode aware: inline writes are enabled by default for high-frequency caching, changes, and fast append workloads and disabled for full, snapshot, and slower append.
  • Query planning: Join filter propagation across equi-join keys (gated behind runtime.params.cayenne_filter_propagation), range fallback for large join filters, hot-path clone elimination, and IN-list rewrites for large filter lists.
  • Correctness: Synchronized partition commits across partitions, correct NULL-sentinel handling for nullable partition expressions (e.g. bucket(N, col)), Vortex panic fix on highly compressible data, and live reads through expired protected snapshots.
  • Catalog and platform: Refresh-mode-aware compaction defaults, rejection of non-distributed Cayenne catalog configurations, and a vendored Vortex DataFusion integration for faster iteration on the Cayenne planner.

Mutual TLS (mTLS)​

Spice.ai Enterprise feature. See Enterprise Security.

Spice now supports full mutual TLS for both HTTP and Arrow Flight endpoints.

TLS cert hot-reload (#10727): The Spice runtime watches for SIGHUP and reloads TLS certificates without restarting, enabling cert rotation with zero downtime.

Public mTLS for HTTP and Flight (#10753): Two client_auth_mode values control how the server handles client certificates:

  • request — optional mTLS: the server requests a client cert but accepts connections without one (useful for migration windows).
  • required — strict mTLS: the server requires a valid client cert signed by the configured CA.

mTLS client certs for FlightSQL and Spice.ai connectors (#10764): Outbound connections from the FlightSQL and Spice.ai data connectors can now present client certificates for mutual authentication with upstream services.

Example configuration:

runtime:
  tls:
    enabled: true
    certificate_file: /etc/spice/tls/server.crt
    key_file: /etc/spice/tls/server.key
    client_auth_mode: required
    client_auth_ca_file: /etc/spice/tls/client-ca.crt

MongoDB Change Streams​

MongoDB datasets configured with refresh_mode: changes now stream changes from MongoDB Change Streams into any local accelerator (#10813), providing real-time CDC without Debezium or Kafka.

Example configuration:

datasets:
  - from: mongodb:my_collection
    name: my_collection
    params:
      host: my-cluster.mongodb.net
      db: mydb
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: changes

CDC Improvements​

See Change Data Capture (CDC) for an overview of CDC in Spice.

  • Kafka CDC offset persistence (#10823): Kafka CDC offsets are persisted in sidecar tables for durable, resumable streams. On restart or failover, Spice resumes from the last committed offset.
  • Pipelined CDC ingestion (#10676): Source reads overlap with batch apply, with additional batching, envelope coalescing, and nullability propagation improvements across the apply pipeline.
  • Debezium schema evolution fix (#10144): Schema changes in Debezium-sourced datasets no longer break dataset initialization on reload (fixes #9782).

PostgreSQL DML Support​

The PostgreSQL data connector now supports write-back via INSERT, UPDATE, and DELETE operations (#10446). Combined with the existing read-side federation, PostgreSQL-backed datasets can serve as full read/write tables. The PostgreSQL Catalog connector additionally exposes foreign-key metadata for NSQL and query planning (#10849).

Snowflake DML Support​

The Snowflake data connector now supports write-back via INSERT, UPDATE, and DELETE operations (#10747), complementing its existing read capabilities.

Arrow Primary Key Upserts​

Arrow-accelerated tables now support native upsert operations using primary key matching (#10749), providing efficient update-or-insert semantics for in-memory datasets.

DuckLake Promoted to Beta​

The DuckLake Catalog and Data Connector are promoted to Beta quality (#10743).

DuckLake catalog tables with read_write access now support INSERT operations (#10744), enabling full read/write workflows against DuckLake-backed catalogs. The DuckLake connector also gains a series of correctness fixes for downcast, module registration, schema discovery, and S3 credentials (#10650).

User-Defined Functions​

Spice now supports user-defined functions (UDFs) as a first-class spicepod component (#10571), letting you define reusable SQL functions in the spicepod or invoke remote functions over HTTP. The runtime also gains table user functions with HTTP server gating (#10675).

A security fix closes a remote-UDF SSRF vector (#10757).

Spatial SQL UDFs​

Spice now ships an optional set of geospatial SQL UDFs (ST_*) for geometry workloads (#10833). The functions are gated behind a build feature and can be invoked from any SQL surface.

On-Demand Dataset Loading​

Datasets can now be marked for on-demand loading (#10629). Deferred datasets are registered with a declared schema at startup (#10669) and only fully resolve when first referenced, reducing startup time and memory footprint for spicepods with many seldom-used datasets.

Spicepods also gain columns[].type and columns[].nullable (#10661) with a lenient type parser for declaring schemas inline.

Unified Query Cancellation​

All query execution paths — HTTP, Flight, FlightSQL, MCP, and internal — now honour a unified cancellation signal (#10390). When a client disconnects, presses Ctrl-C in the REPL, or cancels an in-flight HTTP request, the corresponding query is cancelled end-to-end, freeing resources promptly.

Dynamic HTTP Connector​

The HTTP data connector gains dynamic request headers parameterised from query predicates (#10604), subquery-driven request parameters for fan-out queries (#10636), HTTP response metadata as queryable columns via JSON schema decomposition (#10679), no-limit pagination (#10673), and shared rate-control across HTTP-based connectors using the same backend host (#10648).

HTTP Rate-Control Persistence​

The HTTP rate-control state (per-endpoint throttle counters) is now persisted in object storage (#10697), ensuring rate limits survive restarts and are consistent across replicas. Rate-control metrics now use an origin label rather than the connector name for cleaner aggregation (#10689).

The metrics HTTP endpoint (/metrics) is also independently rate-limited (#10162) to prevent scraping from impacting query serving.

refresh_mode: snapshot​

Spice.ai Enterprise feature. See Acceleration Snapshots.

A new refresh_mode: snapshot provides point-in-time snapshot acceleration (#10651), with SQLite and Turso WAL flushing and a Cayenne metastore slice integration so accelerated readers see a consistent snapshot while writes continue.

Storage-Profile Accelerator Tuning​

Acceleration configs gain a new storage_profile field (#10913) with values auto (default), local_ssd, ebs, and tmpfs. Under auto, the runtime detects whether the acceleration store is backed by local SSD, EBS-class network disk, or tmpfs, and applies storage-aware defaults across DuckDB, partitioned DuckDB, SQLite, Turso, and Cayenne file-mode accelerators. Explicit per-accelerator parameters always override the profile defaults.

Provider-Aware LLM Prompt Caching​

LLM calls automatically use provider-aware prompt caching (#10645) when the configured model provider supports it (e.g., Anthropic, OpenAI). System prompts and tool descriptions are marked for caching so repeated invocations within the cache window reuse the provider-side cached prefix, reducing latency and cost.

A new searchable registry mode for LLM tools (#10647) lets agents discover tools by semantic search rather than enumerating all tools in the system prompt, which scales to large tool inventories.

Responses API Improvements​

The Responses API is now supported across all configured model providers (#10724). Streaming delta events via response.output_text.delta are also supported (#10828). The runtime now also accepts Authorization: Bearer headers in addition to x-api-key, bumps async-openai, and stops populating FunctionToolCall.id so OpenAI-compatible servers can assign the ID themselves (#10911).

Distributed Cluster Improvements​

Spice.ai Enterprise feature. See High Availability.

  • Per-request executor readiness gate (#10860): /v1/ready on schedulers waits for a configurable quorum of executors before returning healthy, enabling proper rolling deployments.
  • Ballista S3 shuffle reads under cluster mode (#10910): The shuffle reader builds its S3 client from the executor pod's environment, matching the writer. Async queries with runtime.params.shuffle_location: s3://... now complete instead of failing with AccessDenied on shuffle fetches.
  • Flattened scheduler config (#10450): runtime.scheduler.partition_management.* fields are flattened directly onto runtime.scheduler and renamed under the canonical "partition assignment" terminology. See Breaking Changes.

Caching & Search​

Improvements across Caching and Search:

  • Per-principal cache namespacing (#10702): SQL, search, and caching-accelerator caches are now namespaced per authenticated principal, so cached results never cross identity boundaries.
  • DuckDB HNSW vector indexes (#10695, #10674, #10668): DuckDB-accelerated views support HNSW vector indexes for vector search, vector search SQL is rewritten to activate HNSW_INDEX_SCAN, and HNSW indexes are preserved across data refresh.

Security Improvements​

See Authentication and TLS for configuring Spice security.

  • API key timing-position leak and remote-UDF SSRF (#10757): Closed a timing-based position-disclosure leak in API key comparison and blocked SSRF via remote UDF endpoint parameters.
  • Configurable allowed_hosts for MCP (#10638): MCP servers can be restricted to an explicit allowlist of upstream hosts.

SQL, Query, and Developer Experience​

See the SQL Reference for the full SQL surface area.

  • SQL REPL expanded view (#10797): Toggle \x in the REPL for a vertical key-value layout on wide result sets.
  • FlightSQL Substrait plan support (#10761): The Spice runtime now implements CommandStatementSubstraitPlan, enabling clients that submit plans as Substrait-encoded protobuf.
  • MCP auth for streamable HTTP tools (#10927): Streamable HTTP MCP tools support native authentication via mcp_auth_token and mcp_headers, both with full Spice secret expansion.
  • Elasticsearch FTS engine config and index lifecycle (#10672): Direct FTS engine configuration plus index lifecycle and ingestion controls for the Elasticsearch connector.
  • Self-hosted Spice connector (#10546): Connect Spice to another self-hosted Spice runtime as a federated source.

Connector Bug Fixes​

Notable correctness fixes across the Data Connectors: DynamoDB Streams retry on transient errors (#10794) and typed-NULL handling in DML (#10511); ScyllaDB physical filter pushdown disabled to fix incorrect results (#10772); MSSQL TOP N pushdown for non-nullable sort columns (#10621); DuckLake include filter applied (#10738); DuckDB DELETE/UPDATE on full and caching refresh modes (#10632); checked arithmetic for Turso integer-millis and timestamp-to-nanosecond conversions (#10786, #10666); and Flight GetFlightInfo/DoGet schema parity (#10864). See the Changelog for the full list.

Dependency Updates​

Dependency / ComponentVersion
DuckDBv1.5.2
Icebergv0.9.1
Tursov0.6.0
Vortexv0.69.0

Contributors​

Breaking Changes​

Flattened runtime.scheduler configuration (#10450): The nested runtime.scheduler.partition_management block has been flattened and renamed to use the canonical "partition assignment" terminology. Migrate as follows:

# Before
runtime:
  scheduler:
    partition_management:
      interval: 30s
      max_assignments_per_cycle: 16
      discovery_timeout: 10s

# After
runtime:
  scheduler:
    partition_assignment_interval: 30s
    max_assignments_per_interval: 16
    partition_discovery_timeout: 10s

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.0-rc.5, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.5

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.5 image:

docker pull spiceai/spiceai:2.0.0-rc.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.5

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • Enable DML support for PostgreSQL data connector by @phillipleblanc in #10446
  • feat(postgres): support inline PEM sslrootcert by @claudespice in #10578
  • Add foreign key metadata discovery to PostgreSQL Catalog by @sgrebnov in #10849
  • Add Snowflake DML support by @lukekim in #10747
  • Add MongoDB Change Streams support by @lukekim in #10813
  • Add user-defined functions by @lukekim in #10571
  • Add table user functions and gate HTTP servers by @lukekim in #10675
  • feat: add on-demand dataset loading by @phillipleblanc in #10629
  • feat(runtime): declared-schema deferred datasets by @phillipleblanc in #10669
  • feat(spicepod, runtime): add columns[].type / nullable + lenient type parser by @phillipleblanc in #10661
  • Replace external smb crate with internal SMB 3.1.1 client by @phillipleblanc in #10516
  • Add unified query cancellation across all paths by @lukekim in #10390
  • Add dynamic HTTP request headers by @lukekim in #10604
  • feat(http): Support dynamic HTTP connector request params from subqueries by @lukekim in #10636
  • feat(http): pass through HTTP metadata columns with JSON schema decomposition by @lukekim in #10679
  • Add nolimit HTTP pagination max pages by @lukekim in #10673
  • Add shared HTTP rate control for connectors by @lukekim in #10648
  • Use origin label instead of name for HTTP rate control metrics by @lukekim in #10689
  • fix(http): reject OR across different HTTP filter columns by @lukekim in #10625
  • Add provider-aware LLM prompt caching by @lukekim in #10645
  • Add searchable registry mode for LLM tools by @lukekim in #10647
  • feat: refresh_mode: snapshot + SQLite/Turso WAL flush + Cayenne metastore slice by @phillipleblanc in #10651
  • feat: per-principal cache namespacing for SQL/search/caching-accelerator by @lukekim in #10702
  • Add self-hosted Spice connector support by @phillipleblanc in #10546
  • Add Delta Lake Azure tenant parameter by @phillipleblanc in #10671
  • Support OAuth2 client credentials in 'spice cloud login' by @ewgenius in #10586
  • Add configurable allowed_hosts for MCP by @lukekim in #10638
  • fix: make Helm chart probes configurable by @peasee in #10696
  • Strip high-cardinality datasets dim from anonymous telemetry by @lukekim in #10711
  • feat(elasticsearch): direct FTS engine config + index lifecycle and ingestion controls by @lukekim in #10672
  • Add DuckDB HNSW vector index support for accelerated views by @sgrebnov in #10695
  • Rewrite DuckDB vector search SQL to activate HNSW_INDEX_SCAN by @sgrebnov in #10674
  • Fix DuckDB HNSW vector indexes lost after data refresh by @sgrebnov in #10668
  • Fix DuckDB DELETE/UPDATE on full and caching refresh mode datasets by @phillipleblanc in #10632
  • Fix DuckLake connector: downcast, module registration, schema discovery, and S3 credentials by @sgrebnov in #10650
  • Fix federation pushing denied functions inside subqueries to remote engines by @phillipleblanc in #10692
  • fix(caching): honour refresh_on_startup: always in caching mode by @phillipleblanc in #10594
  • fix(iceberg): rebuild storage factory when Hadoop catalog scheme is inferred by @sgrebnov in #10601
  • Pipeline CDC ingestion: overlap source reads with batch apply by @lukekim in #10676
  • fix: add NULL check to CDC primary key extraction by @lukekim in #10684
  • Properly handle nullability during CDC processing by @krinart in #10803
  • Flatten scheduler config and rename partition management → partition assignment by @lukekim in #10450
  • Improve NSQL UX and harden internal LLM tools by @lukekim in #10715
  • Support Responses API across model providers by @lukekim in #10724
  • Update xAI default model and handle Grok model retirements by @Jeadie in #10723
  • Improve cli table layout by @krinart in #10725
  • TLS cert hot-reload (mTLS plan M1) by @phillipleblanc in #10727
  • Fix DuckLake catalog include filter being ignored by @phillipleblanc in #10738
  • Promote DuckLake Catalog and Data Connector to Beta quality by @sgrebnov in #10743
  • feat(ducklake): Support INSERT on catalog tables with read_write access by @sgrebnov in #10744
  • perf(cdc): coalesce envelopes and overlap commits in apply pipeline by @lukekim in #10745
  • feat: Allow full version tags in spicepod version by @peasee in #10748
  • Add Arrow primary key upserts by @lukekim in #10749
  • fix(snapshot): keep refresh_mode snapshot read-only by @phillipleblanc in #10752
  • feat(tls): public mTLS for HTTP and Flight (channel + identity modes) by @phillipleblanc in #10753
  • perf(cayenne): lock-free deletion caches with bloom-prefiltered probe by @lukekim in #10756
  • fix(security): close API key timing-position leak and remote-UDF SSRF by @lukekim in #10757
  • Fix 'wait_until_dependent_tables_are_ready' for catalogs by @phillipleblanc in #10758
  • Fixes for views and resolved tables on 'spice refresh' CLI by @phillipleblanc in #10759
  • Implement FlightSQL CommandStatementSubstraitPlan support by @lukekim in #10761
  • feat(connectors): mTLS client cert support for flightsql and spiceai connectors by @phillipleblanc in #10764
  • Allow arbitrary filenames when specifying spicepod path + kind validation by @krinart in #10777
  • fix: ignore field metadata in schema compatibility check in index_table_scan by @Jeadie in #10778
  • Display pushed-down limits in EXPLAIN TREE output by @lukekim in #10779
  • fix: enable streaming append for Kafka with Cayenne accelerator by @lukekim in #10780
  • fix: bound chunked-index intermediate batch size to prevent OOM by @phillipleblanc in #10783
  • fix: label all columns in spice cloud metrics table output by @claudespice in #10784
  • fix: use checked arithmetic for Turso integer-millis timestamp read path by @claudespice in #10786
  • fix: use checked arithmetic in timestamp-to-nanosecond conversions by @claudespice in #10666
  • Upgrade to DuckDB v1.5.2 by @sgrebnov in #10788
  • Improve CDC ingestion performance by @lukekim in #10789
  • Fix tool_search/tool_invoke spans by @lukekim in #10791
  • Add Cayenne inline mutations and benchmark coverage by @lukekim in #10792
  • Ensure we always resolve table names in distributed mode/metadata by @Jeadie in #10793
  • Remove permanent errors from DynamoDB Streams by @krinart in #10794
  • Add expanded view mode for wide table display in SQL REPL by @lukekim in #10797
  • Fix Cayenne CDC schema mismatch error by @sgrebnov in #10800
  • Executors should create catalog tables on join by @Jeadie in #10807
  • Add compressed file support for listing connectors by @lukekim in #10809
  • Improve Cayenne mutation, scan, and inline memtable scaling by @lukekim in #10811
  • Add range fallback for large join filters by @lukekim in #10816
  • Improve Cayenne join filter pushdown by @lukekim in #10818
  • Synchronize Cayenne partition commits across partitions by @phillipleblanc in #10819
  • fix: Deny nondistributed cayenne catalog by @peasee in #10821
  • Enable parallel Cayenne Vortex writes by @lukekim in #10822
  • Expand Arrow type handling in formatting and Elasticsearch by @lukekim in #10825
  • Add response.output_text.delta to responses API by @krinart in #10828
  • feat(cayenne): add join filter propagation and no-spill Q21 planning by @lukekim in #10840
  • Upgrade Turso to v0.6.0 by @sgrebnov in #10843
  • feat(cli): add spice feedback command to open community Slack by @lukekim in #10856
  • Upgrade iceberg to v0.9.1 by @sgrebnov in #10859
  • feat(cluster): per-request executor readiness gate on /v1/ready by @phillipleblanc in #10860
  • fix: Require dim-side statistics for CayennePropagateFilterAcrossEquiJoinKeys by @sgrebnov in #10863
  • fix: Debezium schema evolution breaks dataset init on reload by @claudespice in #10144
  • fix(mssql): Push topK limit to SQL Server for non-nullable sort columns by @Jeadie in #10621
  • fix(ScyllaDB): disable physical filter pushdown by @sgrebnov in #10772
  • fix: handle typed NULLs and prevent overflow in DynamoDB DML type conversions by @krinart in #10511
  • fix: use InsertOp::Overwrite in DynamoDB bootstrap scan_and_overwrite_accelerator by @krinart in #10639
  • Improve DynamoDB Bootstrap performance by @krinart in #10616
  • fix: preserve field and schema metadata in Vortex type transformation by @lukekim in #10628
  • fix: GH connector - explicitly use AWS LC RS crypto provider for jwt by @phillipleblanc in #10619
  • fix: add snapshot mode guards to delete_from/update and delegate DML in SwappableTableProvider by @phillipleblanc in #10685
  • Persist HTTP rate-control state in object storage by @lukekim in #10697
  • Rate limit metrics HTTP endpoint by @lukekim in #10162
  • feat(geo): add optional spatial SQL UDF support by @lukekim in #10833
  • feat(cayenne): CDC throughput, compaction, scan caching, and benchmarks by @lukekim in #10852
  • fix(cayenne): fix Vortex panic on highly compressible data by @sgrebnov in #10855
  • fix(cayenne): Read live protected snapshots after cleanup grace period by @sgrebnov in #10901
  • fix: Disable Cayenne HashJoin rewriter optimizer by @sgrebnov in #10882
  • Fix GetFlightInfo vs DoGet Flight Schema by @krinart in #10864
  • fix(search): preserve column casing in /v1/search primary key plumbing by @claudespice in #10909
  • fix(object-store): dedupe s3 url style auto-detection log by @phillipleblanc in #10898
  • Improve Spice CLI manifest editing and direct command modes by @lukekim in #10815
  • Persist Kafka CDC offsets in sidecar tables by @lukekim in #10823
  • feat(task-history): record Ballista stages for distributed queries by @phillipleblanc in #10831
  • Add '#[deny(clippy::missing_trait_methods)]' to wrapper/delegation trait impls by @Jeadie in #10795
  • Optimize Cayenne catalog maintenance paths by @lukekim in #10904
  • Centralize DuckDB settings for accelerator by @ewgenius in #10895
  • deps(ballista): bump to 47e2b494 to fix S3 shuffle reads under cluster mode by @phillipleblanc in #10910
  • Authorization header + Bump async-openai + responses_adapter fix by @krinart in #10911
  • Tune accelerators by storage profile by @lukekim in #10913
  • feat: add dataset-level on_schema_change config by @lukekim in #10908
  • Handle NULL sentinel for nullable partition expressions by @Jeadie in #10880
  • fix: Remove Cayenne Catalog from catalog registration by @peasee in #10914
  • Add catalog name to foreign key metadata in postgres catalog by @Jeadie in #10917
  • Cayenne perf: eliminate redundant clones, PK point-lookup fanout fix, IN-list rewrite + microbench coverage by @lukekim in #10916
  • fix(turso-shared): retry on Turso BEGIN CONCURRENT "Write-write conflict" by @lukekim in #10946
  • Vendor Vortex DataFusion for Cayenne by @lukekim in #10933
  • perf(cayenne): background retention + enable CDC pipelining for retention-configured tables by @lukekim in #10936
  • feat(cayenne): scale metastore pool to 32 + vs_duckdb_scaling benches (1→128 concurrency, sqlite + turso lanes) by @lukekim in #10943
  • feat(mcp): support auth for streamable HTTP tools by @phillipleblanc in #10927
  • Explicit error if v1/search requests a table without search index by @Jeadie in #10968
  • Fix spicepod loading failure when directory name contains dots by @sgrebnov in #10958
  • Extend append tests with arrow engine configurations by @sgrebnov in #10959
  • Remove dataset on_schema_change Policy from rc.5 release notes by @sgrebnov in #10964
  • Skip tpcds_q78 for Cayenne engine at SF100 by @sgrebnov in #10966
  • fix: Update benchmark snapshots May-20 by @app/github-actions in #10952
  • Fix #10951: UdtfExec invariant Vec lengths must match children count by @phillipleblanc in #10953
  • docs(release): update v2.0.0-rc.5 notes with latest trunk PRs by @lukekim in #10949
  • Remove eval related things for v2.0.0 by @Jeadie in #10945
  • build(deps): bump ubuntu from 24.04 to 26.04 in the docker-dependencies group by @app/dependabot in #10883
  • fix: Add publish = false to chbench-driver by @sgrebnov in #10939

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.4...v2.0.0-rc.5

Spice v1.11.6 (May 6, 2026)

· 2 min read
Viktor Yershov
Viktor Yershov
Senior Software Engineer at Spice AI

Spice v1.11.6 is now available! 🛠️

Spice v1.11.6 is a patch release improving DynamoDB bootstrap ingestion performance and making Helm chart health probes configurable.

What's New in v1.11.6​

DynamoDB: Faster Bootstrap Ingestion​

Improved DynamoDB bootstrap ingestion throughput for refresh_mode: changes datasets (#10616). Large tables now maintain a high, steady ingestion rate throughout the entire bootstrap, even for very large tables.

Helm Chart: Configurable Health Probes​

The Spice Helm chart now supports configurable livenessProbe, readinessProbe, and startupProbe settings (#10696), including probe path, port, and timing parameters.

Example values.yaml:

...
livenessProbe:
  httpGet:
    path: /health
    port: 8090
  timeoutSeconds: 1
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /v1/ready
    port: 8090
  timeoutSeconds: 5
  periodSeconds: 10
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /health
    port: 8090
  timeoutSeconds: 1
  periodSeconds: 10
  failureThreshold: 30

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.6, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.6 image:

docker pull spiceai/spiceai:1.11.6

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.6

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.5...v1.11.6

Spice v2.0-rc.4 (Apr 30, 2026)

· 22 min read
William Croxson
William Croxson
Senior Software Engineer at Spice AI

Announcing the release of Spice v2.0-rc.4! 🚀

v2.0.0-rc.4 is the fourth release candidate for advanced testing of v2.0, building on v2.0.0-rc.3.

Highlights in this release candidate include:

  • Elasticsearch Data Connector (Alpha) with native hybrid search (BM25 full-text + kNN vector + RRF)
  • PostgreSQL Native CDC via WAL logical replication, eliminating the need for Debezium or Kafka
  • Multi-vector Embeddings with MaxSim for ColBERT-style late-interaction retrieval
  • Rerank UDTF for hybrid search pipelines with automatic query propagation
  • HashiCorp Vault and Azure Key Vault Secret Stores for enterprise secret management
  • DuckDB Vector Engine with HNSW index support
  • Azure Cosmos DB Connector (RC), Git Connector promoted to RC
  • MCP Streamable HTTP transport
  • Read-only API Key Enforcement on Flight DoGet and async query paths

What's New in v2.0.0-rc.4​

Elasticsearch Data Connector (Alpha, Spice.ai Enterprise)​

The new Elasticsearch data connector enables querying Elasticsearch indexes as SQL tables with full hybrid search support. Currently available in Spice.ai Enterprise.

Key capabilities:

  • SQL Table Access: Query any Elasticsearch index with standard SQL via a native DataFusion TableProvider.
  • kNN Vector Search: Use the vector_search() UDTF against Elasticsearch-backed vector fields.
  • BM25 Full-Text Search: Use the text_search() UDTF for native Elasticsearch full-text queries.
  • Hybrid Search: Combine kNN and BM25 results with the rrf() UDTF for reciprocal rank fusion.
  • Elasticsearch as a Vector Engine: Accelerated datasets can use Elasticsearch as the backing vector engine for embedding storage and retrieval.

Example configuration:

datasets:
  - from: elasticsearch:my_index
    name: my_data
    params:
      elasticsearch_endpoint: https://my-cluster.es.io:9200
      elasticsearch_username: ${secrets:es_user}
      elasticsearch_password: ${secrets:es_password}

PostgreSQL Native Replication via WAL​

Postgres datasets configured with refresh_mode: changes can now stream changes directly from PostgreSQL logical replication (WAL) into any local accelerator without Debezium or Kafka required.

Key capabilities:

  • Native Logical Replication: Uses pgoutput decoding to stream INSERT/UPDATE/DELETE events.
  • Automatic Slot Management: Each Spice replica creates a distinct replication slot (spice_<dataset>_<hash>), so multi-replica deployments work automatically. Publications are shared.
  • Bootstrap Snapshot: An initial REPEATABLE READ snapshot seeds the accelerator before replication begins.
  • LSN Acknowledgement: The LsnCommitter sends durable LSN back to Postgres so WAL segments are reclaimed.
  • All Accelerators Supported: Works with DuckDB, SQLite, Postgres, Cayenne, and Arrow accelerators.

Example configuration:

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_db: mydb
      pg_publication: my_publication   # optional; auto-created if omitted
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: changes

Multi-vector Embeddings with MaxSim (Late Interaction)​

Column-level embeddings now support list-of-string columns, producing one embedding vector per list element and enabling ColBERT-style late-interaction retrieval.

Key capabilities:

  • Multi-vector per Row: Columns of type List<String> produce List<FixedSizeList<F32, D>> — one embedding per list element.
  • MaxSim / Mean / Sum Scoring: Per-row score is the max, mean, or sum cosine over the list elements. Default is MaxSim (ColBERT).
  • _match Column: Returns the specific list element that produced the highest cosine similarity.
  • No Schema Changes Required: Works with existing embedding configurations; activates automatically for list-type columns.

A new rerank() table-valued function reorders scored results from vector_search, text_search, or rrf by a reranker model's relevance judgements. See Search Functionality for an overview of search UDTFs.

Key capabilities:

  • Auto Query Propagation: The query string is automatically inherited from a nested search UDTF — no repetition required.
  • Any Chat Model as Reranker: Any registered chat/completion model can serve as a reranker via the built-in LlmRerank adapter (listwise prompt by default; pointwise available).
  • Filter and Projection Pushdown: The RerankExec physical node supports pushdown, reducing data movement.
  • Extensible: A new RerankerModelStore sits alongside ChatModelStore and EmbeddingModelStore; native providers (Cohere, Voyage, BGE) can be added without runtime plumbing changes.
SELECT * FROM rerank(
    rrf(vector_search('my_table', 'query text'), text_search('my_table', 'query text')),
    document => content
) LIMIT 10;

New Secret Stores: HashiCorp Vault and Azure Key Vault​

Two new enterprise-grade Secret Stores are now available.

HashiCorp Vault (hashicorp_vault):

  • KV v2 (default) and KV v1 mount support.
  • Auth methods: token, approle, kubernetes, jwt.
  • Token leases are cached and automatically re-acquired on expiry.
secrets:
  - from: hashicorp_vault:secret/my-app
    name: my_secrets
    params:
      hashicorp_vault_addr: https://vault.example.com
      hashicorp_vault_auth_method: approle
      hashicorp_vault_role_id: ${env:VAULT_ROLE_ID}
      hashicorp_vault_secret_id: ${secrets:vault_secret_id}

Azure Key Vault (azure_keyvault):

  • Per-key caching with single-flight fetch coalescing.
  • Auth methods: service principal, managed identity, workload identity, Azure CLI, or auto-detect.
  • Supports sovereign clouds via endpoint parameter.
secrets:
  - from: azure_keyvault:my-vault
    name: my_secrets
    params:
      azure_keyvault_auth_method: managed_identity

DuckDB Vector Engine​

DuckDB-accelerated tables can now use DuckDB's HNSW index for vector search via the vector_engine: duckdb option, enabling fast approximate nearest-neighbor search without an external vector store.

Example configuration:

datasets:
  - from: postgres:public.documents
    name: documents
    columns:
      - name: content
        embeddings:
          - from: hf_minilm
            row_id: id
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
    vectors:
      enabled: true
      engine: duckdb
      params:
        duckdb_distance_metric: cosine
        duckdb_hnsw_m: 16
        duckdb_hnsw_ef_construction: 64
        duckdb_hnsw_ef_search: 32

embeddings:
  - from: huggingface:huggingface.co/minishlab/potion-base-2M
    name: hf_minilm

New and Promoted Connectors​

Azure Cosmos DB (Alpha):

A new read-only Azure Cosmos DB NoSQL / Core SQL API connector built on the azure_data_cosmos 0.30 SDK. Supports cross-partition scans, schema inference from document samples, and key-based auth (connection string or account endpoint + key).

Git Connector (RC):

The Git data connector is promoted to RC status with HTTPS/SSH auth (git_token, git_username/git_password, git_ssh_key), Git LFS support (enable_lfs), and per-repo connection resilience (semaphore, bounded retries with exponential backoff, permanent-error circuit breaking).

DynamoDB Write Support (DML)​

DynamoDB datasets now support write-back via INSERT, UPDATE, and DELETE operations, complementing the existing read and CDC streaming capabilities.

MCP Streamable HTTP Transport​

The MCP server has been upgraded to rmcp 1.5.0 and switched to the Streamable HTTP transport (/v1/mcp), replacing the previous SSE-based endpoint. The client-side transport is updated to StreamableHttpClientTransport.

Security Improvements​

Read-only API Key Enforcement: API keys with read-only scope are now strictly enforced on the Flight DoGet path and on async query endpoints, preventing write operations from being issued under a read-only key.

GitHub Workflow Hardening: CI workflows have been hardened with improved security posture to reduce supply-chain risk.

Developer Experience Improvements​

  • Actionable Config Errors: Parameter typos, missing secret references, and unknown engine names now produce specific, actionable error messages with Levenshtein-based suggestions, rather than silent drops or generic "missing required parameter" messages.
  • spice init Improvements: Written spicepods now include a yaml-language-server: $schema=... directive for IDE completions. Creation messages print regardless of log level.
  • REPL Improvements: Log filter honors RUST_LOG when -v is not passed; version banner moves to stderr and prints only on an interactive TTY.
  • 403 / 401 Routing: HTTP 403 responses route to a new PermissionDenied variant; 401 messages point at spice login / SPICE_API_KEY.

OpenTelemetry Improvements​

See Observability & Monitoring and the runtime.telemetry reference for full configuration details.

  • Metric Name Prefix: Configure a prefix for all exported OTLP metric names via runtime.telemetry.metric_prefix.
  • Delta Temporality Default: The OTLP push exporter now defaults to delta temporality, matching Prometheus and most backends.
  • Resource Attributes: runtime.telemetry.properties are applied as OTLP resource attributes on exported metrics.

Full-text Search Performance​

Tantivy full-text search ingestion performance is significantly improved with better batch handling and a rollback-on-error path.

SQL and Query Engine​

  • DataFusion Upgrade: Updated to a newer DataFusion revision with additional bug fixes and performance improvements.
  • Views on DDL Catalogs: DDL-defined catalogs (e.g., Unity Catalog) can now expose and query views.
  • flatten_json / json_tree / expand_maps UDTFs: New table-valued functions for JSON transformation, map expansion, and schema decomposition in query pipelines. See JSON Functions and Operators.
  • cosine_distance Pushdown to DuckDB: cosine_distance is now pushed down to DuckDB accelerators via array_cosine_distance.
  • Snowflake Type Support: Added support for OBJECT, MAP, GEOGRAPHY, GEOMETRY, VECTOR, and TIMESTAMP_LTZ types in the Snowflake connector.
  • MySQL Zero-Date Behavior: The MySQL connector adds a new mysql_zero_date_behavior parameter (null or error) controlling how MySQL zero-date values (0000-00-00) are handled.
  • Databricks Timeouts: The Databricks connector adds new connect_timeout and client_timeout parameters for sql_warehouse mode.

Dependency Updates​

Dependency / ComponentVersion / Update
DataFusionUpdated
rmcpv1.5.0 (from fork pin)
mistral.rsUpdated
openssl0.10.78

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.0-rc.4, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.4

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.4 image:

docker pull spiceai/spiceai:2.0.0-rc.4

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.4

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.3...v2.0.0-rc.4

Spice v2.0-rc.3 (Apr 21, 2026)

· 13 min read
Evgenii Khramkov
Evgenii Khramkov
Senior Software Engineer at Spice AI

Announcing the release of Spice v2.0-rc.3! ⚡

v2.0.0-rc.3 is the third release candidate for advanced testing of v2.0, building on v2.0.0-rc.2.

Highlights in this release candidate include:

  • HTTP Connector Enhancements with OAuth2 refresh-token authentication, query-parameter pagination, and map-to-array conversion for broader API compatibility
  • Databricks and Unity Catalog Reliability Improvements with resilience controls, improved UC-awareness, permission checks, and structured error reporting
  • Snowflake and ADBC Registration Performance Improvements with better observability during dataset registration
  • OpenTelemetry Exporter Improvements with exporter fixes and support for authenticated metrics export
  • Kafka, GitHub, and HTTP Connector Fixes including Kafka reliability improvements, GitHub GraphQL resilience updates, and HTTP JSON union/reload fixes

What's New in v2.0.0-rc.3​

HTTP Connector Enhancements​

The HTTP connector now supports more authentication and API response patterns, making it easier to integrate with modern REST APIs.

Key improvements:

  • OAuth2 Refresh-Token Authentication: Added support for OAuth2 refresh-token flows for APIs that issue short-lived access tokens.
  • Query-Parameter Pagination: Added pagination support using query parameters for APIs that expose page or cursor controls in the URL.
  • Map-to-Array Conversion: Added response transformation support for APIs that return map-shaped payloads that need to be normalized into arrays.
  • Improved JSON Union Handling: Better handling for heterogeneous JSON payloads during ingestion.
  • More Reliable Reloads: Fixed runtime behavior for HTTP-backed datasets during spicepod reloads.

Example configuration of an HTTP connector using the OAuth2 refresh token flow:

datasets:
  - from: https://api.example.com
    name: secure_data
    params:
      file_format: json
      allowed_request_paths: '/v1/**'
      auth_token_url: https://auth.example.com/oauth/token
      http_auth_refresh_token: ${secrets:my_refresh_token}
      http_auth_client_id: ${secrets:my_client_id}
      http_auth_client_secret: ${secrets:my_client_secret}
      auth_scopes: 'read:data offline_access'

Databricks and Unity Catalog Reliability Improvements​

Databricks and Unity Catalog integrations are now more resilient and provide clearer behavior in permission-constrained environments.

Key improvements:

  • Resilience Controls: Added controls to improve reliability when interacting with Databricks services.
  • Unity Catalog Awareness: Improved handling for Unity Catalog-specific behaviors and mixed deployment configurations.
  • Permission Prechecks: Databricks UC permission checks now distinguish explicit denials from ambiguous cases.
  • Structured Error Reporting: Advisory permission failures now surface with more actionable structured errors.
  • Classic SQL Warehouse Compatibility: Improved handling for foreign table scenarios with Classic SQL Warehouse combinations.
  • Task History Instrumentation: Added instrumentation to improve observability for Databricks-related operations.

Snowflake and ADBC Improvements​

Snowflake and ADBC-backed dataset registration is now faster and easier to observe.

Key improvements:

  • Faster Dataset Registration: Improved registration performance for Snowflake and ADBC datasets.
  • Better Observability: Added better instrumentation and visibility into registration workflows.
  • ADBC Alignment: Updated ADBC dependencies and integration points for improved compatibility.
  • Search Schema Fix: Fixed a full-text search schema mismatch issue with the ADBC connector.

OpenTelemetry and Observability Improvements​

Spice improves telemetry export reliability and authenticated metrics delivery for the OpenTelemetry integration.

Key improvements:

  • OTEL Exporter Fixes: Fixed issues in the OpenTelemetry exporter.
  • Authenticated Metrics Export: Added support for authorization headers in the OTEL metrics exporter.
  • Reduced Startup Noise: Suppressed unnecessary AWS SDK noise and improved OTEL-related initialization behavior.
  • Connector Initialization Reliability: Fixed issues that could block connector initialization in telemetry-related code paths.

Dependency and Toolchain Updates​

Dependency / ComponentVersion / Update
Rust toolchainv1.94.1 (from v1.93.1)
DataFusionv52.5.0-rc1
mistral.rs / candlemistral.rs v0.8.x, candle v0.10.1
ADBC Corev0.23

Other Improvements​

  • Improved Query Pushdown: Expanded sort and limit pushdown, including improved pushdown behavior for Oracle and MSSQL connectors.
  • Partitioned Query Planning Improvements: Improved PartitionedTableScanRewrite handling for ORDER BY, partition expressions, and fully qualified table references, while preventing incorrect bucketing partition pushdown to executors.
  • MongoDB SRV Support: Upgraded datafusion-table-providers with MongoDB SRV support.
  • Tantivy Logging: Search logging now defaults to warn unless very verbose logging is enabled.
  • Kafka Connector Fixes: Improved reliability for the Kafka connector behavior.
  • GitHub Connector Resilience: Improved commit fetching for dynamic and slash refs, and reduced GraphQL page sizes on gateway errors for the GitHub connector.
  • GitHub API Efficiency: Lowered default comment fetch sizes to reduce pressure on GitHub GraphQL APIs.
  • Embedding Validation: Added validation for embedding row_id columns during dataset initialization.
  • View Cache Invalidation: Cached plans are now cleared when views are updated.
  • Refresh SQL Dedup Fix: Fixed append refresh deduplication when refresh_sql selects a subset of columns.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.0-rc.3, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.3

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.3 image:

docker pull spiceai/spiceai:2.0.0-rc.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.3

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • fix: Full Text Search schema mismatch with ADBC connector by @lukekim in #10235
  • docs: Update v2.0.0-rc.2 release notes with latest changes by @lukekim in #10238
  • Fix append refresh dedup failure when refresh_sql selects column subset by @sgrebnov in #10225
  • Revert "Properly mark dataset as Ready on Scheduler (#10215)" by @sgrebnov in #10242
  • Fix failing merge conflicts for benchmarks by @krinart in #10247
  • fix(github): fetch commits for dynamic and slash refs by @lukekim in #10233
  • Upgrade DataFusion to v52.5.0-rc1 by @lukekim in #10249
  • Merge develop to trunk (2026-04-09) by @claudespice in #10248
  • fix: Validate embedding row_id columns during dataset init (fixes #8226) by @claudespice in #10208
  • fix: Update tpch benchmark snapshots for federated/glue[csv].yaml by @app/github-actions in #10244
  • feat(databricks): add resilience controls, UC awareness, and task history instrumentation by @lukekim in #10246
  • fix: Make PartitionManager resilient to bare vs fully qualified table references by @sgrebnov in #10257
  • fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-cayenne[file].yaml by @app/github-actions in #10256
  • Merge develop to trunk (2026-04-10) by @claudespice in #10251
  • Improve Snowflake/ADBC dataset registration performance and observability by @lukekim in #10266
  • Fixes for kafka connector by @krinart in #10263
  • fix(runtime): gate otel code tags, suppress aws sdk noise, and unblock connector init by @lukekim in #10260
  • fix(runtime): avoid regionless AWS SDK loads by @lukekim in #10271
  • Add versioned release install workflow coverage by @lukekim in #10276
  • fix(runtime): handle HTTP JSON unions and spicepod reloads by @lukekim in #10277
  • Databricks UC permission prechecks: explicit denial as permanent error, ambiguous cases advisory by @lukekim in #10274
  • Revert component status changes re-introduced by develop merge (#10248) by @sgrebnov in #10293
  • Fix broken CI workflows by @ewgenius in #10294
  • Group dependabot updates by ecosystem by @lukekim in #10296
  • fix(tests): Replace flaky S3 Vectors snapshot tests with structural validation by @lukekim in #10301
  • Update test_github_workflows snapshot by @lukekim in #10304
  • fix(ci): fix Bedrock runner mismatch and snapshot auto-merge failure by @ewgenius in #10306
  • feat(http): Add map-to-array conversion and query-parameter pagination by @lukekim in #10295
  • New crate: datafusion-ddl by @Jeadie in #10205
  • Make Databricks UC permission checks advisory with structured error reporting by @lukekim in #10283
  • build(deps): bump the github-actions-dependencies group with 4 updates by @app/dependabot in #10298
  • fix: Clear cached plans on view updates by @peasee in #10312
  • build(deps): bump the aws-sdk group with 7 updates by @app/dependabot in #10299
  • Code out of runtime. by @Jeadie in #10178
  • fix: Respect function registry denies for accelerated table filter pushdown by @peasee in #10311
  • fix: Don't block heartbeat when all slots acquired by @peasee in #10322
  • fix: strip only outer parens in get_table_partition_expr_from_ctx by @Jeadie in #10323
  • Upgrade datafusion-table-providers with MongoDB SRV support by @lukekim in #10317
  • fix: Avoid pushing down bucketing partition expressions into executors by @peasee in #10324
  • Upgrade datafusion-table-providers to d1b911a5 and bump adbc to 0.23 by @lukekim in #10329
  • fix: Update Search integration test snapshots by @app/github-actions in #10308
  • Handle foreign table + Classic sql warehouse combination gracefully by @krinart in #10318
  • New crate datafusion-flightsql by @Jeadie in #10201
  • Set tantivy=warn unless very verbose logging by @Jeadie in #10338
  • Remove image registry and image name options from spidapter by @ewgenius in #10241
  • build(deps): bump sysinfo from 0.37.2 to 0.38.4 by @app/dependabot in #10291
  • build(deps): bump futures from 0.3.31 to 0.3.32 by @app/dependabot in #10289
  • New crate 'datafusion-dml' by @Jeadie in #10334
  • Jeadie/26 04 16/spice sql by @Jeadie in #10343
  • Add Teraswitch/Pittsburgh apt mirrors + retry config for CI runners by @lukekim in #10349
  • Implement sort pushdown and fix pushdown gaps across providers by @lukekim in #10337
  • Merge develop to trunk (2026-04-16) by @claudespice in #10345
  • Update candle and mistral.rs lock-step pins by @lukekim in #10278
  • docs: fix status badges in README by @lukekim in #10350
  • Migrate secrets to vars by @krinart in #10354
  • Add limit pushdown and improve sort pushdown for Oracle and MSSQL by @sgrebnov in #10351
  • Fix ubuntu mirror configuration by @ewgenius in #10359
  • fix: Increase throughput test default ready_wait from 30s to 300s (fixes #8207) by @claudespice in #10344
  • Add auth headers support to OTEL metrics exporter by @lukekim in #10347
  • fix(github): shrink GraphQL page size on gateway errors; lower comment defaults by @lukekim in #10355
  • Relax apt mirror substitution failure to warning in CI action by @ewgenius in #10361
  • feat(http): Add OAuth2 refresh-token auth to HTTP connector by @lukekim in #10348
  • Upgrade Rust toolchain to 1.94.1 by @lukekim in #10353
  • Handle order by and sort in PartitionedTableScanRewrite by @Jeadie in #9656
  • Fix OTEL Exporter by @krinart in #10363
  • Pin spiceai candle / TEI forks to merged revs; drop local [patch] overrides by @lukekim in #10362

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.2...v2.0.0-rc.3

Spice v2.0-rc.2 (Apr 10, 2026)

· 28 min read
Evgenii Khramkov
Evgenii Khramkov
Senior Software Engineer at Spice AI

Announcing the release of Spice v2.0-rc.2! 🔥

v2.0.0-rc.2 is the second release candidate for advanced testing of v2.0, building on v2.0.0-rc.1.

Highlights in this release candidate include:

  • Distributed Spice Cayenne Query and Write Improvements with data-local query routing and partition-aware write-through
  • DataFusion v52.4.0 Upgrade with aligned arrow-rs, datafusion-federation, and datafusion-table-providers
  • MERGE INTO for Spice Cayenne catalog tables with distributed support across executors
  • PARTITION BY Support for Cayenne enabling SQL-defined partitioning in CREATE TABLE statements
  • ADBC Data Connector & Catalog with full query federation, BigQuery support, and schema/table discovery
  • Databricks Lakehouse Federation Improvements with improved reliability, resilience, DESCRIBE TABLE fallback, and source-native type parsing
  • Delta Lake Column Mapping supporting Name and Id mapping modes
  • HTTP Pagination support for paginated API endpoints in the HTTP data connector
  • New Catalog Connectors for PostgreSQL, MySQL, MSSQL, and Snowflake
  • JSON Ingestion Improvements with single-object support, soda (Socrata Open Data) format support, json_pointer extraction, and auto-detection
  • Per-Model Rate-Limited AI UDF Execution for controlling concurrent AI function invocations
  • Dependency upgrades including Turso v0.5.3, iceberg-rust v0.9, and Vortex improvements

What's New in v2.0.0-rc.2​

Distributed Cayenne Query and Write Improvements​

Distributed query for Cayenne-backed tables now has better partition awareness for both reads and writes.

Key improvements:

  • Data-Local Query Routing: Cayenne catalog queries can now be routed to executors that hold the relevant partitions, improving distributed query efficiency.
  • Partition-Aware Write Through: Scheduler-side Flight DoPut ingestion now splits partitioned Cayenne writes and forwards them to the responsible executors instead of routing through a single raw-forward path.
  • Dynamic Partition Assignment: Newly observed partitions can be added and assigned atomically as data arrives, with persisted partition metadata for future routing.
  • Better Cluster Coordination: Partition management is now separated for accelerated and federated tables, improving routing behavior for distributed Cayenne catalog workloads.
  • Distributed UPDATE/DELETE DML: UPDATE and DELETE statements for Cayenne catalog tables are now forwarded to all executors in distributed mode, with all executors required to succeed.
  • Distributed runtime.task_history: Task history is now replicated across the distributed cluster for observability.
  • RefreshDataset Control Stream: Dataset refresh operations are now distributed via the control stream to executors.
  • Executor DDL Sync: When an executor connects, it receives DDL for all existing tables, ensuring late-joining executors have full table state.

MERGE INTO for Spice Cayenne​

Spice now supports MERGE INTO statements for Cayenne catalog tables, enabling upsert-style data operations with full distributed support.

Key improvements:

  • MERGE INTO Support: Execute MERGE INTO statements against Cayenne catalog tables for combined insert/update/delete operations.
  • Distributed MERGE: MERGE operations are automatically distributed across executors in cluster mode.
  • Data Safety: Duplicate source keys are detected and prevented to avoid data loss during MERGE operations.
  • Chunked Delete Filters: Large MERGE delete filter lists are chunked to prevent stack overflow with Vortex IN-list expressions.

PARTITION BY Support for Cayenne​

SQL Partition Management: Spice now supports PARTITION BY for Cayenne-backed CREATE TABLE statements, enabling partition definitions to be expressed directly in SQL and persisted in the Cayenne catalog.

Key improvements:

  • SQL Partition Definition: Define Cayenne table partitioning directly in SQL using CREATE TABLE ... PARTITION BY (...).
  • Partition Validation: Partition expressions are parsed and validated during DDL analysis before table creation.
  • Persisted Partition Metadata: Partition metadata is stored in the Cayenne catalog and can be reloaded by the runtime after restart.
  • Distributed DDL Support: Partition metadata is forwarded when CREATE TABLE is distributed to executors in cluster mode.
  • Improved Type Support: Partition utilities now support newer string scalar variants such as Utf8View.

Example:

CREATE TABLE events (id INT, region TEXT, ts TIMESTAMP) PARTITION BY (region)

Catalog Connector Enhancements​

Spice now includes additional catalog connectors for major database systems, improving schema discovery and federation workflows across external data systems.

Key improvements:

  • New Catalog Connectors: Added catalog connectors for PostgreSQL, MySQL, MSSQL, and Snowflake.
  • Schema and Table Discovery: Connectors use native metadata catalogs such as information_schema / INFORMATION_SCHEMA to discover schemas and tables.
  • Improved Federation Workflows: These connectors make it easier to expose external database metadata through Spice for cross-system federation scenarios.
  • PostgreSQL Partitioned Tables: Fixed schema discovery for PostgreSQL partitioned tables.

Example PostgreSQL catalog configuration:

catalogs:
  - from: pg
    name: pg
    include:
      - 'public.*'
    params:
      pg_host: localhost
      pg_port: 5432
      pg_user: postgres
      pg_pass: ${secrets:POSTGRES_PASSWORD}
      pg_db: my_database
      pg_sslmode: disable

JSON Ingestion Improvements​

JSON ingestion is now more flexible and robust.

Key improvements:

  • More JSON Formats: Added support for single-object JSON documents, auto-detected JSON formats, and Socrata SODA responses.
  • json_pointer Extraction: Extract nested payloads before schema inference and reading using RFC 6901 JSON Pointer syntax.
  • Better Auto-Detection: JSON format detection now handles arrays, objects, JSONL, and BOM-prefixed input more reliably, including single multi-line objects.
  • SODA Support: Added schema extraction and data conversion for Socrata Open Data API responses.
  • Broader Compatibility: Improved handling for BOM-prefixed files, CRLF-delimited JSONL, nested payloads, mixed structures, and wrapped documents.

Example using json_pointer to extract nested data from an API response:

datasets:
  - from: https://api.example.com/v1/data
    name: users
    params:
      json_pointer: /data/users

DataFusion v52.4.0 Upgrade​

Apache DataFusion has been upgraded from v52.2.0 to v52.4.0, with aligned updates across arrow-rs, datafusion-federation, and datafusion-table-providers.

Key improvements:

  • DataFusion v52.4.0: Brings the latest fixes and compatibility improvements across query planning and execution.
  • Strict Overflow Handling: try_cast_to now uses strict cast to return errors on overflow instead of silently producing NULL values.
  • Federation Fix: Fixed SQL unparsing for Inexact filter pushdown with aliases.
  • Partial Aggregation Optimization: Improved partial aggregation performance for FlightSQLExec.

Dependency Upgrades​

DependencyVersion / Update
Turso (libsql)v0.5.3 (from v0.4.4)
iceberg-rustv0.9
VortexMap type support, stack-safe IN-lists
arrow-rsArrow v57.2.0
datafusion-federationUpdated for DataFusion v52.4.0 alignment
datafusion-table-providersUpdated for DataFusion v52.4.0 alignment
datafusion-ballistaBumped to fix BatchCoalescer schema mismatch panic

Other Improvements​

  • Cayenne released as RC: Cayenne data accelerator is now promoted to release candidate status.

  • File Update Acceleration Mode: Added mode: file_update acceleration mode for file-based data refresh.

  • spice completions Command: New CLI command for generating shell completion scripts, with auto-detection of shell directory.

  • --endpoint Flag: Added --endpoint flag to spice run with scheme-based routing for custom endpoints.

  • mTLS Client Auth: Added mTLS client authentication support to the spice sql REPL.

  • DynamoDB DML: Implemented DML (INSERT, UPDATE, DELETE) support for the DynamoDB table provider.

  • Caching Retention: Added retention policies for cached query results.

  • GraphQL Custom Auth Headers: Added custom authorization header support for the GraphQL connector.

  • ClickHouse Date32 Support: Added Date32 type support for the ClickHouse connector.

  • AWS IAM Role Source: Added iam_role_source parameter for fine-grained AWS credential configuration.

  • S3 Metadata Columns: Metadata columns renamed to _location, _last_modified, _size for consistency, with more robust handling in projected queries.

  • S3 URL Style: Added s3_url_style parameter for S3 connector URL addressing (path-style vs virtual-hosted). Useful for S3-compatible stores like MinIO:

    params:
      s3_endpoint: https://minio.local:9000
      s3_url_style: path

  • S3 Parquet Performance: Improved S3 parquet read performance.

  • HTTP Caching: Transient HTTP error responses such as 429 and 5xx are no longer cached, preventing stale error payloads from being served from cache.

  • HTTP Connector Metadata: Added response_headers as structured map data for HTTP datasets.

  • Views on_zero_results: Accelerated views now support on_zero_results: use_source to fall back to the source when no results are found:

    views:
      - name: sales_summary
        sql: |
          SELECT region, SUM(amount) as total
          FROM sales
          GROUP BY region
        acceleration:
          enabled: true
          on_zero_results: use_source

  • Flight DoPut Ingestion Metrics: Added rows_written and bytes_written metrics for Flight DoPut / ADBC ETL ingestion.

  • EXPLAIN ANALYZE Metrics: Added metrics for EXPLAIN ANALYZE in FlightSQLExec.

  • Scheduler Executor Metrics: Added scheduler_active_executors_count metric for monitoring active executors.

  • Query Memory Limit: Updated default query memory limit from 70% to 90%, with GreedyMemoryPool for improved memory management.

  • MetastoreTransaction Support: Added transaction support to prevent concurrent metastore transaction conflicts.

  • Iceberg REST Catalog: Coerce unsupported Arrow types to Iceberg v2 equivalents in the REST catalog API.

  • CDC Cache Invalidation: Improved cache invalidation for CDC-backed datasets.

  • Spice.ai Connector Alignment: Parameter names aligned across catalog and data connectors for Spice.ai Cloud.

  • Cayenne File Size: Cayenne now correctly respects the configured target file size (defaults to 128MB).

  • Cayenne Primary Keys: Properly set primary_keys/on_conflict for Cayenne tables.

  • Turso Metastore Performance: Cached metastore connections and prepared statements for improved Turso and SQLite metastore performance.

  • Turso SQL Robustness: More robust SQL unparsing and date comparison handling for Turso.

  • Dictionary Type Normalization: Normalize Arrow Dictionary types for DuckDB and SQLite acceleration.

  • GitHub Connector Resilience: Improved GraphQL client resilience, performance, and ref filter handling.

  • ODBC Fix: Fixed ODBC queries silently returning 0 rows on query failure.

  • Anthropic Fixes: Fixed compatibility issues with Anthropic model provider.

  • v1/responses API Fix: The /v1/responses API now correctly preserves client instructions when system_prompt is set.

  • Shared Acceleration Snapshots: Show an error when snapshots are enabled on a shared acceleration file.

  • Distributed Mode Error Handling: Improved error handling for distributed mode and state_location configuration.

  • Helm Chart: Added support for ServiceAccount annotations and AWS IRSA example.

  • Perplexity Removed: Removed Perplexity model provider support.

  • Rust v1.93.1: Upgraded Rust toolchain to v1.93.1.

Contributors​

Breaking Changes​

  • S3 metadata columns renamed: S3 metadata columns renamed from location, last_modified, size to _location, _last_modified, _size.
  • v1/evals API removed: The /v1/evals endpoint has been removed.
  • Perplexity removed: Perplexity model provider support has been removed.
  • Default query memory limit changed: Default query memory limit increased from 70% to 90%.

Upgrading​

To upgrade to v2.0.0-rc.2, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.2

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.2 image:

docker pull spiceai/spiceai:2.0.0-rc.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.2

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • ci: fix E2E CLI upgrade test to use latest release for spiced download by @phillipleblanc in #9613
  • fix(DF): Lazily initialize BatchCoalescer in RepartitionExec to avoid schema type mismatch by @sgrebnov in #9623
  • feat: Implement catalog connectors for various databases by @lukekim in #9509
  • Refactor and clean up code across multiple crates by @lukekim in #9620
  • fix: Improve error handling for distributed mode and state_location configuration by @lukekim in #9611
  • Properly install postgres in install-postgres action by @krinart in #9629
  • fix: Use Python venv for schema validation in CI by @phillipleblanc in #9637
  • Update spicepod.schema.json by @app/github-actions in #9640
  • Update testoperator dispatch to use release/2.0 branch by @phillipleblanc in #9641
  • fix: Align CUDA asset names in Dockerfile and install tests with build output by @phillipleblanc in #9639
  • Fix expect test scripts in E2E Installation AI test by @sgrebnov in #9643
  • testoperator for partitioned arrow accelerator by @Jeadie in #9635
  • Remove default 1s refresh_check_interval from spidapter for hive datasets by @phillipleblanc in #9645
  • Fix scheduler panic and cancel race condition by @phillipleblanc in #9644
  • Align Spice.ai connector parameter names across catalog/data connectors by @lukekim in #9632
  • docs: update distribution details and add NAS support in release notes by @lukekim in #9650
  • Enable postgres-accel in CI builds for benchmarks by @sgrebnov in #9649
  • perf: Cache Turso metastore connection across operations by @penberg in #9646
  • Add 'scheduler_state_location' to spidapter by @Jeadie in #9655
  • Implement Cayenne S3 Express multi-zone live test with data validation by @lukekim in #9631
  • chore(spidapter): bump default memory limit from 8Gi to 32Gi by @phillipleblanc in #9661
  • perf: Use prepare_cached() in Turso and SQLite metastore backends by @penberg in #9662
  • Improve CDC cache invalidation by @krinart in #9651
  • Refactor Cayenne IDs to use UUIDv7 strings by @lukekim in #9667
  • fix: add liveness check for dead executors in partition routing by @Jeadie in #9657
  • fix(s3): Fix metadata column schema mismatches in projected queries by @sgrebnov in #9664
  • s3_metadata_columns tests: include test for location outside table prefix by @sgrebnov in #9676
  • docs: Update DuckDB, GCS, Git connector and Cayenne documentation by @lukekim in #9671
  • Add s3_url_style support for S3 connector URL addressing by @phillipleblanc in #9642
  • Consolidate E2E workflows and require WSL for Windows runtime by @lukekim in #9660
  • Upgrade to Rust v1.93.1 by @lukekim in #9669
  • Security fixes and improvements by @lukekim in #9666
  • feat(flight): add DoPut rows/bytes written metrics for DoPut ETL ingestion tracking by @phillipleblanc in #9663
  • Skip caching http error response + add response_headers by @krinart in #9670
  • refactor: Remove v1/evals functionality by @Jeadie in #9420
  • Make a test harness for Distributed Spice integration tests by @Jeadie in #9615
  • Enable on_zero_results: use_source for views by @krinart in #9699
  • fix(spidapter): Lower memory limit, passthrough AWS secrets, override flight URL by @peasee in #9704
  • Show an error on a shared acceleration file with snapshots enabled by @krinart in #9698
  • Fixes for anthropic by @Jeadie in #9707
  • Use max_partitions_per_executor in allocate_initial_partitions by @Jeadie in #9659
  • [SpiceDQ] Accelerations must have partition key by @Jeadie in #9711
  • Upgrade to Turso v0.5 by @lukekim in #9628
  • feat: Rename metadata columns to _location, _last_modified, _size by @phillipleblanc in #9712
  • fix: bump datafusion-ballista to fix BatchCoalescer schema mismatch panic by @phillipleblanc in #9716
  • fix: Ensure Cayenne respects target file size by @peasee in #9730
  • refactor: Make DDL preprocessing generic from Iceberg DDL processing by @peasee in #9731
  • [SpiceDQ] Distribute query of Cayenne Catalog to executors with data by @Jeadie in #9727
  • Properly set primary_keys/on_conflict for Cayenne tables by @krinart in #9739
  • Add executor resource and replica support to cloud app config by @ewgenius in #9734
  • feat: Support PARTITION BY in Cayenne Catalog table creation by @peasee in #9741
  • Update datafusion and related packages to version 52.3.0 by @lukekim in #9708
  • Route FlightSQL statement updates through QueryBuilder by @phillipleblanc in #9754
  • JSON file format improvements by @lukekim in #9743
  • [SpiceDQ] Partition Cayenne catalogs writes through to executors by @Jeadie in #9737
  • Update to DF v52.3.0 versions of datafusion & datafusion-tableproviders by @lukekim in #9756
  • Make S3 metadata column handling more robust by @sgrebnov in #9762
  • Fetch API keys from dedicated endpoint instead of apps response by @phillipleblanc in #9767
  • Update arrow-rs, datafusion-federation, and datafusion-table-providers dependencies by @phillipleblanc in #9769
  • Chunk metastore batch inserts to respect SQLite parameter limits by @phillipleblanc in #9770
  • Improve JSON SODA support by @lukekim in #9795
  • Add ADBC Data Connector by @lukekim in #9723
  • docs: Release Cayenne as RC by @peasee in #9766
  • cli[feat]: cloud mode to use region-specific endpoints by @lukekim in #9803
  • Include updated JSON formats in HTTPS connector by @lukekim in #9800
  • Flight DoPut: Partition-aware write-through forwarding by @Jeadie in #9759
  • Pass through authentication to ADBC connector by @lukekim in #9801
  • Move scheduler_state_location from adapter metadata to env var by @phillipleblanc in #9802
  • Fix Cayenne DoPut upsert returning stale data after 3+ writes by @phillipleblanc in #9806
  • Fix JSON column projection producing schema mismatch by @sgrebnov in #9811
  • Fix http connector by @krinart in #9818
  • Fix ADBC Connector build and test by @lukekim in #9813
  • Support update & delete DML for distributed cayenne catalog by @Jeadie in #9805
  • Set allow_http param when S3 endpoint uses http scheme by @phillipleblanc in #9834
  • fix: Cayenne Catalog DDL requires a connected executor in distributed mode by @Jeadie in #9838
  • fix: Add conditional put support for file:// scheduler state location by @Jeadie in #9842
  • fix: Require the DDL primary key contain the partition key by @Jeadie in #9844
  • fix: Databricks SQL Warehouse schema retrieval with INLINE disposition and async retry by @lukekim in #9846
  • Filter pushdown improvements for SqlTable by @lukekim in #9852
  • feat: add iam_role_source parameter for AWS credential configuration by @lukekim in #9854
  • Fix ODBC queries silently returning 0 rows on query failure by @lukekim in #9864
  • feat(adbc): Add ADBC catalog connector with schema/table discovery by @lukekim in #9865
  • Make Turso SQL unparsing more robust and fix date comparisons by @lukekim in #9871
  • Fix Flight/FlightSQL filter precedence and mutable query consistency by @lukekim in #9876
  • Partial Aggregation optimisation for FlightSQLExec by @lukekim in #9882
  • fix: v1/responses API preserves client instructions when system_prompt is set by @Jeadie in #9884
  • feat: emit scheduler_active_executors_count and use it in spidapter by @Jeadie in #9885
  • feat: Add custom auth header support for GraphQL connector by @krinart in #9899
  • Add --endpoint flag to spice run with scheme-based routing by @lukekim in #9903
  • When executor connects, send DDL for existing tables by @Jeadie in #9904
  • fix: Improve ADBC driver shutdown handling and error classification by @lukekim in #9905
  • fix: require all executors to succeed for distributed DML (DELETE/UPDATE) forwarding by @Jeadie in #9908
  • fix(cayenne catalog): fix catalog refresh race condition causing duplicate primary keys by @Jeadie in #9909
  • Remove Perplexity support by @Jeadie in #9910
  • Fix refresh_sql support for debezium constraints by @krinart in #9912
  • Implement DML for DynamoDBTableProvider by @lukekim in #9915
  • chore: Update iceberg-rust fork to v0.9 by @lukekim in #9917
  • Run physical optimizer on FallbackOnZeroResultsScanExec fallback plan by @sgrebnov in #9927
  • Improve Databricks error message when dataset has no columns by @sgrebnov in #9928
  • Delta Lake: fix data skipping for >= timestamp predicates by @sgrebnov in #9932
  • fix: Ensure distributed Cayenne DML inserts are forwarded to executors by @Jeadie in #9948
  • Add full query federation support for ADBC data connector by @lukekim in #9953
  • Make time_format deserialization case-insensitive by @vyershov in #9955
  • Hash ADBC join-pushdown context to prevent credential leaks in EXPLAIN plans by @lukekim in #9956
  • fix: Normalize Arrow Dictionary types for DuckDB and SQLite acceleration by @sgrebnov in #9959
  • ADBC BigQuery: Improve BigQuery dialect date/time and interval SQL generation by @lukekim in #9967
  • Make BigQueryDialect more robust and add BigQuery TPC-H benchmark support by @lukekim in #9969
  • fix: Show proper unauthorized error instead of misleading runtime unavailable by @lukekim in #9972
  • fix: Enforce target_chunk_size as hard maximum in chunking by @lukekim in #9973
  • Add caching retention by @krinart in #9984
  • fix: improve Databricks schema error detection and messages by @lukekim in #9987
  • fix: Set default S3 region for opendal operator and fix cayenne nextest by @phillipleblanc in #9995
  • fix(PostgreSQL): fix schema discovery for PostgreSQL partitioned tables by @sgrebnov in #9997
  • fix: Defer cache size check until after encoding for compressed results by @krinart in #10001
  • fix: Rewrite numeric BETWEEN to CAST(AS REAL) for Turso by @lukekim in #10003
  • fix: Handle integer time columns in append refresh for all accelerators by @sgrebnov in #10004
  • fix: preserve s3a:// scheme when building OpenDalStorageFactory with custom endpoint by @phillipleblanc in #10006
  • Fix ISO8601 time_format with Vortex/Cayenne append refresh by @sgrebnov in #10009
  • fix: Address data correctness bugs found in audit by @sgrebnov in #10015
  • fix(federation): fix SQL unparsing for Inexact filter pushdown with alias by @lukekim in #10017
  • Improve GitHub connector ref handling and resilience by @lukekim in #10023
  • feat: Add spice completions command for shell completion generation by @lukekim in #10024
  • fix: Fix data correctness bugs in DynamoDB decimal conversion and GraphQL pagination by @sgrebnov in #10054
  • Implement RefreshDataset for distributed control stream by @Jeadie in #10055
  • perf: Improve S3 parquet read performance by @sgrebnov in #10064
  • fix: Prevent write-through stalls and preserve PartitionTableProvider during catalog refresh by @Jeadie in #10066
  • feat: spice completions auto-detects shell directory and writes file by @lukekim in #10068
  • fix: Bug in DynamoDB, GraphQL, and ISO8601 refresh data handling by @sgrebnov in #10063
  • fix partial aggregation deduplication on string checking by @lukekim in #10078
  • fix: add MetastoreTransaction support to prevent concurrent transaction conflicts by @phillipleblanc in #10080
  • fix: Use GreedyMemoryPool, add spidapter query memory limit arg by @phillipleblanc in #10082
  • feat: Add metrics for EXPLAIN ANALYZE in FlightSQLExec by @lukekim in #10084
  • Use strict cast in try_cast_to to error on overflow instead of silent NULL by @sgrebnov in #10104
  • feat: Implement MERGE INTO for Cayenne catalog tables by @peasee in #10105
  • feat: Add distributed MERGE INTO support for Cayenne catalog tables by @peasee in #10106
  • Improve JSON format auto-detection for single multi-line objects by @lukekim in #10107
  • Add mode: file_update acceleration mode by @krinart in #10108
  • Coerce unsupported Arrow types to Iceberg v2 equivalents in REST catalog API by @peasee in #10109
  • fix: Update default query memory limit to 90% from 70% by @phillipleblanc in #10112
  • feat: Add mTLS client auth support to spice sql REPL by @lukekim in #10113
  • fix(datafusion-federation): report error on overflow instead of silent NULL by @sgrebnov in #10124
  • fix: Prevent data loss in MERGE when source has duplicate keys by @peasee in #10126
  • feat: Add ClickHouse Date32 type support by @sgrebnov in #10132
  • Add Delta Lake column mapping support (Name/Id modes) by @sgrebnov in #10134
  • fix: Restore Turso numeric BETWEEN rewrite lost in DML revert by @lukekim in #10139
  • fix: Enable arm64 Linux builds with fp16 and lld workarounds by @lukekim in #10142
  • fix: remove double trailing slash in Unity Catalog storage locations by @sgrebnov in #10147
  • fix: Improve GitHub GraphQL client resilience and performance by @lukekim in #10151
  • Enable reqwest compression and optimize HTTP client settings by @lukekim in #10154
  • fix: executor startup failures by @Jeadie in #10155
  • feat: Distributed runtime.task_history support by @Jeadie in #10156
  • fix: Preserve timestamp timezone in DDL forwarding to executors by @peasee in #10159
  • feat: Per-model rate-limited concurrent AI UDF execution by @Jeadie in #10160
  • fix(Turso): Reject subquery/outer-ref filter pushdown in Turso provider by @lukekim in #10174
  • Fix linux/macos spice upgrade by @phillipleblanc in #10194
  • Improve CREATE TABLE LIKE error messages, success output, EXPLAIN, and validation by @peasee in #10203
  • fix: chunk MERGE delete filters and update Vortex for stack-safe IN-lists by @peasee in #10207
  • Propagate runtime.params.parquet_page_index to Delta Lake connector by @sgrebnov in #10209
  • Properly mark dataset as Ready on Scheduler by @Jeadie in #10215
  • fix: handle Utf8View/LargeUtf8 in GitHub connector ref filters by @lukekim in #10217
  • fix(databricks): Fix schema introspection and timestamp overflow by @lukekim in #10226
  • fix(databricks): Fix schema introspection failures for non-Unity-Catalog environments by @lukekim in #10227
  • feat: Add pagination support to HTTP data connector by @lukekim in #10228
  • feat(databricks): DESCRIBE TABLE fallback and source-native type parsing for Lakehouse Federation by @lukekim in #10229
  • fix(databricks): harden HTTP retries, compression, and token refresh by @lukekim in #10232
  • feat[helm chart]: Add support for ServiceAccount annotations and AWS IRSA example by @peasee in #9833
  • fix: Log warning and fall back gracefully on Cayenne config change by @krinart in #9092
  • fix: Handle engine mismatch gracefully in snapshot fallback loop by @krinart in #9187

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.1...v2.0.0-rc.2

Spice v1.11.5 (Apr 1, 2026)

· 4 min read
Sergei Grebnov
Sergei Grebnov
Senior Software Engineer at Spice AI

Announcing the release of Spice v1.11.5! 🛠️

Spice v1.11.5 is a patch release improving on_zero_results: use_source fallback performance, Delta Lake timestamp predicate data skipping, S3 Parquet read performance, PostgreSQL partitioned table support, Cayenne target file size handling, and preparing the CLI for v2.0 runtime upgrades.

What's New in v1.11.5​

on_zero_results: use_source Fallback Performance Improvement​

Improved the on_zero_results: use_source fallback path to run DataFusion's physical optimizer on the federated scan plan (#9927). The fallback path now runs SessionState::physical_optimizers() rules on the federated scan plan before execution, enabling parallel file group scanning and other optimizations. This results in significantly faster fallback queries on multi-core machines, particularly for file-based data sources like Delta Lake.

Delta Lake: Improved Data Skipping for >= Timestamp Predicates​

Delta Lake table scans with >= timestamp filters now correctly prune files that do not match the predicate (#9932), improving query performance through more effective data skipping (file-level pruning).

PostgreSQL: Partitioned Tables Support​

The PostgreSQL data connector now supports partitioned tables (#9997) for both federated and accelerated queries.

S3 Parquet Read Performance Improvement​

Improved parquet read performance from S3 and other object stores (#10064), particularly for tables with many columns. Column data ranges are now coalesced into fewer, larger requests instead of being fetched individually, reducing the number of HTTP round-trips.

Cayenne: Ensure Target File Size is Respected​

The Cayenne accelerator now correctly respects the configured target file size (#10071). Previously, Cayenne could produce many small, fragmented Vortex files; with this fix, files are written at the expected target size, improving storage efficiency and query performance.

CLI: Support for v2.0 Runtime Upgrades​

The Spice CLI can now upgrade to v2.0 runtime versions. This enables upgrading to v2.0 release candidates and, once released, the v2.0 stable runtime.

spice upgrade v2.0.0-rc.1

Running spice upgrade without a version will upgrade to the latest stable version, including v2.0 once released.

Note: Native Windows runtime builds will no longer be provided in v2.0. Use WSL for local development instead.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.5, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.5 image:

docker pull spiceai/spiceai:1.11.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.5

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • fix(runtime): Run physical optimizer on FallbackOnZeroResultsScanExec fallback plan by @sgrebnov in #9927
  • fix(delta_lake): Fix data skipping for >= timestamp predicates by @sgrebnov in #9932
  • fix(PostgreSQL): Fix schema discovery for PostgreSQL partitioned tables by @sgrebnov in #9997
  • fix(cli): Skip models variant download for v2+ in upgrade/install by @lukekim and @sgrebnov in #10052
  • perf(s3): Improve Parquet read performance by @sgrebnov in #10064
  • fix(cayenne): Ensure Cayenne respects target file size by @krinart in #10071

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.4...v1.11.5

Spice v1.11.4 (Mar 12, 2026)

· 5 min read
Sergei Grebnov
Sergei Grebnov
Senior Software Engineer at Spice AI

Announcing the release of Spice v1.11.4! ⚡

Spice v1.11.4 is a patch release improving S3 metadata column query robustness and enabling on_zero_results: use_source for accelerated views.

What's New in v1.11.4​

Accelerated Views: on_zero_results: use_source Support​

Accelerated views now support the on_zero_results: use_source configuration (#9699). Previously, accelerated views only supported on_zero_results: return_empty, which returned an empty result set when the accelerated data contained no matching rows. With this change, views can fall back to querying the source data when the accelerated query returns zero results, matching the behavior already available for accelerated datasets.

Example configuration:

views:
  - name: sales_summary
    sql: |
      SELECT region, SUM(amount) as total
      FROM sales
      GROUP BY region
    acceleration:
      enabled: true
      on_zero_results: use_source

How the Fallback Works​

When an accelerated view is configured with on_zero_results: use_source, the following happens at query time:

  1. The accelerated store is queried first. The query runs against the view's accelerated data (e.g., Spice Cayenne, Arrow, DuckDB, or SQLite).

  2. If the accelerated query returns zero rows, the runtime falls back to re-executing the view's SQL query against the datasets it references.

  3. Referenced datasets are queried according to their own configuration. The view's SQL is re-executed against each referenced dataset as it is configured. This means:

    • If a referenced dataset is accelerated, the query hits that dataset's accelerated store — not the raw data source.
    • If a referenced dataset is accelerated with on_zero_results: use_source and its accelerated store also returns zero rows, it will independently fall back to its own federated data source (e.g., Postgres, S3, etc.).
    • If a referenced dataset is federated (not accelerated), the query goes directly to the data source.

This means the fallback can chain through multiple layers: first the view's acceleration, then each referenced dataset's acceleration, and finally the original data source — each layer independently applying its own on_zero_results behavior.

Example: Multi-layer fallback

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      refresh_sql: "SELECT * FROM orders WHERE created_at > now() - interval '7 days'"
      on_zero_results: use_source  # Falls back to Postgres if accelerated data has no matches

views:
  - name: recent_orders_summary
    sql: |
      SELECT status, COUNT(*) as order_count
      FROM orders
      GROUP BY status
    acceleration:
      enabled: true
      on_zero_results: use_source  # Falls back to re-running the SQL against referenced datasets

In this example, a query like SELECT * FROM recent_orders_summary WHERE status = 'cancelled' follows this path:

  1. Queries recent_orders_summary in the view's accelerated store (DuckDB/SQLite).
  2. If zero rows are returned, re-executes SELECT status, COUNT(*) ... FROM orders GROUP BY status against the orders dataset.
  3. Since orders is accelerated, the query hits the orders accelerated store.
  4. If orders also returns zero rows (e.g., the refresh_sql excluded cancelled orders), it falls back to querying Postgres directly.

S3 Data Connector: More Robust Metadata Column Handling​

Improved the robustness of metadata column (location, last_modified, size) handling for S3 datasets. Building on the v1.11.3 release, this update addresses an additional edge case where the query optimizer's projection swap could cause an index out of bounds panic when metadata columns are referenced in projections with filters or scalar functions.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.4, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.4 image:

docker pull spiceai/spiceai:1.11.4

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.4

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • fix(s3): Make metadata column handling more robust by @sgrebnov in #9714
  • feat(views): Enable on_zero_results: use_source for accelerated views by @krinart in #9699

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.3...v1.11.4

Spice v1.11.3 (Mar 9, 2026)

· 3 min read
Phillip LeBlanc
Phillip LeBlanc
Co-Founder and CTO of Spice AI

Announcing the release of Spice v1.11.3! 🛠️

Spice v1.11.3 is a patch release fixing schema consistency issues in the S3 and FlightSQL data connectors, improving CDC cache invalidation, and enhancing the HTTP data connector's error handling and response metadata.

What's New in v1.11.3​

S3 Data Connector Fix​

Fixed an issue where queries using metadata columns (location, last_modified, size) on S3 datasets produced Input field name does not match with the projection expression errors (#9647). This occurred when projecting metadata columns with filters or scalar functions (e.g., SELECT lower(location) FROM table WHERE location = '...'), and when projection returned no matching files.

FlightSQL Schema Consistency​

Fixed an issue where the Flight SQL JDBC driver returned Unsupported ArrowType Utf8View errors when performing ::TEXT type casts (#9253). The FlightSQL endpoint now maps view types (e.g., Utf8View, BinaryView) to their non-view equivalents, ensuring compatibility with JDBC and ODBC clients.

CDC Cache Invalidation​

Fixed an issue where the SQL results cache was invalidated on every change stream poll, even when zero records were returned (#9472). This caused near-total cache miss rates for datasets using refresh_mode: changes (e.g., DynamoDB Streams), effectively rendering the cache useless. Cache invalidation now only occurs when a change batch contains actual data changes.

HTTP Data Connector Improvements​

  • HTTP error responses (e.g., 5xx) are now excluded from the cache, preventing transient server errors from polluting cached results.
  • Added a response_headers column (Map type) to HTTP responses, providing access to response header metadata in query results.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.3, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.3 image:

docker pull spiceai/spiceai:1.11.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.3

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • fix(s3): Fix metadata column schema mismatches in projected queries by @sgrebnov in #9664
  • s3_metadata_columns tests: include test for location outside table prefix by @sgrebnov in #9676
  • Fix Flight SQL schema consistency: expand view types and verify field names by @sgrebnov in #9438
  • Improve CDC cache invalidation by @krinart in #9651
  • Skip caching http error response + add response_headers by @krinart in #9670

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.2...v1.11.3

Spice v2.0-rc.1 (Mar 4, 2026)

· 23 min read
Sergei Grebnov
Sergei Grebnov
Senior Software Engineer at Spice AI

Announcing the release of Spice v2.0-rc.1! 🚀

v2.0.0-rc.1 is the first release candidate for early testing of v2.0.

Highlights in this release candidate include:

  • Active-Active Highly-Available Distributed Query that is object-store-native and built on Apache Ballista, with dynamic cluster sizing, distributed ingestion, and cluster observability
  • Spice Cayenne RC with staged append writes, file-based retention deletes, composite partitioning, and distributed ingestion
  • DataFusion v52.2.0 Upgrade with sort pushdown, a new merge join, and dynamic filters
  • DDL Support for CREATE TABLE and DROP TABLE via SQL for Iceberg and Cayenne catalogs
  • DuckLake Catalog & Data Connector for lakehouse-style data management
  • GCS Data Connector (Alpha) for Google Cloud Storage
  • Rust CLI Rewrite for a unified single-binary experience
  • Dependency upgrades including DuckDB v1.4.4, delta_kernel v0.18.2, and mistral.rs

Spice v2.0 includes several breaking changes. Review the breaking changes section before upgrading.

Distribution Changes​

AI/ML support including local LLM/ML model and hosted LLM inference is now included in the default Spice build and image. The separate models build variant has been removed.

With models now included by default, the data-only distribution (without AI/ML support) is only published in nightly builds. Official production-ready data-only distributions are available exclusively through Spice Cloud and the Enterprise release.

A new Network Attached Storage (NAS) distribution with built-in SMB and NFS data connector support is also now available in nightly builds and with Spice.ai Enterprise.

Distribution / VariantOpen SourceSpice CloudEnterprise
Default✅✅✅
DataNightly only✅✅
NAS (SMB + NFS)Nightly only❌✅
Metal (macOS)✅✅✅
CUDA (Linux)Nightly only✅✅
Allocator variantsNightly only✅✅
ODBC connectorLocal build only✅✅

For more details, see the Distributions documentation.

What's New in v2.0.0-rc.1​

Active-Active HA Distributed Query​

Distributed Query exits Beta with active-active highly-available object-store-based distributed query.

Distributed query supports two execution modes:

  • Synchronous: Queries for accelerated datasets are distributed across executors and results are streamed back in real-time. Non-accelerated datasets execute only on the scheduler. Best for interactive queries where low latency is critical.
  • Asynchronous: Queries are submitted via the new HTTP-only /v1/queries API and results are materialized to object storage for later retrieval. Best for long-running analytical workloads, batch processing, and non-accelerated datasets in distributed mode.

Key improvements:

  • Dynamic Cluster Sizing: The query planner automatically adjusts parallelism based on the number of active executors in the cluster, ensuring optimal resource utilization as nodes are added or removed.
  • Distributed Ingestion: Data ingestion for partitioned accelerated tables is now distributed across executor nodes, enabling higher throughput and parallel data loading in cluster mode. Regular (non-partitioned) accelerated tables do not distribute ingestion loads.
  • Synchronous Execution on Scheduler: /v1/sql and FlightSQL queries now execute synchronously on the scheduler when appropriate, reducing inter-node overhead for queries that don't benefit from distribution.
  • Faster Failure Detection: Executor heartbeat timeout reduced from 180s to 30s, enabling the cluster to quickly detect and respond to executor failures.
  • Cluster Observability: New metrics and Grafana dashboard for monitoring distributed query clusters.

Spice Cayenne Improvements​

The Spice Cayenne data accelerator exits Beta with significant reliability and performance improvements:

  • Staged Append Writes: WAL-based staged append writes prevent partial writes and data loss on stream errors. Batches are written to a WAL file before being committed, ensuring atomicity.
  • File-Based Retention Deletes: Time-based retention now supports file-level deletes for both position-based and primary-key tables, reducing I/O overhead compared to row-level deletion.
  • Multiple Partition Expressions: Support for composite partitioning with partition_by: [col1, col2] using hierarchical path-like keys (e.g., 2025/10/15).
  • Distributed Ingestion: Cayenne catalog now supports distributed ingestion across executor nodes in cluster mode, including UPDATE operations.
  • Improved Robustness: Fixed CDC edge case where DELETE + UPSERT sequences could produce duplicate primary keys across protected snapshots. Improved upsert handling during runtime restarts.

DataFusion v52.2.0 Upgrade​

Apache DataFusion has been upgraded to v52.2.0, bringing significant performance improvements, new query features, and enhanced extensibility.

Performance Improvements:

  • Faster CASE Expressions: Lookup-table-based evaluation for certain CASE expressions avoids repeated evaluation, accelerating common ETL patterns
  • MIN/MAX Aggregate Dynamic Filters: Queries with MIN/MAX aggregates now create dynamic filters during scan to prune files and rows as tighter bounds are discovered during execution
  • New Merge Join: Rewritten sort-merge join (SMJ) operator with speedups of three orders of magnitude in pathological cases (e.g., TPC-H Q21: minutes → milliseconds)
  • Caching Improvements: New statistics cache for file metadata avoids repeatedly recalculating statistics, significantly improving planning time. A prefix-aware list-files cache accelerates evaluating partition predicates for Hive partitioned tables
  • Improved Hash Join Filter Pushdown: Build-side hash map contents are now passed dynamically to probe-side scans for pruning files, row groups, and individual rows

Major Features:

  • Sort Pushdown to Scans: Sorts are pushed into data sources, enabling ~30x performance improvement on pre-sorted data with top-K queries. Parquet scans now reverse row group order for DESC queries on ASC-sorted files
  • TableProvider supports DELETE and UPDATE: New hooks for DELETE and UPDATE statements in the TableProvider trait, enabling Iceberg and Cayenne connectors to implement SQL DELETE and UPDATE operations
  • More Extensible SQL Planning: New RelationPlanner API for extending SQL planning for FROM clauses, enabling support for vendor-specific SQL dialects

DDL Support for Iceberg and Cayenne​

SQL Schema Management: Spice now supports CREATE TABLE and DROP TABLE DDL operations for Iceberg and Cayenne catalogs via FlightSQL and the /v1/sql API. DML validation has been updated for catalog-level writability.

DuckLake Catalog & Data Connector​

Lakehouse-Style Data Management: New DuckLake catalog and data connector enable lakehouse-style data management with DuckDB as the metadata catalog and object storage for data files. DuckLake provides ACID transactions, time travel, and schema evolution on top of Parquet files.

GCS Data Connector (Alpha)​

Google Cloud Storage Support: New Google Cloud Storage data connector enables federated queries against data stored in GCS buckets, with Iceberg table support.

Rust CLI Rewrite​

Unified Single-Binary Experience: The Spice CLI has been completely rewritten from Go to Rust, eliminating the Go dependency and providing a single spice binary built from the same codebase as spiced. This improves startup performance, reduces distribution size, and ensures consistent behavior between CLI and runtime.

Key Features:

  • Full Feature Parity: All 27+ CLI commands re-implemented in Rust with identical behavior
  • New spice query Command: Interactive REPL for async queries via the /v1/queries API with multi-line SQL input, spinner progress indicator, Ctrl+C cancellation, and partial query ID matching
  • --output=json Flag: Machine-readable JSON output for CLI commands, enabling scripting and automation
  • spice login --output: New output modes (env, json, keychain) for flexible credential management
  • spice cloud metrics: New command for Spice Cloud deployment metrics

Models Included by Default​

Local LLM/ML model inference (via mistral.rs) is now included in the default Spice build. The separate models build variant has been removed. This simplifies installation and ensures all users have access to local AI inference capabilities.

Error Propagation for Dataset and Model Status APIs​

The /v1/datasets and /v1/models APIs now return structured error information when a component is in an Error state. The ?status=true query parameter must be passed to retrieve the real-time component status, including the error state and details. Previously, the status field only indicated Error with no further detail. Now, two new fields are included when ?status=true is specified:

  • error: A structured object with category, type, and code fields for programmatic error handling (e.g. { "category": "dataset", "type": "auth", "code": "dataset.auth" }).
  • error_message: A human-readable description of why the component entered an error state.

These fields are only present when ?status=true is passed and the component is in an error state.

Example /v1/datasets?status=true response:

[
  {
    "from": "postgres:syncs",
    "name": "daily_journal",
    "replication_enabled": false,
    "acceleration_enabled": true,
    "status": "Ready"
  },
  {
    "from": "databricks:hive_metastore.default.messages",
    "name": "messages",
    "replication_enabled": false,
    "acceleration_enabled": true,
    "status": "Error",
    "error": {
      "category": "dataset",
      "type": "auth",
      "code": "dataset.auth"
    },
    "error_message": "Unable to authenticate with datasource credentials"
  }
]

The spice datasets and spice models CLI commands now include an ERROR column that displays the error message for any component in an error state.

Additional Dependency Upgrades​

DependencyVersion
Ballistav52.0.0
DuckDBv1.4.4
delta_kernelv0.18.2
mistral.rsv0.7.0 (candle fork removed, now uses candle 0.9.2 from crates.io)
Turso (libsql)v0.4.4
VortexUpgraded with CASE-WHEN support
AWS SDKMultiple crates updated + APN user-agent support

Other Improvements​

  • Spicepod v2 Support: Spicepods now support version v2, and spice init generates spicepod.yaml files with version: v2 by default while maintaining backward compatibility for existing v1 spicepods.
  • x.ai Models: x.ai models now exclusively use the /v1/responses endpoint with rate limiting support.
  • HuggingFace Chat Templates: Added support for chat templates in HuggingFace model configurations.
  • Databricks SQL Dialect: Added Databricks SQL dialect for DataFusion unparser, improving federation query generation.
  • Snowflake: Added snowflake_private_key parameter for key-pair authentication.
  • Acceleration Metrics: New rows_written, bytes_written, and dataset_acceleration_size_bytes metrics for acceleration refresh ingestion.
  • Refresh SQL UDFs: Core scalar UDFs are now enabled in refresh SQL expressions.
  • FlightSQL: Fixed TLS connection handling for grpc+tls:// endpoints with custom CA certificate support.
  • FlightSQL: Fixed schema consistency by expanding view types and verifying field names.
  • Hash Index: Fixed query correctness when hash index is used with additional filters.
  • Results Cache: Fixed schema preservation for empty query results.
  • Query Nullability: Reconciled execution stream nullability with logical plan schema.
  • Schema Evolution: Graceful handling of schema evolution mismatch errors during data refresh.
  • Internal YAML Parser: Replaced deprecated serde_yaml with an internal YAML implementation.

Spicepod v1 to v2 Changes​

Spicepod v2 introduces configuration improvements while maintaining backward compatibility with v1. Existing v1 spicepods continue to work — deprecated fields are automatically migrated at load time.

Version support:

VersionStatus
v2Default. Used by spice init.
v1Supported. Deprecated fields auto-migrate.
v1beta1Removed. No longer accepted.

Configuration changes:

v1 (deprecated)v2 (preferred)Notes
runtime.results_cacheruntime.caching.sql_resultsAll fields migrate automatically. cache_max_size → max_size.
runtime.memory_limitruntime.query.memory_limitAuto-migrated. query.memory_limit takes priority if both set.
runtime.temp_directoryruntime.query.temp_directoryAuto-migrated. query.temp_directory takes priority if both set.
dataset.invalid_type_actiondataset.unsupported_type_actionAuto-migrated. v2 adds a new string variant.

New v2 fields:

  • runtime.ready_state — Controls when the runtime reports ready (on_load default, or on_registration).
  • runtime.flight.do_put_rate_limit_enabled — Enable/disable FlightSQL DoPut rate limiting (default: true).
  • runtime.query.spill_compression — Compression for query spill files (e.g., lz4_frame).
  • runtime.scheduler.partition_management — Configure partition assignment interval, limits, and timeouts for distributed mode.
  • runtime.caching.sql_results.stale_while_revalidate_ttl — Serve stale cached results while revalidating in the background.
  • runtime.caching.sql_results.encoding — Cache entry compression (e.g., zstd).
  • catalog.access: read_write_create — New access mode for catalogs that support DDL operations.

Migration note: When both the deprecated v1 field and its v2 equivalent are set, the v2 field takes priority.

Contributors​

Breaking Changes​

  • Cayenne and Distributed Query exit Beta: Beta warnings have been removed from documentation and code. Both features are now considered GA-ready.
  • Models included by default: The separate models build variant has been removed. Local LLM inference is now always included.
  • Spicepod version defaults to v2: New spicepods created with spice init now default to version: v2. Existing v1 spicepods remain supported, and v1beta1 is no longer accepted.
  • Windows native builds removed: Native Windows builds are no longer provided. Use WSL for local development instead.
  • Metric renames: accelerated_refresh metrics renamed to acceleration_refresh for consistency. last_refresh_time gauge renamed to include milliseconds unit.
  • Caching config renamed: ResultsCache replaced with SQLResultsCacheConfig in configuration.
  • DuckDB parameter rename: partitioned_write_flush_threshold renamed to partitioned_write_flush_threshold_rows.
  • v1/search API: The /v1/search API now always returns an array in matches, even for single results.
  • x.ai model endpoint: x.ai models now exclusively use the /v1/responses endpoint.
  • Error messages: Error messages across S3 Vectors, ScyllaDB, Snowflake, ClickHouse, and other components have been refactored for clarity and consistency.

Cookbook Updates​

New and updated Spice Cookbook recipes:

  • Async Queries: Submit long-running queries asynchronously and retrieve results later.
  • DuckLake Catalog Connector: Use DuckLake for lakehouse-style data management with ACID transactions and time travel.

The Spice Cookbook includes 88 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.0-rc.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.1 image:

docker pull spiceai/spiceai:2.0.0-rc.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • Add TPC-DS integration tests with S3 source and PostgreSQL acceleration by @phillipleblanc in #9006
  • fix(tests): fix flaky/slow/failing unit tests by @phillipleblanc in #9009
  • fix: Update benchmark snapshots for DF51 upgrade by @app/github-actions in #9008
  • fix: add feature gate to rrf TEST_EMBEDDING_MODEL by @phillipleblanc in #9017
  • fix: features check by @phillipleblanc in #9014
  • fix: Enable Cayenne acceleration snapshots by @lukekim in #9020
  • URL table support by @lukekim in #9018
  • ScyllaDB key filter by @lukekim in #8997
  • fix: Schema mismatch when using column projection with HTTP caching by @phillipleblanc in #9021
  • Add more tests for HTTP caching with columns selection by @sgrebnov in #9025
  • HTTP cache snapshots: default to time_interval and fix snapshots_creation_policy: on_change by @sgrebnov in #9026
  • Fix duplicate snapshot creation on startup by @sgrebnov in #9029
  • Add ScyllaDB and SMB to the README table by @krinart in #9034
  • Remove waiting for runtime to be ready before creating snapshot by @krinart in #9033
  • Fix snapshot on_change policy to skip when no writes occurred by @sgrebnov in #9028
  • Release notes for release release/1.11.0-rc.2 by @krinart in #9016
  • ci: use arduino/setup-protoc for official protobuf compiler by @phillipleblanc in #9036
  • ci: install unzip on aarch64 runner for arduino/setup-protoc by @phillipleblanc in #9038
  • fix: don't fail release if upload to minio fails by @phillipleblanc in #9039
  • Add missing protoc step to setup-cc action by @krinart in #9041
  • fix: Update Search integration test snapshots by @app/github-actions in #9013
  • Fix formula_1 and codebase_community in bird-bench by @Jeadie in #9000
  • Cayenne S3 Express One Zone improvements by @lukekim in #9015
  • Add zlib1g-dev to CI by @lukekim in #9052
  • Improve validation and logging for hash indexes by @lukekim in #9047
  • Upgrade Vortex with CASE-WHEN by @lukekim in #9051
  • x.ai models now exclusively use /v1/responses endpoint by @lukekim in #9400
  • Improvements for snapshot schema comparison by @krinart in #9401
  • v2.0 breaking changes by @lukekim in #9233
  • Create PartitionManagementTask for scheduler to update accelerated table partition assignments by @Jeadie in #9378
  • refactor(Cayenne): route all write orchestration through CayenneDataSink by @sgrebnov in #9402
  • Refactor benchmark to use QueryExecutor trait by @Jeadie in #9418
  • feat: Add spidapter build and release workflow by @peasee in #9427
  • Testoperator: add support for api-key when connecting to external spice instance by @sgrebnov in #9421
  • Initial implementation of Ducklake catalog & data connectors by @lukekim in #9083
  • Require aws_lc_rs since jsonwebtoken upgrade by @Jeadie in #9426
  • feat: Add spidapter tool by @peasee in #9425
  • Add release notes for 1.11.2 patch release by @sgrebnov in #9430
  • feat(spidapter): integrate system-adapter-protocol with SCP provisioning by @phillipleblanc in #9434
  • Add DuckLake TPCH E2E workflow and federated Spicepod configuration by @lukekim in #9431
  • fix(spidapter): use Flight handshake auth instead of x-api-key header by @phillipleblanc in #9435
  • [spidapter] Keep only what sparks joy by @Jeadie in #9439
  • Refactor binary operator balancing by @Jeadie in #9424
  • feat: Add Iceberg DDL support (CREATE TABLE / DROP TABLE) for default catalog override by @phillipleblanc in #9440
  • Fix Flight SQL schema consistency: expand view types and verify field names by @sgrebnov in #9438
  • Update spidapter for new system-adapter-protocol by @sgrebnov in #9442
  • docs: fix typos and syntax errors in style guide and error handling docs by @cluster2600 in #9445
  • Add acceleration refresh ingestion metrics (rows_written, bytes_written) by @phillipleblanc in #9461
  • Refactor(Cayenne): Replace CatalogError and string based errors with Snafu errors by @sgrebnov in #9403
  • Replace deprecated claude-3-5-haiku-latest with claude-haiku-4-5 by @Jeadie in #9492
  • Fix #9481: Preserve schema in results cache for empty query results by @phillipleblanc in #9485
  • Fix partition by serializing by @Jeadie in #9474
  • query: reconcile execution stream nullability with logical plan schema by @phillipleblanc in #9486
  • initial spice-cloud-client crate and spice cloud metrics --app <app-name>. by @Jeadie in #9480
  • feat: Return dataset error message in datasets API by @peasee in #9487
  • Spicebench by @lukekim in #9447
  • build(deps): consolidate dependabot dependency updates by @phillipleblanc in #9504
  • fix(cluster): route non-partitioned accelerated tables in distributed mode by @phillipleblanc in #9508
  • Enable core scalar UDFs in refresh SQL by @sgrebnov in #9502
  • Fix metrics in Spidapter again by @Jeadie in #9497
  • fix(cluster): tolerate Completed->status propagation race in distributed query handle by @phillipleblanc in #9510
  • feat: Support distributed ingestion in cayenne catalog by @peasee in #9506
  • Fix Cayenne duplicate primary keys after DELETE + UPSERT CDC sequences by @krinart in #9494
  • fix(cluster): rewrite table scans inside subqueries for distributed execution by @phillipleblanc in #9518
  • fix: Set catalog mode to readwritecreate in spidapter by @peasee in #9519
  • Upgrade AWS SDK crates & set APN user-agent in AWS SDK credential bridge by @lukekim in #8328
  • feat(runtime): add runtime ready_state on_registration semantics by @lukekim in #9522
  • fix: Add spidapter post-setup retries by @peasee in #9526
  • Make partition discovery more robust and make initialization non-blocking by @sgrebnov in #9499
  • Make lint-rust-fix support targeted packages and features by @Jeadie in #9511
  • Handle new Cloud SCP API by @Jeadie in #9532
  • Refactor and simplify streaming benchmarks by @krinart in #9405
  • fix: ensure spidapter only increments attempts on failures by @peasee in #9534
  • feat: Support specifying app resources in spidapter by @peasee in #9536
  • test(runtime): Spice Cayenne DDL integration test by @lukekim in #9535
  • fix: Handle schema evolution mismatch errors during data refresh by @lukekim in #9527
  • fix: resolve clippy lint warnings by @phillipleblanc in #9547
  • pr-builds --tag <TAG> for build_and_release.yml by @Jeadie in #9507
  • Add --output flag to spice login with env/json/keychain modes by @Jeadie in #9541
  • Don't use 'PartitionedTableScanRewrite' in async distributed query by @Jeadie in #9548
  • feat(spidapter): add local backend mode with single executor by @phillipleblanc in #9531
  • support chat template in HF by @Jeadie in #9543
  • fix(cayenne): stream PK retention deletes and run OOM regression in CI by @phillipleblanc in #9533
  • cayenne: Staged append writes to prevent partial writes and data loss on stream error by @sgrebnov in #9491
  • AcceleratedTable::scan use FederatedTable::scan when ClusterRole::Scheduler by @Jeadie in #9550
  • Upgrade to delta-kernel-rs v0.18.2 by @lukekim in #9528
  • Run cayenne tests as part of PR CI by @sgrebnov in #9554
  • Upgrade to DataFusion v52.2.0 by @lukekim in #9419
  • Remove Snapshot Compaction + Add snapshot existence check by @krinart in #9523
  • Update dependencies by @lukekim in #9566
  • fix: Update benchmark snapshots by @app/github-actions in #9565
  • fix: Compare Cayenne table configuration on startup by @peasee in #9529
  • Make Refresh::refresh_sql more robust to alterations over time. by @Jeadie in #9549
  • fix: Update datafusion-table-providers dependency to latest revision by @lukekim in #9574
  • Unset AWS_ENDPOINT_URL when empty by @krinart in #9575
  • fix: allow BytesProcessedExec repartitioning for unordered input by @lukekim in #9540
  • Sanitize DataFusion errors by @lukekim in #9530
  • Add conditional logging for partition assignments by @Jeadie in #9577
  • use 'properly early exit on SIGTERM' by @Jeadie in #9573
  • Update datafusion to 52.2.0 by @phillipleblanc in #9582
  • Ensure we query one and only one partition per request by @Jeadie in #9416
  • feat: Add support for Spicepod version v2 by @lukekim in #9583
  • [SpiceDQ] Improve error messages; Avoid race condition on allocate_initial_partitions. by @Jeadie in #9579
  • Update ballista dependencies to latest 52.0.0 revision by @lukekim in #9581
  • Fix Databricks spark_connect mode always disabled by @phillipleblanc in #9586
  • Support partitioning in Arrow accelerator by @Jeadie in #9571
  • Fix spice query CLI response deserialization by @phillipleblanc in #9588
  • fix: Update benchmark snapshots by @app/github-actions in #9584
  • fix: Share RuntimeEnv across Cayenne read/write/delete paths for targeted list_files_cache invalidation by @sgrebnov in #9589
  • feat: Add file:// state_location support for async queries scheduler by @phillipleblanc in #9590
  • Update endgame links by @krinart in #9598

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.11.2...v2.0.0-rc.1

Spice v1.11.1 (Feb 10, 2026)

· 4 min read
Jack Eadie
Jack Eadie
Token Plumber at Spice AI

Announcing the release of Spice v1.11.1! 🛠️

v1.11.1 is a patch release improving Spice Cayenne accelerator reliability and performance, enhancing DynamoDB Streams and HTTP data connectors, and fixing issues in Federated Task History and FlightSQL.

What's New in v1.11.1​

Spice Cayenne Accelerator Improvements​

This release includes stability and performance fixes for the Spice Cayenne accelerator:

  • Row-based Deletion Logic: Refactored row-based delete operations to use per-file deletion vectors with RoaringBitmap. Deletion scans now use Vortex-native streaming with filter pushdown and project only row indices, achieving zero data I/O for delete operations.
  • Constraints & On Conflict: constraints and on_conflict configurations are now automatically inferred from federated table metadata, enabling datasets like DynamoDB to work without explicitly defining primary_key in the Spicepod.
  • Partitioned Table Deletion: Fixed an issue where DELETE operations on partitioned Cayenne tables failed.
  • Data Integrity: Fixed two issues with acceleration snapshot handling: protected snapshots are now included in conflict detection keyset scans (preventing duplicate key creation during append refresh), and snapshot cleanup no longer deletes protected snapshots.

Data Connector Improvements​

  • DynamoDB Streams: Added automatic re-bootstrapping when the stream lag exceeds DynamoDB shard retention (24h). Configurable via the new lag_exceeds_shard_retention_behavior parameter with values error (default), ready_before_load, or ready_after_load.
  • HTTP Connector: HTTP responses now include a response_status column (UInt16). 4xx responses (e.g., 404 Not Found) are treated as valid queryable data and cached normally. 5xx responses are retried with backoff, returned to the user, but excluded from the cache to prevent transient server errors from polluting cached results.

Other Improvements​

  • Reliability: Added retries for SnapshotManager operations and general snapshot reliability improvements.
  • Reliability: Fixed handling of timestamp precision mismatches in query result caching.
  • Reliability: Fixed a double projection issue in federated task history queries that caused Schema error: project index out of bounds errors in cluster mode.
  • Developer Experience: Added cookie middleware support to the FlightSQL data connector.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates. The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.1 image:

docker pull spiceai/spiceai:1.11.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • Cayenne: row-based delete logic improvements by @sgrebnov in #9237
  • Proper support for constraints/on_conflict in Cayenne Acceleration by @krinart in #9335
  • Retries for SnapshotManager by @krinart in #9334
  • fix(cayenne): Include protected snapshots in conflict detection keyset scan by @sgrebnov in #9176
  • fix(cayenne): Fix data loss by preserving protected snapshots during cleanup by @sgrebnov in #9182
  • Simplify retention filter expressions before pushdown by @sgrebnov in #9244
  • Fix test_retention_complex_sql by @sgrebnov in #9270
  • runtime: avoid double projection in federated task history by @phillipleblanc in #9326
  • feat(http): Return all HTTP responses as data, skip caching 5xx by @sgrebnov in #9313
  • Snapshots Improvements by @krinart in #9318
  • fix(caching): Handle timestamp precision mismatch and add more tests by @sgrebnov in #9315
  • DynamoDB Streams Table Rebootstrapping by @krinart in #9305
  • Fix Cayenne partitioned table deletion support by @sgrebnov in #9267
  • FlightSQL: add cookie middleware support by @phillipleblanc in #9282
  • Apply SchemaCastScanExec before applying changes in process_upsert_batch by @krinart in #9297