Skip to main content

Spice v2.4.2 (Nov 9, 2026)

ยท 15 min read
Sergei Grebnov
Member of Technical Staff at Spice AI

Spice v2.4.2 is now available! ๐Ÿ› ๏ธ

Spice v2.4.2 is a patch release that continues security hardening and fixes Snowflake key-pair authentication. Security fixes limit the nesting depth, the expanded size, and the memory use of SQL queries, and HTTP datasets refuse redirects to other internal hosts. The release also fixes several SQL, Spice Cayenne, and runtime issues.

What's New in v2.4.2โ€‹

SQL Query Limits Protect the Runtimeโ€‹

Spice now limits the nesting depth, the expanded size, and the memory use of SQL queries. A query over a limit fails with HTTP 400 on /v1/sql, or with InvalidArgument on Flight SQL. The runtime continues to serve other queries. Before this release, any client that could send SQL could stop the runtime with one query. A deeply nested query overflowed the stack, and the process stopped with fatal runtime error: stack overflow, aborting. Other queries made one function or one plan allocate memory without a limit.

The checks apply to HTTP /v1/sql, /v1/nsql, Flight SQL statements and prepared statements, Flight SQL updates, and transactions. The checks cover these shapes:

  • Long expressions. A chain of +, ||, AND, or OR terms deeper than 512 levels fails. This includes a /v1/search where condition.
  • Nested queries. Subqueries, or common table expressions (CTEs) that each read the CTE before them, fail past 64 levels.
  • CTE expansion. A query fails when its CTE references expand to more than 4096 table references. The planner makes a copy of the plan of a CTE for each reference to it.
  • Nested data types. A data type such as ARRAY<ARRAY<โ€ฆ>> fails past 32 levels. A type such as INT[][]โ€ฆ fails past 64 levels.
  • Arrow type strings. An Arrow type string, such as an arrow_cast argument or a column type that a dataset declares, fails past 64 levels.
  • Grouping sets. A GROUP BY that expands to more than 4096 grouping sets fails. CUBE of 13 expressions makes 8192 sets. 4096 is also the limit in PostgreSQL.
  • Search keywords. A /v1/search request with more than 512 keywords fails with HTTP 400.

These scalar functions now keep their output within runtime.query.memory_limit, or within a fixed cap:

  • lpad and rpad charge the whole output buffer to the memory limit. This includes an empty fill string and negative lengths. A query that does not fit fails with Resources exhausted.
  • array_repeat and list_repeat charge the real size of a list, struct, binary, boolean, or NULL element. Before this release, they charged 1 byte for each copy of such an element.
  • flatten_json_properties, json_tree, and flatten_json stop at 32 ร— max_bytes of output for one document. max_depth is limited to 128. A small JSON schema with repeated allOf or $ref entries no longer expands exponentially.
  • truncate cuts text by characters, not bytes. Non-ASCII text made the query fail with Query driver task ended unexpectedly.

Flight SQL metadata requests also have limits. GetSqlInfo info IDs and GetTables table types are limited to 1024 entries. GetTables and GetDbSchemas filter patterns are limited to 1024 bytes.

The limits are:

ShapeLimit
Expression depth, such as a chain of AND, OR, +, or ||512 levels
Query nesting: subqueries, or CTEs that read each other64 levels
Table references after CTE expansion4096
SQL data type nesting with ARRAY, STRUCT, MAP, and similar keywords32 levels
SQL data type nesting with []64 levels
Arrow type string nesting64 levels
GROUP BY grouping sets4096 (CUBE of up to 12 expressions)
/v1/search keywords512
Flight SQL GetSqlInfo info IDs and GetTables table types1024 entries
Flight SQL GetTables and GetDbSchemas filter patterns1024 bytes

CTEs that do not read each other are not limited by depth. Table references outside of CTEs do not count toward the expansion limit.

Credentials Stay out of API Responses and Logsโ€‹

GET /v1/datasets and GET /v1/catalogs no longer return credentials from the from value of a component. Before this release, any API key that could list components could read a password in a source URL, such as https://user:password@host/file.csv. This included a read-only key. The runtime now removes user:password@ from each URL. It replaces values of query keys such as password, secret, token, and sig with ***. With status=true, /v1/datasets also removes credentials from error_message. Sources with no credentials do not change.

The runtime also removes credentials from these errors and log lines:

  • ADBC connection errors. Setup errors no longer contain adbc_uri. The runtime replaces each configured secret in the driver message with ***. Before this release, the full URI went to the warning log and to /v1/datasets?status=true.
  • MCP transport errors. The error no longer contains the URL of the MCP server. A URL can contain a credential.
  • .env files. The warning for a malformed line gives only the line number, not the line content.

HTTP Datasets Refuse Redirects to Other Internal Hostsโ€‹

An http or https dataset in a file format, such as CSV or Parquet, no longer follows a redirect to an internal host. An internal host is a different host that is a loopback, private, or link-local address, or that resolves to one. The dataset fails with this error:

Refused to connect to <target>: the HTTPS data connector does not follow a redirect to a private, loopback, or link-local address

The host of the dataset itself is always allowed, also when it is internal. Redirects that stay on that host still work. SPICE_URL_TABLE_ALLOW_PRIVATE_HOSTS=true turns off the check.

Snowflake Key-Pair Authentication Fixโ€‹

Snowflake datasets with key-pair authentication now load correctly. Before this release, such a dataset stopped at Initializing with this error:

Could not automatically determine the process-level CryptoProvider from jsonwebtoken crate features.

Other Bug Fixesโ€‹

  • IS [NOT] DISTINCT FROM with AND or OR. These queries no longer return wrong boolean results or fail with an integer type error. For example, SELECT true IS DISTINCT FROM true OR true now returns true, not false. The comparison now binds before AND or OR.
  • COUNT(DISTINCT โ€ฆ) on list or struct columns. A query with two or more COUNT(DISTINCT โ€ฆ) aggregates no longer fails with Resources exhausted when one of them reads a list or struct column. Before this release, the query failed at about 50,000 rows and requested tens of GB. Each stored value held a reference to its whole input batch. This applied to federated and Cayenne queries, with or without GROUP BY.
  • Spice Cayenne file mode. At startup, a file-mode acceleration checks that the data files in its metastore exist and have the expected size. If they do not match, the acceleration reloads from the source or fails to load, and it does not serve incomplete data. Accelerations on S3 and in memory mode are not checked.
  • Queries after a burst of stale cache hits. With stale_while_revalidate_ttl set on the SQL results cache, new client queries no longer wait behind background revalidations. In a test, 600 stale hits made the next uncached query wait 321.8 seconds, and 6.3 seconds with this fix. At most one quarter of the query admission limit runs revalidations at the same time. A stale hit over that limit gets the cached result with no revalidation, and results_cache_stale_swr_count counts it. The runtime cancels a revalidation after 5 minutes and records the outcome timed_out.
  • truncate on numbers. truncate no longer returns wrong values for integers and decimals near the limits of their type. For example, truncate(100, arrow_cast(50, 'Int8')) returned 56, and the correct result is 0. A result that does not fit its type is now an error.
  • SQL scalar functions with no arguments. A from: sql scalar function with no arguments no longer fails on every call.
  • Certificate errors in the CLI. spice HTTP commands, such as status and refresh, report a TLS certificate failure as a TLS error. In v2.4.1, the release build reported it as The Spice runtime is unavailable.
  • AWS session tokens. The runtime no longer sends an empty AWS session token together with static keys. A secret that is not set gives an empty token. This applies to S3, Delta Lake, and AWS SDK connectors.

Dependency Updatesโ€‹

  • arrow-rs (Spice fork): 59.3.0 โ†’ 59.3.2. This update fixes RUSTSEC-2026-0442 in run-end-encoded array validation, and it adds the 64-level limit for Arrow type strings.
  • azure_security_keyvault_secrets: 0.10.0 โ†’ 1.0.1. This update fixes RUSTSEC-2026-0438 in the Azure Key Vault secret store. azure_identity and azure_core move to the 1.x line for that store.
  • sqlparser: 0.62.0, vendored with the precedence fix from apache/datafusion-sqlparser-rs#2436.
  • DataFusion (Spice fork): the pin adds the fixes for COUNT(DISTINCT โ€ฆ) memory and for nested SQL data types.

Breaking Changesโ€‹

Query limits. Queries and requests over the new limits fail with HTTP 400 or Flight InvalidArgument. Most of them stopped the runtime before this release, but some ran in v2.4.1. Examples are a GROUP BY CUBE of 13 expressions, a /v1/search request with more than 512 keywords, and a query whose CTEs expand to more than 4096 table references. For each limit, see SQL Query Limits Protect the Runtime.

HTTP dataset redirects. An http or https dataset in a file format fails when its server redirects to a different internal host. To allow it, set SPICE_URL_TABLE_ALLOW_PRIVATE_HOSTS=true.

JSON flattening. flatten_json, json_tree, and flatten_json_properties stop at 32 ร— max_bytes of output for one document (256 MiB by default). They return the rows up to that point, with no error. The limit is on output, so a small document with long, deeply nested keys can reach it. To get all rows, increase max_bytes.

Upgradingโ€‹

To upgrade to v2.4.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.4.2 image:

docker pull spiceai/spiceai:2.4.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.2

AWS Marketplace:

Spice is available in the AWS Marketplace.

Cookbook Updatesโ€‹

No new cookbook recipes.

The Spice Cookbook provides recipes to help you get started with Spice.

Contributorsโ€‹

What's Changedโ€‹

Changelogโ€‹

  • [release/2.4] fix(sql): honour IS [NOT] DISTINCT FROM precedence before AND/OR (backport #15935) by @lukekim in #15936
  • fix(snowflake): select the jsonwebtoken crypto provider at startup by @sgrebnov in #15988
  • fix: reject over-deep SQL expressions and search keyword lists by @phillipleblanc in #15348
  • Reject over-deep expressions before they overflow the query stack by @phillipleblanc in #15433
  • Reject over-deep query nesting before planning by @phillipleblanc in #15722
  • fix(sql): reject queries whose CTE references expand to too many table references by @phillipleblanc in #15766
  • fix: bound nested SQL data types before parsing by @phillipleblanc in #15764
  • fix(sql): limit the nesting depth of SQL data types by @phillipleblanc in #15705
  • fix(query): reject GROUP BY that expands past 4096 grouping sets before planning by @phillipleblanc in #15843
  • fix: limit Arrow type-string nesting depth on every parse path by @phillipleblanc in #15845
  • fix(datafusion): charge lpad/rpad output buffers to the query memory limit by @phillipleblanc in #15846
  • fix(runtime): reserve array_repeat output by element size by @phillipleblanc in #15885
  • fix(udfs): bound JSON function expansion and output; fix truncate overflow and non-ASCII text by @phillipleblanc in #15912
  • fix(flight): bound Flight SQL metadata request list and pattern sizes by @phillipleblanc in #15848
  • fix(http): remove URL credentials from from in /v1/datasets and /v1/catalogs by @lukekim in #15834
  • fix(adbc): keep the connection URI out of ADBC setup errors by @lukekim in #15835
  • fix(http): refuse dataset redirects to other internal hosts by @phillipleblanc in #15877
  • [2.4] fix(cayenne): don't serve a file-mode acceleration whose data files no longer match its metastore by @lukekim in #15943
  • [release/2.4]fix(datafusion): charge each nested COUNT(DISTINCT) value for its own bytes (2.4) by @lukekim in #15986
  • fix(cache): bound stale-while-revalidate background revalidations and give them a deadline by @phillipleblanc in #15820
  • fix(cache): stop a timed-out stale-while-revalidate query that still waits for admission by @phillipleblanc in #15880
  • fix(cli): use rustls for the HTTP client so certificate errors are recognized (2.4 backport of #15811) by @lukekim in #15984
  • fix(deps): bump spiceai/arrow-rs to upstream 59.3.2 (2.4) by @lukekim in #16041
  • fix(deps): bump azure_security_keyvault_secrets to 1.0.1 on release/2.4 (RUSTSEC-2026-0438) by @lukekim in #16043
  • [release/2.4] fix(deps): replace temporary datafusion branch with spiceai-55-release-2.4 by @sgrebnov in #16046
  • Update spicepod.schema.json by @github-actions in #15828
  • ci: add rustfs-credentials action for GitHub OIDC credentials to RustFS by @phillipleblanc in #15976
  • ci: move CI credentials to OIDC and environment secrets by @phillipleblanc in #16010
  • ci: get RustFS credentials from the GitHub OIDC token by @phillipleblanc in #16005
  • test: remove CI tests that need a live Dremio server by @phillipleblanc in #15973
  • ci: authenticate BigQuery tests with workload identity federation by @sgrebnov in #16009
  • ci: authenticate CI calls to Anthropic with workload identity federation by @lukekim in #15954
  • ci: pin MongoDB 8.0 integration tests to 8.0.4 by @krinart in #15958
  • ci: run the released-runtime spice chat E2E test against xAI by @Jeadie in #15974
  • ci(cuda): least-privilege tokens and drop the MinIO copy by @sgrebnov in #16020
  • ci: scope workflow jobs to GitHub environments by @Jeadie in #15966
  • ci: pass secrets to run scripts through env, and use the read-write Postgres for load runs by @Jeadie in #15975
  • [release/2.4] ci: move CI credentials to OIDC and environment secrets (backport) by @sgrebnov in #16035
  • ci: fail a tag build when the tag does not match version.txt by @sgrebnov in #16021
  • ci(docker): build release images on GitHub-hosted runners by @sgrebnov in #16022
  • ci(release): cancel an older build of the same tag by @peasee in #16032
  • [release/2.4] ci: hosted Docker release builds and release hardening by @sgrebnov in #16044
  • ci: Use OIDC-based AWS roles for short-lived AWS credentials by @Jeadie in #16039
  • [release/2.4] Revert workload identity for models and GitHub App keys in Key Vault/KMS by @Jeadie in #16052
  • v2.4.2: remaining security backports, version bump to 2.4.2 by @sgrebnov in #16055

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.4.1...v2.4.2

Spice v2.4.1 (Oct 29, 2026)

ยท 15 min read
Viktor Yershov
Member of Technical Staff at Spice AI

Spice v2.4.1 is now available! ๐Ÿ› ๏ธ

Spice v2.4.1 is a patch release with MySQL connection improvements and security fixes. MySQL pools now replace each connection after 5 minutes by default. This spreads connections across the servers behind a load-balanced hostname, such as an Aurora reader endpoint. It also adds security hardening: stricter checks for internal network addresses and memory limits for untrusted input.

What's New in v2.4.1โ€‹

MySQL Connection Poolsโ€‹

MySQL datasets and catalogs now replace each pooled connection after mysql_connection_max_lifetime. Thus the pool spreads across the servers behind a load-balanced hostname, such as an Aurora reader endpoint. The default is 5m, and 0s keeps connections open. Idle connections above mysql_pool_min stay open for 30 seconds after their last use, so queries that follow each other closely reuse a connection.

With mysql_connection_string, mysql_pool_min and mysql_pool_max now apply. See MySQL deployment.

Each MySQL pool logs the server that its new connections reached, with the Aurora instance on Aurora MySQL:

MySQL dataset 'orders' opened connection 12345 to 'reader.example.internal:3306', which reached server 'ip-10-0-1-23:3306' (Aurora instance 'app-reader-2').

The line is INFO the first time in 10 minutes that a pool reaches a server, and DEBUG after that.

Slow-Refresh Warningsโ€‹

Accelerated datasets and views now log one warning while a full or append refresh runs slower than usual. This applies to every data source, not only MySQL. Spice compares the refresh with the median of the last 10 successful refreshes in the same mode. The threshold is the larger of 3 times and 30 seconds more than that median. The warning needs 3 earlier refreshes and has no setting:

Refresh of dataset 'orders' has run for 1m 3s, more than 3 times the 21s its last 3 full refreshes took at the median, so its accelerated data is falling further behind the source than usual. ...

See Data Refresh.

Internal Addresses Are Refused for Catalog Storage, HTTP Redirects, and URL Tablesโ€‹

Spice no longer sends a request to a private, loopback, or link-local address only because a data source names that address. The checks apply to these paths:

  • Iceberg REST catalogs: Spice refuses a storage endpoint or region that the catalog vends when it differs from the Spicepod value and reaches an internal host. Spice connects only to the addresses that its DNS check approved. Spice refuses a redirect from vended storage to an internal host, and it removes Authorization and Cookie headers when a redirect changes the origin.
  • AWS Glue: Spice rejects a table location that is not s3://.
  • HTTP and HTTPS datasets: A redirect to a host name that resolves to an internal address fails with Refused to connect to <host>.
  • URL tables: With runtime.params.url_tables: enabled, Spice checks every redirect and every resolved address. An s3, abfs, abfss, gs, or gcs URL table cannot set an endpoint or credentials in the URL fragment.

Storage endpoints and regions that the Spicepod sets keep working. SPICE_URL_TABLE_ALLOW_PRIVATE_HOSTS=true turns off the checks for Iceberg and URL tables. See Breaking Changes for the setups that need a setting.

HTTP Connector Passwords Stay out of Errorsโ€‹

Errors from http and https file datasets no longer include the password from the URL. This applies to the warning log, /v1/datasets?status=true, and Cloud Connect. Spice now sends the user name and http_password in an Authorization: Basic header, only to the origin that the dataset configures. CSV, Parquet, and other file reads through the HTTP connector now authenticate, and the server no longer answers them with 401.

Compressed HTTP Responses Have a Limitโ€‹

Spice refuses a compressed HTTP response after it decodes more than 64 MiB at more than 100 times the bytes received. In a test, a 583 KB gzip response grew the runtime to 2,464 MB of memory. With this fix, the runtime refused the response at 232 MB. The check applies to the HTTP connector, OAuth2 token exchange, Unity Catalog, Databricks, and the Debezium schema registry. Token and metadata responses also have a 128 MiB limit on decoded size.

Flight SQL Rejects Forged Handles and Oversized IPC Messagesโ€‹

Spice now signs each Flight SQL prepared statement handle with a keyed BLAKE3 tag, and it rejects a handle that does not verify. A client could build a handle by hand. Then a few hundred bytes of parameters made the Arrow IPC decoder allocate any size. In a test, one 389-byte GetFlightInfo request grew the runtime to 15 GB, and the kernel stopped it. A handle is valid only on the runtime process that issued it. See Breaking Changes.

Spice also refuses a compressed Arrow IPC message that declares more than 64 MiB and more than 100 times its compressed body. This applies to Flight SQL parameters, Flight DoPut, and the spice.ai and flightsql connectors.

Memory Limits for Untrusted Files, MCP Tool Results, and Scalar Functionsโ€‹

A small malformed file or a single query can no longer make the runtime allocate gigabytes of memory or crash:

  • Parquet pages: A page that declares more than 64 MiB and more than 1,000 times its compressed bytes fails with an error. In a test, 8 concurrent scans of a 34 KB file used 4,271 MiB of memory. With this fix, they used 166 MiB.
  • Parquet metadata: A footer list that declares more elements than the file can hold fails with a decode error. In a test, one malformed footer used 22.6 GiB of memory. With this fix, it used 0.43 GiB.
  • Parquet schema depth: A schema that nests more than 100 levels fails with Parquet schema nesting depth exceeds the maximum of 100. A deeper schema overflowed the stack and stopped the runtime.
  • ORC metadata: A metadata stream that decodes to more than 64 MiB and more than 100 times its compressed size fails with an error. In a test, a 1 MiB ORC file used 6.07 GiB of memory. With this fix, it used 337 MiB.
  • ORC type tree: A footer type tree that nests more than 100 levels, or that uses a type more than once, fails with an error. A deeper tree overflowed the stack and stopped the runtime.
  • DOCX, XLSX, PPTX, and PDF documents: A document fails with DocumentTooLarge past 64 MiB for one ZIP entry, 256 MiB decompressed in total, or 64 MiB of extracted text. In a test, a 1.2 MB DOCX file used 6.1 GiB of memory. With this fix, it used 0.27 GiB.
  • MCP tool results: An MCP tool call fails when the result is larger than 16 MiB. The runtime logs a warning that names the MCP server. In a test, a 200 MB result used 2,423 MB more memory. With this fix, it used less than 1 MB more.
  • Scalar functions: repeat, lpad, rpad, array_repeat, array_resize, range, and generate_series reserve their output from runtime.query.memory_limit. A call that does not fit fails with Resources exhausted. In a test with a 512 MiB limit, length(repeat('a', 1e9)) used 3.9 GB. With this fix, the query failed, and peak memory was 97 MB.

Read-Only API Keys Are Refused Earlier on Write Pathsโ€‹

A read-only API key now gets 403 API key does not allow write access on a write route before Spice reads the request body. A malformed body got a parsing error that named the required fields of the route. A read-only key also can no longer export OpenTelemetry metrics through gRPC on the Flight port.

Request Headers No Longer Create Metric Seriesโ€‹

Spice no longer copies User-Agent or W3C baggage values into metric labels. Any caller could send these headers, also to /health, and each distinct value created new series in memory. In a test, 500 requests with distinct values grew /metrics from 365 to 34,865 series. With this fix, the count stays at 365. The client label now has a fixed set of values. Refer to Breaking Changes.

WASM User-Defined Functions on arm64 (GHSA-mghf-wjfp-p5xm)โ€‹

Wasmtime 48.0.7 fixes a sandbox escape for WASM user-defined functions on arm64 hosts. On aarch64, the Cranelift compiler could jump to the wrong target for a large Wasm br_table instruction. The advisory is GHSA-mghf-wjfp-p5xm. It does not affect x86_64 hosts.

TLS Certificate Reload Rejects a Mismatched Keyโ€‹

A certificate rotation no longer breaks TLS handshakes when the new certificate does not match the private key. The runtime keeps the current certificate, logs a warning that names both files, and increments runtime_tls_reload_total{result="key_mismatch"}. At startup, a mismatched pair stops the runtime. See TLS.

The spice CLI HTTP commands, such as status, datasets, and refresh, now trust the CA in --tls-root-certificate-file. A certificate failure reports the TLS error and a fix, not The Spice runtime is unavailable.

Docker Images Get Current Packagesโ€‹

Each release build of the spiceai/spiceai image now runs apt-get upgrade. The build cache can no longer reuse a package layer from before a security update, such as DSA-6572-1. The CUDA image moves from nvidia/cuda:12.6.0-cudnn-runtime-ubuntu24.04, from August 2024, to 12.9.2 with current packages.

Dependency Updatesโ€‹

  • wasmtime: 47.0.3 โ†’ 48.0.7. This update fixes GHSA-mghf-wjfp-p5xm.
  • rustls: 0.23.45 โ†’ 0.23.46.
  • aws-lc-rs: 1.18.1 โ†’ 1.18.2. The bundled AWS-LC moves from v5.7.0 to v5.11.0.
  • webpki-root-certs: 1.0.10 โ†’ 1.0.11. This update removes the emSign Root CA - C1 and emSign ECC Root CA - C3 roots. v2.4.0 already removed them from webpki-roots.
  • http-body-limit: a new crate that limits decoded HTTP response bodies.
  • arrow-rs (Spice fork): 2e2cc330 โ†’ 2124b9e8. This update adds the Parquet page, metadata, and schema-depth limits. Arrow remains at v59.3.0.
  • orc-rust: 0.9.0 โ†’ the Spice fork at d9632e7a. This update adds the ORC metadata and type-tree limits.
  • CUDA Docker image base: nvidia/cuda 12.6.0 โ†’ 12.9.2.

DataFusion remains at v55.2.0 and Arrow remains at v59.3.0.

Contributorsโ€‹

Breaking Changesโ€‹

MySQL pool. A dataset with mysql_connection_string and no pool settings uses 1 to 5 connections, not 10 to 100. Set mysql_pool_max if the dataset needs more concurrent queries. A non-integer or conflicting pool setting now fails the dataset.

Iceberg and Glue storage endpoints. Spice refuses a catalog-vended endpoint on an internal host, with an error that names the endpoint. These setups need a change:

  • A REST catalog, such as a local Nessie, vends a private or loopback MinIO endpoint. Set iceberg_s3_endpoint and iceberg_s3_region.
  • A REST catalog vends only a region, and S3 resolves to private addresses through a VPC interface endpoint. Set iceberg_s3_endpoint to the regional S3 endpoint.
  • A Glue table location is s3a://, http://, or https://. Use an s3:// location.

URL table fragments. A URL table cannot set endpoint, allow_http, access keys, or proxy settings in the URL fragment. region, url_style, auth, account, and client_timeout still work. Configure a dataset for a custom endpoint. A URL table behind a proxy on a private address needs SPICE_URL_TABLE_ALLOW_PRIVATE_HOSTS=true.

Compressed HTTP responses. Very sparse rows can compress at more than 100:1, and Spice refuses them past 64 MiB. To turn off the check for a dataset, add Accept-Encoding: identity to http_headers.

Metric labels. The runtime no longer records the user_agent, client_version, client_system, or baggage labels. The client label is one of spice, repl, spicepy, spice-rs, gospice, spice.js, spice-java, spice-dotnet, or other. Update dashboards and alerts that use these labels.

Flight SQL prepared statements. A prepared statement handle is valid only on the runtime process that issued it. After a restart, or on another replica without session affinity, the client must prepare the statement again.

Documents. An XLSX file that lists cells out of document order fails with an error. A PDF that uses LZWDecode or an indirect /Filter reference is rejected.

Memory limits. Parquet, ORC, and document files past the limits in Memory Limits fail to load. MCP tool results larger than 16 MiB fail. Queries with these scalar functions fail when their output does not fit runtime.query.memory_limit.

TLS and the CLI. The runtime does not start when the TLS certificate does not match its private key. A missing or invalid --tls-root-certificate-file fails the CLI command.

Cookbook Updatesโ€‹

No new cookbook recipes.

The Spice Cookbook provides recipes to help you get started with Spice.

Upgradingโ€‹

To upgrade to v2.4.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.4.1 image:

docker pull spiceai/spiceai:2.4.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changedโ€‹

Changelogโ€‹

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.4.0...v2.4.1

Spice v2.4.0 (Oct 27, 2026)

ยท 76 min read
Jack Eadie
Member of Technical Staff at Spice AI

Spice v2.4.0 is a feature release that expands data ingestion, caching, and AI decisions, and makes Spice Cayenne the default accelerator. It also upgrades the query engine to Apache DataFusion v55 and Apache Arrow v59.3. Datasets and views that do not set acceleration.engine now use Cayenne instead of Arrow, except on native Windows. Other accelerators are deprecated and will be removed in v2.5. In tests, Cayenne served uncached primary-key lookups about twice as fast, and query latency stayed steady during full refreshes.

SQL queries and the new OpenAI-compatible /v1/decisions API can now ask TypeSafe or chat models for decisions. With refresh_mode: changes, accelerated S3 datasets read new objects from S3 when SQS delivers S3 event notifications. This release also adds SQL results-cache warmup, Enterprise acceleration snapshots, and new data connectors and formats.

Read Breaking Changes before you upgrade. SQL results-cache entries now expire only by TTL, not when a dataset refreshes. Cayenne ignores on_conflict and keeps one row for each primary key. Some datasets that loaded on Arrow fail to load on Cayenne unless they set engine: arrow.

Highlights in v2.4.0 include:

  • Spice Cayenne by Default: Datasets without acceleration.engine now use Cayenne instead of Arrow, except on native Windows. Cayenne adds automatic primary-key handling, persistent secondary indexes, and adaptive tuning (preview).
  • DataFusion v55: the query engine upgrades to DataFusion v55 and Arrow v59.3. In DataFusion benchmarks, the TopK query suite ran about 43% faster. Ballista v55, Vortex v0.86, and Iceberg v0.11 also ship in this release.
  • Faster Cayenne Lookups and Refreshes: in tests, uncached Cayenne primary-key lookups ran about twice as fast. P99 query latency during full refreshes fell from 130โ€“690 ms to 44โ€“50 ms.
  • AI Decisions in SQL and HTTP: five SQL decision functions and an OpenAI-compatible POST /v1/decisions API, with TypeSafe models or any chat model.
  • S3 Event-Driven Refresh: with refresh_mode: changes, accelerated S3 datasets read new objects when SQS delivers S3 event notifications. A periodic listing loads objects whose notification did not arrive.
  • Acceleration Snapshots (Enterprise): serve a dataset from published snapshots without its source connector. Snapshots use the shared state location and expire automatically.
  • SQL Results-Cache Warmup: record query shapes, then replay them after a restart so that the first queries use a warm cache.
  • New Data Connectors and Formats: query Hugging Face datasets, ORC files, and Iceberg tables without a catalog.

What's New in v2.4.0โ€‹

Spice Cayenne by Defaultโ€‹

Spice Cayenne now accelerates datasets and views that omit acceleration.engine. Before this release, they used Arrow. Native Windows keeps Arrow. On native Windows, run Spice in Docker or WSL to use Cayenne. Storage still defaults to mode: memory. Set mode: file for persistent acceleration. To move an existing acceleration, read the Cayenne migration guide.

Automatic ON CONFLICT handling for Primary Keysโ€‹

A Cayenne dataset with a primary_key now keeps one row for each key in every refresh mode, without an on_conflict setting. This table shows which version of a repeated key Cayenne keeps:

DatasetVersion kept for a repeated key
No time_columnThe last arrival.
time_column with full or append refreshThe newest time. The last arrival wins a tie. A NULL time is oldest.
refresh_mode: changesEach change applies in source order.

With a time_column, a late row with an older time does not replace a newer stored row. Sources such as Iceberg, where writers can commit out of time order, keep the newest version of each key.

datasets:
- from: s3://my-bucket/orders/
name: orders
time_column: updated_at
params:
file_format: parquet
acceleration:
enabled: true
mode: file
primary_key: id

Cayenne no longer uses on_conflict to select its conflict policy. For read-write datasets whose writes must stay in the acceleration, set access: read_write and acceleration.write_mode: acceleration. The source need not support writes. This setting cannot accompany a change-stream refresh (refresh_mode: changes, or a CDC source whose default refresh mode is changes).

An UPDATE that creates a primary-key collision fails before it changes rows. See duplicate primary keys and the migration guidance.

Runtime-Wide Adaptive Tuning (Preview)โ€‹

Cayenne can now change its own settings to meet runtime-wide targets for freshness, replication lag, query latency, and throughput. Set runtime.adaptive_tuning to enable it for Cayenne datasets and catalogs. Adaptive tuning is in preview.

runtime:
adaptive_tuning: enabled
params:
target_freshness: 30s

datasets:
- from: postgres:public.orders
name: orders
acceleration:
enabled: true
mode: file
refresh_mode: changes
primary_key: id

Adaptive tuning accepts four target objectives and one pacing setting in runtime.params. Cayenne changes its settings in small steps to move each measured value toward its target:

  • target_replication_lag: Desired maximum age of the newest applied source commit. Applies only to change-stream (refresh_mode: changes) datasets.
  • target_freshness: Desired maximum delay from a source commit until its data becomes queryable. Applies only to change-stream datasets.
  • target_query_latency: Desired P99 latency of the queries that read each Cayenne dataset.
  • target_qph: Desired throughput, in queries per hour (QPH), of queries that read Cayenne datasets.
  • target_convergence_window: Time budget for tuning to approach its targets. Default: 60s.

Adaptive tuning defaults to disabled. Targets apply only when adaptive tuning is enabled. A refresh_mode: full dataset does not use adaptive tuning. Each refresh replaces the whole table, so Cayenne turns off background compaction (cayenne_compaction_background_interval_ms: 0). The tuner runs on the compaction task, so it does not run for that dataset. The runtime logs a warning for each such dataset. See Cayenne self-tuning and Cayenne Tuning Settings.

Indexes, Initial Loads, and Compactionโ€‹

  • Persistent secondary indexes: File-mode Cayenne stores index runs with its acceleration. Writes maintain the indexes, and acceleration snapshots retain them. Dynamic join filters can probe secondary indexes.
  • Keyed initial loads: Cayenne can build the primary-key filter during the first load within its memory budget. Otherwise, it retains the existing warmup path. Bulk loads avoid per-row routing and a subsequent whole-table fold.
  • Maintained aggregates: Views share a compact index of per-key contributions. Rebuilds capture concurrent writes and apply them after the scan. A bounded query pool assigns 10% to aggregates without the former 512 MiB cap. An unbounded pool retains that cap.
  • Append-only compaction: For unkeyed datasets without position deletes, Cayenne encodes merged files outside the write lock. Publication still takes the lock. Datasets with position deletes retain the full-rewrite path.
  • Configuration updates: A restart applies changed write settings to an existing acceleration. Identity changes can rebuild eligible full-refresh accelerations from their source. See the Cayenne documentation.

Apache DataFusion v55 and Query Engine Upgradesโ€‹

Spice now runs on Apache DataFusion v55 and Apache Arrow v59.3. This release also upgrades Apache Ballista to v55, Vortex to v0.86.1, and Apache Iceberg to v0.11.0. The numbers in this section come from each project's own benchmarks, not from Spice tests.

Apache DataFusion v55โ€‹

The DataFusion v55 release adds these changes:

  • TopK pruning: As the threshold of an ORDER BY ... LIMIT query tightens, Parquet scans check the unread row groups again. They skip groups that cannot change the result. Compound ORDER BY keys are also supported. The topk_tpch suite ran about 43% faster in total.
  • Join planning: The optimizer converts eligible inner joins to semi joins. It removes redundant sides of LEFT and RIGHT joins and orders filter predicates by estimated cost. ClickBench Q21 ran 10โ€“13% faster.
  • Planning speed: The optimizer collapses chained projections and rewrites subqueries in place. One planning case fell from 623 ms to 155 ms, and TPC-DS optimization fell from 220 ms to 170 ms.
  • Functions: IN lists with few primitive values evaluate up to 14.7ร— faster in microbenchmarks. approx_distinct over 50,000 groups fell from 1,723 ms to 17 ms. More string functions keep dictionary encoding.
  • Aggregation: Multi-column GROUP BY keeps column-oriented storage for all key types, such as fixed-size binary UUIDs. Grouping 200 million UUIDs ran about 5% faster and used less memory.
  • Parquet scans: Scans skip nested fields that the table schema does not declare. They also skip the page index when a file has none. ClickBench Q1 on a single file ran 38% faster.
  • Spills: A sort under memory pressure spills its largest stream again in smaller batches instead of failing. Merges also limit their number of input streams.
  • SQL: EXPLAIN accepts PostgreSQL-style options and FORMAT pgjson. New array functions add, subtract, scale, sum, and average array elements.

Spice carries these changes through its query plans and keeps statistics across its plan wrappers. Cayenne does not split Vortex scans below 10 MiB, to avoid repeated footer reads.

Apache Ballista v55โ€‹

Spice.ai Enterprise feature. See Distributed Query.

Spice distributed query runs on a Spice fork of Apache Ballista v55. The Ballista v55 release adds these upstream changes:

  • Adaptive query execution by default: The scheduler replans between stages with row counts from completed stages. All 99 TPC-DS queries complete and match single-process DataFusion in the Ballista test suite.
  • Fewer tasks: One task can cover several input partitions, up to the available virtual cores of its executor. This change reduces planning, serialization, and dispatch work.
  • Shuffle: Remote shuffle reads receive blocks without a copy. Sort shuffle writes one file for each task.
  • Protocol version: Clients receive a clear error when their major version differs from the version of the scheduler.
  • Benchmark: On TPC-H at scale factor 1,000, Ballista ran the 22 queries in 502.4 seconds. Apache Spark 4.1.3 took 898.2 seconds on the same cluster.

Schedulers and executors must run the same version, so upgrade all cluster components together.

Vortex v0.79.0 to v0.86.1โ€‹

Spice v2.3.3 used the Vortex v0.79.0 fork. This upgrade covers every upstream release from v0.79.0 through v0.86.1:

  • Types and arithmetic: Vortex adds native Map arrays, union arrays, and decimal addition, subtraction, multiplication, and division.
  • Row selection: Piecewise-sequence indices represent contiguous selections without an index entry for every row.
  • Scans and expressions: Layout scans get a physical plan and expression optimization. Filters can pass through scalar functions with more than one argument. Filters on wide lists read only the selected range of child elements.
  • Compression: Binary arrays support FSST compression with variable-length offsets. OnPair becomes a stable encoding with a dictionary in storage.
  • File metadata: Files can store custom metadata. Readers cache decoded type descriptions.
  • Correctness and validation: Vortex corrects NULL handling for dictionary predicates, BETWEEN bounds, and empty arrays. Readers validate footer offsets, compression metadata, and array indices.

See the Vortex release notes for v0.80.0 through v0.86.0 and v0.86.1.

Apache Iceberg v0.11.0โ€‹

Spice updates its Iceberg reader and catalog integrations from v0.10.1 to the v0.11.0 fork for DataFusion v55 and Arrow v59. The upstream v0.11 changes extend reads and catalog compatibility:

  • Iceberg v3 reads: The reader applies deletion vectors from Puffin files. It carries row identifiers and sequence numbers through scans.
  • Metadata-only scans: Projections of metadata only do not read data columns. Positional-delete processing buffers runs instead of one key for each row.
  • REST catalogs: Clients negotiate the endpoints that the server advertises. They use OAuth2 authentication for each session.

The upgrade also aligns Iceberg storage with OpenDAL v0.58.

Performanceโ€‹

Spice Cayenne serves lookups, loads, and refreshes faster in this release. Each number below comes from the test in the linked pull request. The tests ran on different machines, so do not compare numbers between them.

Spice Cayenneโ€‹

  • Steady query latency during refreshes: Full and append refreshes now write on dedicated refresh threads by default. Before, a large refresh write ran on the threads that serve queries. In a test, a client sent 500 point-lookup joins per second to a Cayenne table of 2.2 million rows and about 8 GB. P99 latency during each full-refresh commit fell from 130โ€“690 ms to the steady 44โ€“50 ms. With runtime.params.dedicated_thread_pool: disabled, refresh writes run as before. See #15262.
  • Primary-key point lookups: Cayenne serves uncached primary-key point lookups about twice as fast. On a file-mode table of 10 million rows, P50 latency improved 2.11โ€“2.41ร— at 1 to 128 clients. See #14433.
  • Lookups after the first load: Uncached primary-key lookups right after the first full refresh are faster. On a table of 10 million rows, P50 latency at one client fell from 6.0 ms to 1.1 ms. At 16 clients, P99 latency fell from 16.3 ms to 3.9 ms. See #14314.
  • First refresh after a load: Cayenne builds the primary-key filter during the first load, so the first append refresh does not rebuild it. On a keyed table of 40.2 million rows, the first append refresh fell from a median of 4.2 s to 0.11 s. See #15099.
  • Keyed bulk loads: A first load into an empty keyed table no longer routes each row separately. On an Iceberg table of 40.2 million rows, the slowest first load fell from 21.0 s to 13.5 s. Iceberg scans also decode at the session batch size instead of 1,024 rows. See #15076 and #15072.
  • Append visibility on clustered tables: Cayenne orders each scan partition of a whole-table rewrite separately, then merges them. On an append table of 20 million rows with cayenne_cluster_by, P99 time from append to query visibility fell from 17.2 s to 3.0 s. See #14488.
  • Append-only ingestion: Small-file compaction no longer stalls writers on tables without a primary key or position deletes. In an OTLP log-ingestion test, export P99 latency fell from about 1.2 s to 0.12 s. Throughput rose from about 210,000 to 775,000 records per second. See #15227.
  • Metastore writes: Cayenne batches manifest rewrites in its SQLite metastore and runs every write on one writer connection. At 128 clients, P99 latency of a 1,000-file manifest rewrite fell from 5.39 s to 374 ms. See #14369.
  • Filtered key deletes: A filtered delete on a keyed table now reads only the key and filter columns. In a test, P99 latency of a key-only delete fell from 27.7 ms to 8.9 ms. See #14374.

Cachingโ€‹

  • Caching mode under sustained misses: Cayenne datasets with refresh_mode: caching now batch cache replacements. In a test with 500 lookups per second over 2.5 million random keys, request P99 latency fell from 363 ms to 9 ms. Dropped cache fills fell from 122,508 to 0. See #15253.
  • Caching mode without a primary key: Lookups stay fast when the cache replaces entries continuously on a dataset without a primary key. In a two-core test at 300 lookups per second, P99 latency fell from 1,377 ms to 9.3 ms. See #15362.
  • SQL results-cache expiry: Expiry work now grows with the number of expired entries, not with the size of the cache. With 3 million entries, insert P99.9 latency fell from 10.8โ€“15.7 ms to 70โ€“84 ยตs. See #14919.

Object-Store Listingsโ€‹

  • Metadata-only queries: GROUP BY, DISTINCT, MIN, and MAX over partition or metadata columns no longer scan whole files. Spice answers them from the file listing, or from the first record of each file. count, sum, and avg still scan the files. See #14116.
  • Listing pruning: Filters on _last_modified and other metadata columns now prune the file listing, so Spice does not open the excluded objects. Timestamp bounds joined by AND prune the listing. Filters with OR or NOT read the full listing. See #14265 and #14303.

AI Decisions in SQL and HTTPโ€‹

SQL queries and HTTP clients can now ask a model for yes-or-no answers, probabilities, labels, and scores. This release adds five SQL decision functions and an OpenAI-compatible POST /v1/decisions API. TypeSafe decision models return calibrated probabilities. Any chat model can also answer decisions, but its probabilities are uncalibrated estimates.

Add a TypeSafe model to the Spicepod:

models:
- from: typesafe:jev
name: support_model
params:
typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Then call a decision function in SQL. Each function selects its model with the model argument:

FunctionResult
ai_if(input, condition)BOOLEAN: true when the probability of yes exceeds 0.5.
ai_probability(input, condition)DOUBLE: a probability from 0 to 1.
ai_classify(input, labels)VARCHAR: one of the supplied labels.
ai_score(input, instructions, levels)DOUBLE: a probability-weighted score.
ai_decide(input, questions)STRUCT: answers, probabilities, and confidence.
SELECT
id,
ai_classify(body, ['billing', 'technical', 'account'],
model => 'support_model') AS team,
ai_probability(body, 'The customer requests a refund',
model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI decisions.

S3 Event-Driven Refreshโ€‹

With refresh_mode: changes, accelerated S3 datasets read new objects from S3 when SQS delivers S3 event notifications. Set refresh_mode: changes and s3_changes_queue_url together. The queue must serve only this dataset, and from must be a prefix, not a single object or a glob:

datasets:
- from: s3://my-bucket/events/
name: events
params:
file_format: parquet
s3_region: us-east-1
s3_changes_queue_url: ${secrets:events_queue_url}
acceleration:
enabled: true
mode: file
refresh_mode: changes

A periodic listing of the prefix (s3_changes_backfill_interval, default 1h) loads objects whose notification did not arrive. Spice loads each object key one time. By default, rows from deleted objects stay in the acceleration (s3_on_object_removed: ignore).

Acceleration Snapshotsโ€‹

Spice.ai Enterprise feature. See Acceleration Snapshots.

A dataset can now serve published acceleration snapshots without its original source connector. Set file_format: snapshot, and set from to the snapshot location:

datasets:
- from: s3://my-bucket/spice/snapshots/
name: orders
params:
file_format: snapshot
s3_region: us-east-1

Snapshots now expire after 28 days, except for the current and newest snapshot of each dataset. A reader that cannot load a newer snapshot keeps serving its loaded snapshot and tries again on the next poll. Each failure increments dataset_acceleration_refresh_errors{mode="snapshot", reason="snapshot_download_failed"}. OSS builds refuse snapshot datasets and do not publish snapshots. See Serve a dataset from published snapshots and Shared State and Snapshot Migration.

One Location for Shared Runtime Stateโ€‹

runtime.state sets one object-store location for the state that the runtime keeps across restarts and instances. Each of these features stores its state in its own subdirectory:

  • Cluster state
  • SQL results-cache warmup
  • Shared HTTP rate control (Spice.ai Enterprise)
  • Acceleration snapshots (Spice.ai Enterprise)
runtime:
state:
location: s3://my-bucket/spice/
params:
s3_region: us-east-1

datasets:
- from: postgres:orders
name: orders
acceleration:
enabled: true
mode: file
snapshots: enabled

Earlier per-feature locations, such as runtime.scheduler.state_location, still work but are deprecated. See Shared State and Snapshot Migration.

SQL Results Cache and HTTP Cachingโ€‹

SQL Results-Cache Warmupโ€‹

After a restart, the first queries can now read from a warm SQL results cache. The cache records query shapes and stores them. When Spice starts again, it replays the stored shapes after the first full or append refresh of each accelerated dataset:

runtime:
caching:
sql_results:
enabled: true
warmup: on_first_refresh

Spice records the first 10 distinct shapes of cacheable, unauthenticated queries. Queries that differ only in equality-filter values share one shape. During replay, warmup reads up to 1,024 distinct combinations of filter-column values for each shape. It reruns the shape with each combination until the cache is full. Datasets do not report ready until warmup completes. The runtime does not start when a Spicepod sets both warmup and cache_key_type: sql.

Cache Backendโ€‹

The SQL results, search, and embedding caches now use the Spice sharded cache. The runtime accepts engine: moka and engine: pingora for compatibility, but these values no longer select a backend.

Caching Refresh Modeโ€‹

These changes apply to datasets with refresh_mode: caching, such as HTTP datasets:

  • Shared cache-miss fetches: Concurrent cache misses for the same key now share one origin fetch.
  • stale-if-error: Cached HTTP responses now follow the RFC 5861 stale-if-error rules.

New Data Connectors and Formatsโ€‹

Hugging Face Datasetsโ€‹

The Hugging Face data connector queries and accelerates Parquet, CSV, TSV, JSON, and ORC files from the Hub:

datasets:
- from: hf://datasets/stanfordnlp/imdb/plain_text/
name: imdb
acceleration:
enabled: true

ORC and Object Metadataโ€‹

Listing connectors now support file_format: orc. Enable metadata columns to query the location and modification time of each row's source object:

datasets:
- from: s3://my-bucket/events/
name: events
params:
file_format: orc
s3_region: us-east-1
metadata:
_location: enabled
_last_modified: enabled

For example, query objects modified since October 1:

SELECT DISTINCT _location, _last_modified
FROM events
WHERE _last_modified >= TIMESTAMP '2026-10-01 00:00:00';

Apache Iceberg Without a Catalogโ€‹

Apache Iceberg datasets can read a table directly from its root directory on local or object storage. Spice finds the current metadata file on every query and every refresh, so it reads new commits without a restart:

datasets:
- from: iceberg:s3://my-bucket/warehouse/db/taxis
name: taxis
params:
iceberg_s3_region: us-east-1

To read one fixed table version, set iceberg_metadata_path to a metadata file. A pinned dataset is read-only, so it fails to load with access: read_write:

datasets:
- from: iceberg:s3://my-bucket/warehouse/db/taxis
name: taxis_v3
params:
iceberg_s3_region: us-east-1
iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Iceberg Append Refresh Follows Table Snapshotsโ€‹

An Iceberg dataset with refresh_mode: append and a time_column now loads the rows that each new commit adds, whatever their time values. This applies to Cayenne datasets and to datasets without a primary key. Before this release, an append refresh read only rows later than the stored maximum time minus refresh_append_overlap. A writer that committed out of time order could put rows outside that window. The refresh now saves the last Iceberg snapshot that it loaded, and the next refresh reads the data files that later snapshots added. See append refresh.

Operations and Observabilityโ€‹

The runtime can now send structured logs through the existing OTLP exporter:

runtime:
telemetry:
otel_exporter:
endpoint: otel-collector:4317
signals: [metrics, logs]
query:
log_errors: execution

Log records carry severity, typed attributes, and the query trace ID. The runtime batches delivery on a dedicated export runtime. signals defaults to [metrics].

runtime.query.log_errorsBehavior
none (default)Memory refusals at WARN, other query failures at DEBUG.
executionAlso logs execution failures at WARN.
allAlso logs syntax and planning failures at INFO.

Failure records contain error_code and datasets. They do not contain SQL text. See runtime.query.log_errors and observability.

Other changes:

  • Adaptive HTTP rate control: HTTP rate control now reduces admission after upstream failures and stays within the configured request limits. In Spice.ai Enterprise, instances that share runtime.state also share per-second and per-minute limits. See HTTP rate control.
  • Open-file limits on Unix: The runtime raises its soft open-file limit to the hard limit at startup. On macOS, kern.maxfilesperproc also bounds the result. If the hard limit remains too low, the Cayenne error names the effective limit and the remedy.
  • Shared-state metrics: state_store_request_duration_ms{feature, operation}, state_store_conflicts_total{feature}, and state_store_cleanup_deleted_total{feature} report requests, conditional-write conflicts, and cleanup.
  • Prometheus scrapes: Derived summary families emit one HELP and TYPE block per family. Strict parsers no longer reject the scrape for repeated metadata.
  • Datadog dashboard: New panels cover snapshot freshness, Cayenne segment caches, and pod health.

Security Improvementsโ€‹

  • Brotli decoder (CVE-2026-111497): The DuckDB that Spice bundles no longer writes out of bounds when it decodes a malformed Brotli-compressed Parquet page. Most Parquet reads in Spice do not go through DuckDB. The fix matters where DuckDB decodes a Parquet file that an untrusted party can supply.
  • Flight sessions: Authentication precedes session creation. Sessions remain bound to their creating principal. TLS handshakes have a 10-second deadline, and Flight SQL statement updates count against the Flight write limit.
  • HTTP paths: The HTTP connector rejects request_path values that URL parsing would rewrite outside their allowed paths. This covers encoded dot segments and control-character rewrites.

Bug Fixesโ€‹

The fixes with the widest effect are below. The changelog lists every fix.

  • Wrong rows from Cayenne queries: Cayenne queries no longer return wrong rows from maintained views, Vortex IN lists, or shared dynamic filters. See #14427.
  • NOT IN and NULL values: Queries with NOT IN no longer lose their NULL semantics after a join reorder or the Cayenne sort-merge rewrite. See #14429.
  • Wrong results from DataFusion: The DataFusion upgrade includes upstream fixes for wrong results from filter pushdown, expression simplification, and planning. See #14430.
  • Federated pushdown: Federated queries no longer send some casts and filters to a source that evaluates them differently from Spice. Affected sources include DuckDB, PostgreSQL, MySQL, BigQuery, DynamoDB, Cosmos DB, and MongoDB. See #14484, #14601, and #14419.
  • Accelerated HTTP datasets: An HTTP error response no longer replaces the rows of an accelerated HTTP dataset. See #13538.
  • Unavailable sources at startup: A dataset whose source is unreachable at startup now retries. An existing acceleration keeps serving queries while its source is unavailable. See #14623 and #14624.
  • Stale plans and results after a reload: Queries no longer use cached plans or results after a hot reload replaces a catalog or unloads a dataset. See #13914 and #14365.
  • CDC positions at shutdown: Change-data-capture sources now record their positions before shutdown closes the accelerations. See #14702.
  • Tool calls: Tool-use streams no longer end on a stale tool_calls finish. /v1/responses runs Spice tools and honors tool_recursion_limit when it streams. See #14548 and #14930.
  • Vector search on chunked columns: vector_search returns its top k rows by score on chunked and multi-vector columns. See #15056.

SDK Updatesโ€‹

Updated SDKs release alongside v2.4.0:

  • spice.js v3.3.0: adds query cancellation with an AbortSignal, runtimeStatus(), and refreshDataset(). refreshAcceleration() is deprecated. Fixes intermittent HTTP query failures on Node, API keys on plaintext gRPC, and typed Flight parameters. If you set only one of flightUrl or httpUrl, the client now uses the matching address for the other. An empty URL or a Flight authentication failure now throws.
  • spicepy v4.0.1: sends API keys as a Bearer token on Flight and ADBC calls to self-managed runtimes, and binds numpy and pandas scalars as query parameters. refresh_dataset() and runtime_status() now raise SpiceAIError with the runtime's error message instead of requests.HTTPError. Requires certifi 2026.10.19 or later.
  • spice-rs v5.0.0: upgrades to Arrow 59 and binds chrono and std::time values as query parameters. Parameterized queries now work with API keys, and queries retry after a connection reset. Applications that also depend on arrow must move to Arrow 59.
  • spice-java v0.8.1: accepts any non-empty API key, and sends it on refreshDataset. Fixes the NoClassDefFoundError that v0.8.0 raised on every TLS connection, including Spice.ai Cloud.
  • spice-dotnet v0.5.0: adds GetRuntimeStatusAsync() for /v1/status, which all six SDKs now support. Accepts any non-empty API key, and supports grpc:// Flight addresses.
  • gospice v9.1.0: binds time.Time, time.Duration, pointers, sql.Null* types, and named types as query parameters. Accepts any non-empty API key, and uses one Flight session for each client. Requires Go 1.26 or later. Decimal128Param and Decimal256Param now read Arrow little-endian bytes, so check those calls before you upgrade.

Agent Skills Updatesโ€‹

Updated Agent Skills release alongside v2.4.0:

  • spiceai/skills v2.4.0:
    • New launch skill: deploys a Spice.ai Cloud project, validates that it serves data and model responses, and sets up monitors and alerts.
    • New sdk skill: covers the six client SDKs.
    • cloud skill (previously spice-cloud-management): now works through the management MCP server and can fork projects.
    • All skills are updated for v2.4.0, including Cayenne as the default accelerator and the deprecation of other accelerators.

Dependency Updatesโ€‹

Dependency / Componentv2.3.3v2.4.0
Apache DataFusionv54.1.0v55.2.0
Apache Arrowv58.3.0v59.3.0
LF+AI Vortexv0.79.0v0.86.1
Apache Icebergv0.10.1v0.11.0
Apache Ballistav54.0.0v55.0.0
Tursov0.7.2v0.8.2
ADBCv0.23.0v0.24.0
Rust toolchainv1.97.1v1.98.1

Breaking Changesโ€‹

Accelerator Selection and Primary-Key Policiesโ€‹

Spice Cayenne is now the default accelerator on supported platforms. Datasets and views without acceleration.engine switch from Arrow to Cayenne. Native Windows keeps Arrow. If datasets rely on another accelerator's behavior, review the Cayenne migration guide before the upgrade.

Other accelerators are deprecated for removal in v2.5. This applies to Arrow, DuckDB, SQLite, Turso, and PostgreSQL acceleration, and to the DuckDB vector engine. They remain available in v2.4. A startup warning groups affected components by accelerator.

On native Windows, run Spice in Docker or WSL to use Cayenne before v2.5.

Some datasets that loaded on Arrow fail to load on Cayenne. These configurations loaded on Arrow when they did not set engine. They now fail to load unless the dataset sets engine: arrow:

  • partition_by in memory mode, or with no mode
  • A column of type Interval, Duration, FixedSizeBinary, Union, or RunEndEncoded, such as a PostgreSQL interval column
  • A Debezium or cdc: source with no key
  • A numeric time_column with retention_period
  • maintained_aggregates with MIN or MAX and no primary key

To retain Arrow temporarily in v2.4, name it explicitly:

acceleration:
enabled: true
engine: arrow

Plan acceleration migration before v2.5. Do not delete acceleration-only data to change accelerators. Keep those datasets on v2.4 until a supported migration can preserve their data. The migration guide describes other limitations and vector-score changes.

on_conflict is deprecated for removal in v3.0. Cayenne accepts the field but no longer applies its policy. A primary key selects automatic version resolution. Before an upgrade, review drop policies and policies that reject conflicts. Other accelerators retain their existing policies in v2.4.

on_conflict no longer routes writes to the acceleration, for all accelerators. In v2.3, a read-write dataset that set on_conflict sent its writes to the acceleration, so its source did not need to accept writes. In v2.4, only write_mode selects where writes go. Such a dataset does not load if its source cannot accept writes. To keep writes in the acceleration, set write_mode: acceleration:

datasets:
- from: s3://my-bucket/orders/
name: orders
access: read_write
params:
file_format: parquet
acceleration:
enabled: true
mode: file
primary_key: id
write_mode: acceleration

SQL Results-Cache TTL and Freshnessโ€‹

SQL results-cache entries now expire only by TTL. Full and append refreshes, CDC batches, and retention passes no longer invalidate them. Queries can return data from before the latest refresh until item_ttl expires. The stale_while_revalidate_ttl setting can extend this period. Review both settings before the upgrade. Reduce them if results must reflect refreshed data sooner. See Cached Results and Dataset Refresh.

HTTP Rate-Control Wait Timeoutsโ€‹

If rate_control_acquire_timeout is unset, an HTTP request now waits up to the connector's client_timeout for rate-control capacity. Requests previously waited without a limit. Set rate_control_acquire_timeout: 0 under the dataset's params to retain unbounded waits. Otherwise, choose a duration that matches the request budget, such as 30s.

Datasets that share an HTTP origin share one rate controller. They must use the same max_concurrent_requests, requests_per_second_limit, requests_per_minute_limit, rate_control_jitter_min, rate_control_jitter_max, rate_control_failure_threshold, and rate_control_window values. If the values differ, the dataset fails to load with an error that names these settings. See rate control parameters.

Iceberg REST Compatibilityโ€‹

GET /v1/namespaces/{namespace}/tables/{table} now returns HTTP 400 with BadRequestException for accelerated datasets, views, and other unsupported datasets. The endpoint no longer synthesizes metadata for those datasets. For schema discovery, use SQL DESCRIBE or information_schema.columns. For queries, use /v1/sql or Arrow Flight SQL. Eligible unaccelerated Iceberg datasets expose their source metadata, and clients need their own storage credentials.

Instance Identity and Cluster Upgradesโ€‹

Instance identity resolves in this order: nonempty SPICE_INSTANCE_ID, HOSTNAME, COMPUTERNAME, then the host name. A scheduler's default changes from host:port to the host name. Executors also register under the instance identity.

If several schedulers or executors share a host, give each a distinct SPICE_INSTANCE_ID. Upgrade schedulers before executors. See distributed query. Older schedulers cannot fan out task-history requests to newer identity-based registrations. All cluster components must use compatible Ballista protocol versions.

PostgreSQL replication slots keep their names when a nonempty identity environment variable already selected the instance name. Without those variables, the host-name fallback creates a new slot. An empty SPICE_INSTANCE_ID also changes the previous slot name. After the new slot works, drop the unused old slot to stop its WAL retention. See PostgreSQL replication:

SELECT pg_drop_replication_slot('<old slot name>');

Shared State and Snapshot Migrationโ€‹

The top-level snapshots section and runtime.scheduler.state_location and params are deprecated. They continue to work, and when set they take priority over runtime.state for their feature. Prefer runtime.state, with per-dataset overrides (acceleration.snapshots_location, acceleration.snapshots_params) when necessary.

For upgrades from v2.4.0-rc.1, move state from the shared root into these paths:

StateSubdirectory
Cluster schedulercluster/
Rate controlrate-control/
SQL results-cache warmupresults-cache/

An explicit legacy scheduler location retains its earlier layout until migrated. In rc.1, Azure state ignored the URL path and used the container root. Move that state into the configured path and feature subdirectory.

The configured store must support writes and conditional writes. The runtime refuses newer state formats and stores that ignore write conditions. Check storage permissions before the upgrade.

Snapshots now expire automatically after 28 days, except for each dataset's current and newest snapshot. Rollback cannot select a pruned snapshot. Review external retention requirements before the upgrade.

Upgrade snapshot readers before, or alongside, their writers. Cayenne writes format 2 of the metastore slice when a snapshot contains partition children or pending write-back markers. Older readers reject that format. Other slices retain format 1.

Cayenne cannot publish acceleration snapshots for datasets with cayenne_datalake_location. Review snapshot settings for those datasets.

For a serving snapshot reader, bootstrap_on_failure_behavior: retry no longer retries a newer snapshot indefinitely inside one refresh. The reader reports the failure and retries at the next poll. Startup behavior does not change.

Cayenne Tuning Settingsโ€‹

Replace retired per-dataset and per-catalog tuning settings with runtime settings:

Previous settingReplacement
cayenne_tuning: adaptiveruntime.adaptive_tuning: enabled
cayenne_tuning: autoruntime.adaptive_tuning: disabled
cayenne_goal_replication_lagruntime.params.target_replication_lag
cayenne_goal_freshnessruntime.params.target_freshness
cayenne_goal_query_latencyruntime.params.target_query_latency
cayenne_goal_convergence_windowruntime.params.target_convergence_window
cayenne_goal_qphruntime.params.target_qph

The runtime no longer applies the old names. It warns with their replacements. Targets are runtime-wide. runtime.adaptive_tuning accepts only enabled or disabled. A Boolean such as true fails Spicepod loading.

Upgradingโ€‹

Review Breaking Changes before upgrading. Upgrade all distributed cluster components together, with schedulers before executors.

Upgrade snapshot readers before, or alongside, their writers so they can read the newer Cayenne metastore slice format.

To upgrade to v2.4.0, use one of these methods:

CLI:

spice upgrade v2.4.0

Homebrew:

brew upgrade spiceai/spiceai/spice

Homebrew follows the version published in the tap.

Docker:

docker pull spiceai/spiceai:2.4.0

See Docker Hub for available tags.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0

AWS Marketplace:

Spice is available in the AWS Marketplace. Marketplace availability follows its published versions.

Cookbook Updatesโ€‹

The Spice Cookbook provides recipes to help you get started with Spice.

Contributorsโ€‹

What's Changedโ€‹

Changelogโ€‹

  • fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
  • fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
  • fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
  • fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
  • fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
  • fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
  • fix(cayenne): release the keyset bytes an abandoned PK checkout accounted (fixes #13668) by @grokspice in #13925
  • fix(duckdb): deny the regexp built-ins DuckDB cannot answer faithfully (fixes #13809) by @claudespice in #13871
  • fix(search): drop the chunks a shortened row no longer produces from a chunked index (refs #13717) by @claudespice in #13960
  • Reduce Cayenne allocations during primary-key validation and filtering by @lukekim in #14009
  • fix: clarify OpenDAL S3 retry warnings by @lukekim in #14040
  • feat(cayenne): materialize multi-reference CTEs on the query path by @lukekim in #13918
  • fix(smb): pad an empty CREATE buffer so Samba lists the share root (fixes #13293) by @grokspice in #14050
  • feat(mcp): support MCP specification 2026-07-28 (dual-era) by @lukekim in #14043
  • feat(connectors): add ORC listing format via in-repo FileFormat by @lukekim in #14075
  • feat(cloud-connect): add a GetDatasets command that answers the /v1/datasets document (refs #13369) by @grokspice in #14051
  • Suppress Cayenne startup logs when no Cayenne dataset is configured by @Jeadie in #14042
  • fix(cache): re-bind parameter values when revalidating a stale result (fixes #14099) by @bjchambers in #14100
  • fix(deps): remediate open Dependabot alerts by @phillipleblanc in #14111
  • feat(caching): Make caching_stale_if_error RFC-5861 compliant (with stale-if-error header) by @Jeadie in #14134
  • perf(runtime-table): defer cache-eviction key extraction to entries a delete actually names by @Jeadie in #14138
  • fix(runtime): report the acceleration.ready_state deprecation once per component (fixes #13749) by @claudespice in #14006
  • fix(runtime): write the inferred Arrow sort order under the prefixed key its validation accepts (fixes #14023) by @claudespice in #14032
  • fix(connectors): Fix JSON/Orca files using metadata columns by @Jeadie in #14115
  • fix(turso): build a dictionary column directly so a dictionary over a list, map or boolean value reads back (fixes #13033) by @grokspice in #14181
  • fix(github): paginate nested GraphQL connections and pace to GitHub's rate limits by @lukekim in #14179
  • fix(vortex): stop an IN list holding a NULL from panicking the scan by @krinart in #14163
  • fix(runtime-table): stop rebuilding SessionContext on every cache fetch by @Jeadie in #14141
  • fix(runtime): count a first load as still loading in the Dataset load summary (fixes #13974) by @claudespice in #14020
  • fix(duckdb): push regexp_count down again at a rendering that counts as the kernel does (fixes #13870) by @claudespice in #14153
  • fix(cayenne): build the DELETE sink inside the execution-time write lock (fixes #13828) by @claudespice in #14218
  • fix(cluster): decide what a Flight message carries by its IPC header, not its body length (refs #13737) by @claudespice in #14212
  • fix(runtime): install the Spice function deny-list on the PostgreSQL catalog connector (refs #13664) by @claudespice in #14225
  • perf(cayenne): share inline-cache view entries by Arc instead of cloning them per scan by @krinart in #14191
  • feat(cache): Spice sharded cache as the sole LruCache engine by @lukekim in #14206
  • feat: TypeSafe Jev System One evaluation provider by @lukekim in #14215
  • fix(cayenne): tag file statistics bounds as the column's Arrow type (fixes #14280) by @phillipleblanc in #14283
  • fix(cayenne): coalesce inline writes into one batch by @sgrebnov in #14279
  • fix(cayenne): count each Arrow allocation once in the inline-cache gauge by @krinart in #14272
  • feat(s3): SQS event-driven changes for refresh_mode: changes by @lukekim in #14121
  • feat(caching): single-flight coalesce concurrent cache-miss fetches by @Jeadie in #14142
  • fix(caching): make caching_stale_if_error detect transient HTTP failures on real schemas by @krinart in #14161
  • Use Cayenne secondary indexes for dynamic join filters by @phillipleblanc in #14282
  • fix(cli): surface the full error chain for spice chat connection failures by @krinart in #14289
  • fix(runtime-table): serialize a direct write against acceleration snapshot creation (fixes #13548) by @claudespice in #14310
  • fix(duckdb): screen regexp_like and regexp_replace as regexp_count is screened (fixes #14148) by @claudespice in #14321
  • fix(http): honor retry budget without an extra origin request by @phillipleblanc in #14313
  • Fix SchemaCastScanExec's schema conversion in fn partition_statistics by @Jeadie in #14258
  • fix(cluster): recognise a Flight keepalive by its empty envelope, not by what its header declares (fixes #13737) by @claudespice in #14327
  • perf(http): defer zero-TTL acceleration lookup until origin failure by @phillipleblanc in #14302
  • fix(cayenne): keep a key visible when it is re-inserted over a stale-insert tombstone by @sgrebnov in #14312
  • perf(cayenne): read only key and filter columns in filtered key deletes by @sgrebnov in #14374
  • fix(cayenne): let small protected-snapshot merges run during a long large-tier merge by @sgrebnov in #14296
  • perf(cayenne): serve primary-key lookups on a freshly loaded table in ~1 ms by @lukekim in #14314
  • fix(cayenne): skip min/max statistics for nested columns (fixes #14368) by @sgrebnov in #14392
  • fix(turso): keep a quantified comparison out of Turso SQL (fixes #14041) by @grokspice in #14393
  • perf(cayenne): batch metastore manifest rewrites, upsert in place, and run every write on one writer connection by @lukekim in #14369
  • fix(runtime-table): ignore zero-row stale fallback batches by @phillipleblanc in #14331
  • Prune object-store file listing by _last_modified predicates by @Jeadie in #14265
  • Reading only partition or metadata columns needlessly scans all file contents by @Jeadie in #14116
  • fix(duckdb): keep a concat over a binary operand out of the federated plan (fixes #13915) by @claudespice in #14333
  • fix(federation): keep a correlated subquery predicate above a join of two sources (refs #8220) by @claudespice in #14372
  • fix(caching): snapshot staleness before the origin fetch for stale_if_error by @Jeadie in #14263
  • perf(snapshots): skip unchanged snapshot metadata with a conditional GET by @sgrebnov in #14409
  • fix(search): prune the rest of a key group from an Elasticsearch chunked index (refs #13717) by @claudespice in #14320
  • fix(graphql): debit a LIMIT by the rows a page returned, not the declared page size (fixes #14308) by @claudespice in #14353
  • fix(snapshots): resolve snapshot entries relative to the metadata location (#14425) by @sgrebnov in #14426
  • Lower GitHub connector default concurrency limit to 4 by @lukekim in #14431
  • fix(cache): don't report an invalidation cancelled by runtime shutdown as a failure by @grokspice in #14417
  • fix(runtime): keep the built-in date_part so both weekday spellings agree (fixes #13920) by @claudespice in #14154
  • fix(cayenne): include the in-memory CDC tier when an overwrite or a retention pass covers the whole table by @lukekim in #14428
  • fix: keep NOT IN null-aware through the join reorder and the Cayenne sort-merge rewrite by @lukekim in #14429
  • fix(cayenne): discard a compaction whose snapshot an overwrite replaced mid-pass by @lukekim in #14432
  • fix(cayenne): stop maintained views, Vortex IN lists and dynamic-filter sharing from returning wrong rows by @lukekim in #14427
  • fix(cayenne): hide spilled rows on CDC upsert fallback by @bjchambers in #14416
  • fix(cayenne): judge a filtered key delete's captured sources by the index captured with them (refs #13913) by @claudespice in #14455
  • fix(cayenne): correctness fixes for the goal-driven adaptive controller, with a closed-loop simulation harness by @lukekim in #14443
  • fix(cdc): keep the newest source commit timestamp on a coalesced change batch by @lukekim in #14463
  • fix(cayenne): draw a sequence for a current-snapshot append so per-key OCC can order it (fixes #13685) by @claudespice in #14360
  • fix(http): name a configured model's load failure instead of reporting it not found (fixes #13303) by @claudespice in #14395
  • fix(duckdb): keep inferred source indexes off change-stream accelerations so upserts commit under concurrent reads (refs #13929) by @claudespice in #14396
  • fix(duckdb): keep a text cast over a binary operand out of the federated plan (fixes #14355) by @claudespice in #14448
  • fix(llms): honor tool_choice required and allowed_tools on mistral.rs-hosted models instead of panicking (fixes #14230) by @claudespice in #14460
  • fix(runtime): stop a replaced dataset configuration's load from registering over the new one (fixes #1458) by @claudespice in #14367
  • fix(install): stop asking for sudo on a first install into a fresh HOME (fixes #14445) by @claudespice in #14495
  • perf(cayenne): serve primary-key point lookups in half the time by @lukekim in #14433
  • fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning by @lukekim in #14430
  • fix(runtime): size every internal DataFusion session from the CPU budget by @bjchambers in #14412
  • fix(runtime): serve nested, zoned and half-float columns from the Iceberg catalog API (fixes #4815) by @claudespice in #14480
  • fix(acceleration): keep cached results when a snapshot refresh finds no newer snapshot by @sgrebnov in #14497
  • fix(cayenne): stop the small-file compaction of a position-mode PK table from deadlocking on its own write lock (fixes #14420) by @claudespice in #14481
  • fix(runtime-table): report refresh bytes for the rows each batch holds by @Jeadie in #14469
  • fix(spark,databricks): keep Spice-only functions out of SQL sent to Spark Connect and Databricks SQL Warehouse (refs #13664) by @claudespice in #14498
  • fix(vortex): size a cached footer by what it retains, not its serialized bytes (fixes #12917) by @claudespice in #14502
  • fix(cayenne,telemetry): Register accelerated sink dataset immediately from existing acceleration by @peasee in #13955
  • fix(cayenne): keep a rewrite's count inexact when it retains a late protected snapshot (fixes #14383) by @claudespice in #14385
  • fix(postgres): release a shared slot's hold on a table its publication cannot drop (fixes #13032) by @claudespice in #14527
  • fix(cayenne): size the build side before sort-merging a non-Cayenne outer join by @krinart in #14520
  • fix(deps): move the DataFusion pin past the four unparser fixes, and guard each of them (fixes #13022) by @grokspice in #14570
  • fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does by @lukekim in #14419
  • feat(snapshots): serve a dataset from published snapshots with from: s3://โ€ฆ and file_format: snapshot by @lukekim in #14529
  • perf(cayenne): keep the per-shard PK index across checkpoint flushes and back off futile bakes; fix q17 and two wrong-results races (fixes #14235) by @lukekim in #14555
  • fix(graphql): show the parse-failure bytes in JSON decode previews by @Jeadie in #14530
  • Defer source-first cache fallback planning by @phillipleblanc in #14556
  • perf(cayenne): order whole-table rewrites per scan partition, then merge by @bjchambers in #14488
  • fix(graphql): retry an inferred credential refusal instead of failing the refresh by @Jeadie in #14539
  • fix(cayenne): serve a widened table's files from their persisted statistics (refs #13829) by @claudespice in #14316
  • fix(acceleration): checkpoint DuckDB's write-ahead log before a snapshot copies its file (fixes #13912) by @claudespice in #14325
  • fix(models): apply a tool_choice that forces a call to one round of the tool-use loop (fixes #14459) by @claudespice in #14635
  • fix(runtime): don't end a tool-use stream on a stale tool_calls finish (fixes #13309) by @claudespice in #14548
  • fix(smb): name listed locations under the share so directory datasets load (fixes #14060) by @claudespice in #14549
  • fix(models): name a configured model's load failure in ai() and streaming nsql (fixes #14394) by @claudespice in #14632
  • fix(cayenne): record sharded CDC keys in the table-wide PK index by @lukekim in #14603
  • fix(sqlite): keep TRY_CAST and every cast SQLite answers differently out of the federated plan (fixes #14398) by @claudespice in #14496
  • fix(snapshots): retry a failed snapshot attempt with backoff by @sgrebnov in #14676
  • fix(snapshots): large snapshots no longer fail bootstrap on slow connection by @sgrebnov in #14571
  • fix(refresh): start a refresh's source scan only when the sink reads it, so the full-text index keeps its rows (fixes #14619) by @claudespice in #14663
  • Evaluate any chat model via /v1/evaluate by @lukekim in #14568
  • fix(cayenne): stop snapshotting datasets with a datalake tier, whose restored copies deleted each other's files by @lukekim in #14583
  • fix(cayenne): count each Arrow allocation once for the mem-tier limit and checkpoint write sizing by @krinart in #14675
  • fix(runtime): retry a dataset whose source is unreachable at startup (fixes #14609) by @bjchambers in #14623
  • fix(cayenne): archive only the snapshot directories a dataset snapshot references (fixes #14605) by @sgrebnov in #14627
  • fix(smb): serve every share on a host from the one store registered for it (fixes #14550) by @grokspice in #14689
  • fix(runtime): discard cached plans and results when a dataset is unloaded (refs #14251) by @claudespice in #14365
  • feat(snapshots): reload snapshot-mode datasets from S3 event notifications (SQS) by @lukekim in #14335
  • fix(cayenne): report runs per size tier when protected-snapshot compaction declines (fixes #13622) by @claudespice in #14476
  • fix(cayenne): re-run a post-write compaction pass a concurrent append asked for (fixes #13906) by @claudespice in #14479
  • fix(http): keep the request URL out of HTTP connector errors (fixes #13534) by @claudespice in #14490
  • fix(runtime): load every localpod dataset that reads from one parent at startup (fixes #13087) by @claudespice in #14532
  • fix(duckdb): push regexp_replace down again for a one-digit group reference, and refresh the Postgres ClickBench q35 plan (refs #13966) by @claudespice in #14552
  • fix(duckdb): keep a cast into binary out of the federated plan (refs #14397) by @claudespice in #14633
  • fix(runtime): keep finished async query jobs finished, and delete expired ones by @lukekim in #14585
  • fix(federation): keep DataFusion's cast built-ins out of every federated plan (fixes #14444) by @claudespice in #14484
  • fix(runtime): decide every Spark/built-in function collision by name, and refuse an undecided one (fixes #14361) by @grokspice in #14400
  • fix(runtime-component): skip an inferred Cayenne index on a floating-point column instead of failing the load (fixes #14590) by @grokspice in #14599
  • fix(cache): serve stale cached results during frequent table updates (stale_while_revalidate_ttl) by @sgrebnov in #14708
  • Update Turso to 0.8.1 and retry write conflicts a metastore statement raises by @lukekim in #14680
  • fix(snapshots): stop a snapshot dataset's load when a reload replaces it by @lukekim in #14673
  • fix(cayenne): hand non-partition filters to every partition scan (fixes #12959) by @claudespice in #14370
  • fix(postgres-accel): resolve secret references in the sidecar connection parameters (fixes #13296) by @claudespice in #14544
  • fix(cayenne): drain the in-memory CDC tier before a full rewrite scans it (fixes #14450) by @claudespice in #14486
  • feat(caching): warm SQL results cache on first refresh from persisted plan shapes by @lukekim in #14178
  • fix(federation): keep a fractional-to-integer cast out of the plans pushed to DuckDB, PostgreSQL, MySQL and BigQuery (fixes #14482) by @grokspice in #14601
  • fix(federation): keep arrow_typeof and its plan-introspection siblings local on every backend (fixes #14334) by @grokspice in #14695
  • fix(cayenne): back off futile bakes under a violated query goal too by @lukekim in #14736
  • fix(snapshots): bootstrap readers from late snapshot publications by @phillipleblanc in #14468
  • fix(cayenne): apply retention_sql to a mode: memory acceleration (fixes #14045) by @claudespice in #14344
  • fix(duckdb, sqlite): keep the first copy of a key a write repeats under on_conflict: drop (fixes #14629) by @claudespice in #14748
  • fix(cache): release a removed dataset's memory when the plan cache discards its plans (fixes #14251) by @claudespice in #14760
  • fix(llms): explain an Anthropic model's refusal of a sampling control instead of passing the bare 400 through (fixes #13564) by @grokspice in #14700
  • fix(snapshots): allow cayenne_file_path and cayenne_metadata_dir on file_format: snapshot datasets (fixes #14696) by @sgrebnov in #14698
  • feat(cayenne): keep the secondary index current across every write by @bjchambers in #14593
  • fix(http): stop a non-2xx response body replacing an accelerated table's rows (fixes #13515) by @claudespice in #13538
  • fix(chat-api): answer a tool-call-only assistant turn instead of panicking (fixes #13207) by @grokspice in #14232
  • fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492) by @claudespice in #14670
  • fix(spiceai, duckdb, sharepoint): name the Spicepod key a missing-parameter error asks for (refs #14446) by @claudespice in #14769
  • fix(cayenne): keep the per-shard PK index when the table-wide index is discarded by @lukekim in #14604
  • Upgrade to DataFusion 55.1 and Arrow 59.3 by @krinart in #14612
  • Upgrade to iceberg-rust to 0.11.0 and DF 55.2 by @sgrebnov in #14771
  • fix(cayenne): write an append into an empty unkeyed table as one load by @phillipleblanc in #14784
  • fix(mysql_replication): don't re-send a live member's delivered commits after a reconnect by @lukekim in #14751
  • fix(acceleration): accept time_format iso8601 when the time_column is already a timestamp by @lukekim in #14777
  • fix(cache): store SQL results that go stale mid-query by @lukekim in #14710
  • fix(cayenne): serve filtered and global maintained aggregates by @lukekim in #14761
  • perf(cayenne): build the checkpoint's tombstone union after releasing the capture locks by @lukekim in #14759
  • fix(cayenne): keep maintenance from deleting files a snapshot is archiving by @sgrebnov in #14789
  • fix(snapshots): never overwrite shared snapshot metadata, and publish more than once to a file:// location by @lukekim in #14582
  • perf(cdc): build deferred change rows in the reader while the apply runs by @lukekim in #14746
  • fix(runtime,data_components): wait for change-data-capture sources to record their positions before shutdown closes the accelerations (fixes #14523) by @grokspice in #14702
  • fix(graphql): detect non-JSON responses and treat gateway errors as retryable by @lukekim in #14781
  • fix(runtime): require keys for s3_auth: key, honor gs:// state location params, and leave newer rate-control state alone by @lukekim in #14584
  • fix(listing): skip zero-byte objects (S3 folder markers) in format-selected listings by @sgrebnov in #14822
  • Avoid unnecessary repartitioning in indexed dynamic-filter joins by @bjchambers in #14799
  • feat(cayenne): keep one row per primary key automatically and deprecate on_conflict by @bjchambers in #14726
  • fix(object-store): stop reading a response body at its first error by @phillipleblanc in #14831
  • fix(listing): keep a partition predicate as a residual filter on the _location fast path by @grokspice in #14790
  • fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC (fixes #14753) by @claudespice in #14840
  • perf(runtime): skip EnsureRequirements while planning a point lookup by @lukekim in #14807
  • feat(acceleration): make Cayenne the default acceleration engine by @phillipleblanc in #14837
  • fix(cayenne): keep a join's LIMIT when the oversized-join rewrite makes it a sort-merge join by @lukekim in #14805
  • fix(cayenne): fail contradictory write settings once, and order NULL times below every time by @bjchambers in #14847
  • fix(runtime): serve an existing acceleration while its source is unavailable (fixes #14610) by @bjchambers in #14624
  • Prune object-store file listing by metadata columns (#14264) by @Jeadie in #14303
  • fix(vortex): fold partition values into the file-pruning predicate; test null-equal joins under mode:file by @lukekim in #14797
  • feat(rate-control): adaptive, bounded and cluster-coordinated HTTP rate controls by @Jeadie in #14143
  • fix(sql): align date_part('dow') with EXTRACT(dow) Sunday=0 by @lukekim in #14796
  • fix(runtime): carry API-key principal into MCP tools/call by @lukekim in #14828
  • fix(llms): un-mute the LLM integration suite and make it pass (fixes #6399; Bedrock deferred to #13558) by @Jeadie in #13656
  • perf(cayenne): hold maintained-aggregate retraction state in a compact shared index by @lukekim in #14762
  • fix(deps): keep a hash join's order once it has a dynamic filter, so vector_search plans again by @Jeadie in #14857
  • feat(cayenne): persist secondary index runs, recorded in the metastore by @bjchambers in #14594
  • fix(cayenne): fence schema statistics and control maintenance tests by @bjchambers in #14856
  • fix(github): retry nested GraphQL pages in place and fail incomplete nested connections by @sgrebnov in #14862
  • fix(cayenne): load an append dataset that has retention_sql and orders versions by time by @phillipleblanc in #14878
  • fix(cdc): Cayenne replication needs only a primary key, not on_conflict by @phillipleblanc in #14880
  • feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
  • fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588
  • fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart in #14851
  • fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice in #14792
  • fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice in #14884
  • fix(anthropic): drop empty text blocks from assistant messages by @Jeadie in #14834
  • fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim in #14882
  • fix(cayenne): a hash join over a secondary index reads the indexed table's files once (fixes #14933) by @lukekim in #14934
  • fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc in #14946
  • fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc in #14939
  • fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim in #14944
  • fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim in #14955
  • fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc in #14942
  • fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc in #14960
  • fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc in #14957
  • fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc in #14956
  • fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim in #14798
  • fix(llms): stream Anthropic responses that open with a thinking block by @krinart in #14920
  • fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov in #14968
  • feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim in #14852
  • fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart in #14972
  • fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart in #14927
  • feat(cayenne): always persist secondary index runs by @sgrebnov in #14979
  • fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers in #14919
  • feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim in #14838
  • fix(search): keep named arguments on search calls nested in rrf() (fixes #14923) by @phillipleblanc in #14975
  • fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart in #14932
  • fix(vortex): don't push down temporal casts Vortex can't evaluate by @sgrebnov in #14985
  • fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers in #14928
  • fix(http): reject a request_path that URL parsing would rewrite by @bjchambers in #14965
  • fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc in #14953
  • Add anthropic_workspace_id param for Anthropic models by @Jeadie in #14829
  • fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim in #14783
  • fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius in #14791
  • fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim in #14806
  • fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim in #15009
  • fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart in #14930
  • fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc in #15019
  • fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim in #14993
  • fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers in #14891
  • feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee in #14844
  • fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc in #15028
  • fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc in #15053
  • fix(runtime): give each runtime.state feature its own folder by @phillipleblanc in #15036
  • fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc in #15054
  • fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov in #15068
  • feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie in #14782
  • fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov in #15069
  • feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov in #15081
  • Fixes for MCP server by @Jeadie in #15041
  • fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice in #15056
  • fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice in #15037
  • perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers in #15072
  • fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart in #14931
  • fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart in #14937
  • fix(decisions): Fixes for /v1/decisions by @Jeadie in #15040
  • perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers in #15076
  • fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers in #15077
  • fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim in #15083
  • fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc in #15096
  • feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie in #14340
  • feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov in #15095
  • fix(runtime): fail recovery of an unsaved running query by @phillipleblanc in #15092
  • perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc in #15099
  • feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers in #15033
  • fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers in #14140
  • fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim in #15106
  • fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim in #15108
  • fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers in #15111
  • feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov in #15117
  • fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov in #15110
  • fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart in #14977
  • feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim in #15042
  • fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim in #15105
  • fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc in #15128
  • fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart in #15126
  • feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc in #15138
  • feat(acceleration): apply changed acceleration settings on restart by @lukekim in #15094
  • fix(models): enable Luna chat and NSQL support by @krinart in #15166
  • fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc in #15177
  • fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov in #15179
  • feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim in #15176
  • fix(iceberg): find the committed Hadoop metadata version by @phillipleblanc in #15142
  • feat(iceberg): append refresh follows Iceberg snapshots so late-arriving rows are not missed by @sgrebnov in #15147
  • Drive accelerator refresh writes on the refresh runtime by @sgrebnov in #15262
  • fix(models): answer ai_* decisions from Bedrock models instead of failing them as refusals by @krinart in #15270
  • fix(cache): coalesce Cayenne cache replacements so caching mode keeps up with sustained misses by @krinart in #15253
  • feat(cluster): lend a distributed task's slot while it waits on an HTTP source by @peasee in #15123
  • fix(cayenne): evict a folded snapshot's cached listing once its superseded files are removed by @bjchambers in #15303
  • fix(cache): delete replaced Cayenne cache entries by primary key, not by scanning the table by @krinart in #15312
  • fix(cayenne): list snapshot directories under the listing fence by @bjchambers in #15304
  • fix(cayenne): checkpoint before a durable CDC write only when a source callback is installed by @phillipleblanc in #15315
  • fix(iceberg): read tables whose last commit is schema-only (ADD/RENAME COLUMN) by @lukekim in #15334
  • fix(runtime): skip the refresh_append_overlap warning for Iceberg append datasets by @sgrebnov in #15340
  • fix(spiced): raise the soft open file limit to the hard limit at startup by @bjchambers in #15339
  • fix(iceberg): keyed DuckDB/SQLite append refreshes read by time so late rows don't overwrite newer versions by @sgrebnov in #15346
  • fix(acceleration): restore the strict high-water mark in append dedupe by @sgrebnov in #15358
  • fix(observability): export a pinned query's execution log records under its trace id by @krinart in #15363
  • fix(caching): let an empty origin response replace a cached HTTP entry by @krinart in #15365
  • feat: one instance identity for every shared-state feature by @phillipleblanc in #15135
  • feat(snapshots): per-dataset snapshot settings; deprecate top-level snapshots and runtime.scheduler state settings by @phillipleblanc in #15136
  • fix(state): make every shared-state write in object storage conditional by @phillipleblanc in #15137
  • fix(mysql): read the shared binlog on a thread of its own by @lukekim in #15181
  • fix(datafusion): keep placeholder type inference linear over deep expressions by @lukekim in #15187
  • fix(sql): cast float literals to decimal through their exact spelling by @lukekim in #15082
  • fix(federation): keep a Spice-only function local without unfederating the plan around it (fixes #13770) by @krinart in #15078
  • fix(snapshots): count every failed snapshot in dataset_acceleration_snapshot_failure_count by @sgrebnov in #15212
  • feat(state): clean up unused shared state and keep 14 days of snapshots by @phillipleblanc in #15199
  • feat(state): add state-store request, conflict and cleanup-delete metrics by @phillipleblanc in #15203
  • fix(sql): keep float token spellings when planning refresh_sql by @lukekim in #15235
  • feat(state): one ownership primitive for scheduler membership, job ownership and snapshot writers by @phillipleblanc in #15196
  • fix(search): reject empty or whitespace-only text on /v1/search by @lukekim in #15119
  • feat(snapshots): keep a dataset's snapshots for 28 days by @phillipleblanc in #15241
  • fix(snapshots): release the snapshot writer lease on graceful shutdown by @sgrebnov in #15256
  • feat(monitoring): snapshot freshness, Cayenne segment cache and pod health panels on the Datadog dashboard by @sgrebnov in #15309
  • fix(observability): report snapshot reader refresh failures in metrics and label them mode=snapshot by @sgrebnov in #15310
  • fix(cayenne): leave the Turso metastore log out of snapshots by @phillipleblanc in #15318
  • fix(object_store_occ): never let a local version repeat an earlier version's ETag by @claudespice in #15437
  • fix(cayenne): complete, version, and validate a dataset's metastore slice (fixes #13241) by @claudespice in #14238
  • fix(acceleration): fail and retry a refresh whose source goes silent mid-stream by @sgrebnov in #15457
  • fix: emit one HELP/TYPE per derived Prometheus summary family by @lukekim in #15416
  • fix: serve append accelerations with snapshots enabled while the source is down by @sgrebnov in #15463
  • perf(cayenne): compact an append-only table's small files without stalling its writers by @lukekim in #15227
  • fix(cayenne): keep lookups fast under sustained replacement on tables without a primary key by @krinart in #15362
  • fix(snapshots): report a snapshot reader that cannot load a newer snapshot by @sgrebnov in #15472
  • fix(cayenne): rematerialize autocommit UPDATE inside write_lock by @lukekim in #15490
  • feat: deprecate non-Cayenne acceleration engines and the DuckDB vector engine by @bjchambers in #15496
  • fix(cayenne): count logical nulls so IS NULL keeps rows in inline and RAM-tier data by @lukekim in #15186
  • fix(cayenne): filter a scan view's rows against the deletions it captured by @lukekim in #15169
  • fix(snapshots): stop a restarted snapshot reader from serving a partial table by @sgrebnov in #15536
  • fix(cluster): keep executor shuffle fetches on cluster mTLS by @phillipleblanc in #15484
  • Add table-bound ChangeSink ownership and backends by @phillipleblanc in #14704
  • chore: Update Turso crate to 0.8.2 by @spiceemma in #15022
  • chore(deps): raise the datafusion-table-providers pin to include the NUMERIC result-column fix by @phillipleblanc in #14254
  • feat(key-index): immutable secondary index runs over compound Arrow keys by @bjchambers in #14592
  • feat(object_store_occ): add transactional WAL and MVCC snapshots by @lukekim in #14732
  • Route CDC refresh and Cayenne cache writes through the table-bound ChangeSink by @phillipleblanc in #14788
  • Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart in #14888
  • fix(duckdb): pull in the bundled Brotli decoder fix (CVE-2026-111497) by @phillipleblanc in #15580

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.3...v2.4.0

Spice v2.4.0-rc.3 (Oct 22, 2026)

ยท 20 min read
Sergei Grebnov
Member of Technical Staff at Spice AI

Spice v2.4.0-rc.3 is now available! ๐Ÿ› ๏ธ

Spice v2.4.0-rc.3 is the third release candidate for v2.4.0, with mostly fixes since rc.2. By default, full and append refreshes now write on their own threads, not on the threads that serve queries. In a Cayenne test, P99 query latency at each refresh commit fell from 130โ€“690 ms to the steady P99 of 44โ€“50 ms. On Unix, the runtime also raises its soft open file limit at startup, up to the hard limit. Before, a few hundred client connections in a container could make Cayenne files fail to open.

Twelve other changes apply to Iceberg append refresh and table reads, Spice Cayenne, Bedrock decisions, OTLP logs, distributed queries, and TLS verification. Some apply only in specific setups. Two fix regressions from earlier v2.4.0 release candidates: Failed to open Vortex file errors after a Cayenne load, and failed MySQL CDC snapshots into Cayenne.

Testers of rc.2 should retest Iceberg append refresh, Cayenne refresh_mode: caching, and ai_* functions on Bedrock models. If you are new to v2.4.0, start with the rc.1 and rc.2 notes for its main features and breaking changes.

Highlights in this release candidate include:

  • Lower query latency during refreshes: By default, full and append refreshes write to the accelerator on dedicated refresh threads. In a Cayenne test, P99 latency at each refresh commit fell from 130โ€“690 ms to the steady P99 of 44โ€“50 ms.
  • Higher open file limit on Unix: The runtime raises its soft open file limit at startup, up to the hard limit. In a container with a soft limit of 1024 and a hard limit of 524288, the runtime raised its soft limit to 524288.

What's New in v2.4.0-rc.3โ€‹

Lower Query Latency During Refreshesโ€‹

Full and append refreshes now write to the accelerator on the dedicated refresh threads. Before this release, the write ran on the threads that also serve HTTP requests. For Cayenne, the write sorts, encodes, checks for duplicate keys, and publishes the data. A large write held those threads, and query latency rose until the write ended.

The change applies to these refresh writes, for all accelerators:

  • refresh_mode: full, and the initial load
  • refresh_mode: append
  • refresh_mode: changes, only when it rebuilds the table from a snapshot

Steady CDC apply, the caching and snapshot refresh modes, and retention deletes do not change. With runtime.params.dedicated_thread_pool: disabled, there are no refresh threads, and refresh writes run as before.

In a test, Cayenne ran a full refresh every 5 minutes on a table of 2.2 million rows and about 8 GB. A client sent 500 point-lookup joins per second over HTTP. Before the change, P99 latency in the 15-second window of each refresh commit was 130โ€“690 ms. After the change, it matched the steady P99 of 44โ€“50 ms. The test did not cover append refreshes or other accelerators. See #15262.

Higher Open File Limit on Unixโ€‹

On Unix systems such as Linux and macOS, the runtime now raises its soft open file limit (RLIMIT_NOFILE) to the hard limit at startup. Client connections, acceleration files, and pipes all count against this one limit. Containers often start the runtime with a soft limit of 1024 and a hard limit of 524288. Before this release, a few hundred client connections could make Cayenne files fail to open. Queries then failed with Unable to open Cayenne acceleration file, or the client saw Connection reset by peer.

  • Measured: On macOS with a soft limit of 256, v2.3.1 failed queries with 176 to 220 idle client connections open. A build with the fix raised the limit to 245760, the kern.maxfilesperproc value of that Mac. It answered queries with 600 connections open.
  • Containers: In a Docker container with --ulimit nofile=1024:524288, the runtime raised its soft limit from 1024 to 524288.
  • macOS: The runtime raises the limit to at most kern.maxfilesperproc.
  • Logs: The runtime logs the result at DEBUG, for example Raised the open file limit from 1024 to 524288. The runtime also logs a failed raise only at DEBUG. Start the runtime with -v or set SPICED_LOG to see it.
  • Windows: Nothing changes.

If the hard limit is also low, the runtime cannot raise the soft limit above it. For this case, the Cayenne error now names the effective limit and says that client connections count against it. It also keeps the Too many open files (os error 24) text from the operating system:

Failed to open Cayenne acceleration file 'โ€ฆ/01a15b2f5c697c01b2a8acdede2d637d_p000_00011.vortex' because the process reached its open file limit of 256, which client connections and all other open files also count against. Raise the limit where spiced is started (`ulimit -n`, or the container's `nofile` limit), or write fewer, larger acceleration files with `cayenne_target_file_size_mb`. See: https://spiceai.org/docs/components/data-accelerators/cayenne#params. Cause: โ€ฆ Too many open files (os error 24)

See #15339.

Iceberg Append Refresh Follows Table Snapshotsโ€‹

An Iceberg dataset with refresh_mode: append and a time_column now loads the rows that each new commit adds, whatever their time values. This applies to Cayenne datasets and to datasets without a primary key. It covers Iceberg tables read through the Iceberg or Glue data connectors. Glue tables in Parquet, CSV, or ORC format still read by time.

Before this release, an append refresh read only the rows with a time_column value later than the stored maximum minus refresh_append_overlap. A writer that commits out of time order can put rows outside that window. A sink with several shards, one of them behind, is an example. Those rows did not load until a full refresh.

The refresh now saves the Iceberg snapshot that it loaded last. The next refresh reads the data files that later snapshots added. An unchanged table scans no data. This change adds no setting:

datasets:
- from: iceberg:https://iceberg-catalog.example.com/v1/namespaces/sales/tables/orders
name: orders
time_column: updated_at
acceleration:
enabled: true
engine: cayenne
mode: file
primary_key: order_id
refresh_mode: append
  • Newest version by time: With a primary key, Cayenne keeps the newest version of each key by time_column. A late row with an older time does not replace a newer stored row.
  • Skipped commits: The refresh skips compactions and commits that add no data file. An append refresh never removes a row.
  • Fallback to reads by time: If the runtime cannot follow the snapshots, it reads by time for one refresh. This happens after an overwrite on a dataset without a primary key, or after an expired snapshot in the interval. A rollback, a branch switch, or a recreated table has the same effect.
  • After a fallback: A refresh that reads by time does not load rows older than refresh_append_overlap. A full refresh loads them. The runtime logs a warning when the reason changes. The new dataset_acceleration_refresh_fallbacks metric counts each fallback by reason.
  • Saved position: File-mode accelerations store the last loaded snapshot with the acceleration data. Other accelerations keep it in memory. A full refresh saves the current snapshot, so the next append refresh continues from it.
  • Computed columns: A dataset that computes columns over the Iceberg table, such as embedding columns, still reads by time.
  • Overlap: refresh_append_overlap now applies only to the first load and to refreshes that read by time. Datasets that follow snapshots no longer log the startup warning that asks for refresh_append_overlap.

With a primary key, a dataset on DuckDB, SQLite, or another accelerator that keeps the last row to arrive still reads by time. This matches rc.2. A primary key that includes the time_column is the exception: such a dataset follows snapshots. On those accelerators, a late row with an older time would replace the newer stored row. If such a dataset has no refresh_append_overlap, the runtime logs this warning when it loads the dataset:

Dataset 'acc' (`engine: duckdb`) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. Use `engine: cayenne` to load every new row, or set `refresh_append_overlap` to how late rows can arrive. See: https://spiceai.org/docs/features/data-acceleration/data-refresh

To load every late row on such a dataset, use engine: cayenne, or set refresh_append_overlap to how late rows can arrive. Append refresh on other data sources does not change. See append refresh, #15147, and #15346.

Spice Cayenne Bug Fixesโ€‹

  • Stale file lists (rare): In rare cases, Cayenne kept a stale file list for a dataset in mode: file. Queries then missed the rows of an appended file with no error. After retention removed a file, every query failed with Failed to open Vortex file โ€ฆ not found instead. Later appends did not repair the list, but a restart did.

    The fault needed a secondary-index build or some queries to list the files while retention or an append changed them. Tests reproduced it only with a delayed file listing. No spiced run has shown it so far. Cayenne now lists files under the same lock that its writers take. See #15304.

  • Vortex file errors after a load: Some keyed Cayenne datasets on network block storage no longer fail every query and refresh with Failed to open Vortex file โ€ฆ not found. This was a regression in earlier v2.4.0 release candidates. The fault needed all of these conditions:

    • mode: file, a primary_key, and key deletion mode (cayenne_deletion_mode: key, or auto when it uses key deletes)
    • a first load or a full refresh with more than one version of some keys
    • network block storage: storage_profile: ebs, or auto when it resolves to EBS or NAS

    Cayenne rewrote the files that held older versions and deleted the originals, but a cached file list still named the deleted files. The data on disk was correct, and a restart cleared the error. Tests reproduced the bug on rc.2. Cayenne now clears the cached list after it deletes the files. See #15303.

  • MySQL CDC snapshots (rc.1 regression): A MySQL dataset with refresh_mode: changes no longer fails its initial snapshot on large tables when a Cayenne file acceleration has maintained aggregates. The error was Bootstrap snapshot error: snapshot row read: โ€ฆ bytes remaining on stream, and the dataset stayed in the Error state.

    Since rc.1, Cayenne ran a checkpoint before each snapshot batch. The checkpoint could rebuild the maintained aggregates with a scan of the whole table. The MySQL reader then waited longer than net_write_timeout, and MySQL closed the connection. Cayenne now runs this checkpoint only when it can hold CDC rows in memory, as before rc.1. In a test with 40 million rows and net_write_timeout=2, the snapshot completed in 36โ€“44 seconds. See #15315.

Cayenne Caching Mode Fixesโ€‹

These fixes apply to Cayenne datasets with refresh_mode: caching.

  • Removed rows served from the cache: An empty origin response now replaces the cached entry of a GET request. Before this release, Cayenne refused that write when the origin started to answer [] for a cached request. Each refusal logged An empty cache replacement requires the complete logical request key. The query that received [] still returned no rows. Later queries could return the rows that the origin had removed:

    • With caching_stale_if_error: enabled, the next origin failure served the removed rows.
    • With caching_stale_while_revalidate_ttl, lookups served the removed rows for the rest of that window. Each lookup started another refresh that Cayenne refused.

    An empty response now replaces only the entry of its own request. Cayenne still refuses an empty response when the lookup filters do not name one request, and it logs An empty cache replacement requires filters that name one request. Entries for a POST request or another query string under the same path stay. DuckDB, SQLite, and Arrow caching datasets still have this fault, because their writer drops empty responses. To get this fix, use engine: cayenne. See #15365.

  • Dropped cache writes under load: Cayenne no longer drops most cache writes above about 100 cache misses per second. Before this release, Cayenne applied each replacement of a cached entry as a separate write. Under load, Cayenne fell behind. The runtime then dropped most writes with Cache write admission timed out. The cache held only part of what it served, so caching_stale_if_error had little data to fall back on. Cayenne now combines replacements and new entries into one write when their requests cannot overlap.

    In a test, a client sent 500 lookups per second over 2.5 million random keys for 300 seconds. Cayenne used mode: file, caching_ttl: 60s, and caching_stale_if_error: enabled. Two runs of each build gave these results:

    MeasureBeforeAfter
    Cache writes dropped121,957โ€“122,5080
    Keys stored15,915โ€“16,624143,356โ€“143,488
    Request P99361โ€“363 ms9 ms

    In a second test, the client cached 60,000 keys, and then the origin stopped. Before the change, 2,932 of 5,001 repeated lookups failed. After the change, 0 of 5,000 failed.

    Cayenne also deletes a replaced entry by its primary key when the request names every primary-key column and the new response has rows. Before, each delete read every stored row, so its cost grew with the cache. In a 60-minute test, the share of CPU for these deletes during origin recovery fell from 3.6% to 0.2%.

    In that phase, P99 latency rose from 303 ms to 433 ms, and throughput fell from 487 to 459 requests per second. The extra writes added compaction work. The test limited the runtime to 8 CPUs and 4 GiB. See #15253 and #15312.

Iceberg Bug Fixesโ€‹

  • Old or uncommitted versions of Hadoop tables: Spice now reads the committed version of a Hadoop-layout Iceberg table. Before this release, a stale version-hint.text kept SQL queries and the Iceberg REST API on an older snapshot. Without a hint, Spice could read the uncommitted metadata file of a writer as the current table. The fix applies to a table read from its root directory, a dataset pinned to version-hint.text, and an Iceberg catalog over a Hadoop warehouse.

    A Hadoop writer commits a version when it renames <uuid>.metadata.json to v<N>.metadata.json. It updates version-hint.text after that, on a best-effort basis. Spice now treats the hint as a starting point and checks v<N+1>, v<N+2>, and later versions, as Iceberg Java does. A metadata file that a bare UUID names no longer counts as a version. See #15142.

  • Tables whose last commit changed only the schema: Queries on an Iceberg table no longer fail when its last commit was ADD COLUMN or RENAME COLUMN with no write after it. Such queries failed with DataInvalid => Column value not found in table. A renamed column did not resolve, an added column was hidden, and a dropped column stayed visible. An Arrow acceleration of the table never loaded, so /v1/ready returned 503.

    A schema-only commit creates no snapshot, and the scan used the schema of the last snapshot. Scans now use the current schema of the table. A scan pinned with snapshot_id keeps the schema of its snapshot. v2.3.2, rc.1, and rc.2 had this bug. A column that someone adds while the runtime runs stays hidden until a restart. Distributed scans pinned to a snapshot still use the schema of that snapshot. See #15334.

AI and Observability Bug Fixesโ€‹

  • Decisions on Bedrock models: ai_if, ai_probability, ai_classify, ai_score, and /v1/decisions now return answers from Bedrock models. Before this release, each such request failed with Evaluation model '<name>' failed: the model refused to answer: . rc.1 and rc.2 had this bug.

    The Bedrock adapter set an empty refusal on every non-streaming reply, and the evaluator read it as a refusal. The adapter no longer sets it, so /v1/chat/completions replies from Bedrock models no longer carry "refusal": "". The evaluator now ignores an empty refusal beside an answer.

    This also covers an OpenAI-compatible model whose upstream is an rc.1 or rc.2 runtime that serves Bedrock. A refusal with text still fails the request. ai_decide uses the same evaluator, but the tests for this fix did not cover it. See #15270.

  • OTLP log trace IDs for pinned queries: A query that a client pins to a trace ID with spice-trace-id or traceparent now exports all of its OTLP log records under that ID. Before this release, records that the runtime logged while the query plan ran carried another trace ID. That ID matched neither the spice-trace-id response header nor the runtime.task_history row. A backend filter on the trace ID missed those records.

    Chat, NSQL, Responses, and decisions requests with a pinned trace ID use the same code path. The tests for this fix covered SQL queries only. This bug did not affect queries without a pinned trace ID. OTLP log export first shipped in rc.2. See observability and #15363.

Distributed Queries Lend Task Slots During HTTP Waitsโ€‹

In a distributed query, a task that waits on a slow HTTP response no longer holds its executor task slot for the whole wait. The task lends the slot to other queries until the response arrives. It then takes a slot back before it continues. Before this release, a few HTTP requests to a slow origin could hold every slot, and other queries waited behind them.

This change needs no configuration. Other data sources and non-distributed queries do not change. In a test, slow HTTP requests filled every executor slot, and a fast query still finished while those requests waited. See #15123.

TLS Certificate Verificationโ€‹

The runtime now uses rustls-webpki 0.103.16 and version 1.0.10 of the webpki-roots and webpki-root-certs root certificate bundles. The bundles follow the current root store of Mozilla. rustls-webpki 0.103.16 rejects a certificate chain that passes through a CA key that upstream no longer trusts. This also applies to cross-signed copies of that key.

The check applies wherever rustls verifies a chain: the bundled roots, rustls-native-certs, and rustls-platform-verifier on Linux. A connection to a server with such a chain now fails certificate verification. Connections through OpenSSL or native-tls use the trust store of the operating system and do not change. See #15374.

Dependency Updatesโ€‹

DependencyUpdate
rustls-webpkiv0.103.15 โ†’ v0.103.16
webpki-roots, webpki-root-certsv1.0.7 โ†’ v1.0.10
Apache Iceberg (spiceai/iceberg-rust, v0.11.0)Fork revision a6b07b8b, which adds the schema fix from apache/iceberg-rust#2904
Apache DataFusion (spiceai/datafusion, based on 55.2.0-rc1)Fork revision 63df5106, which adds the I/O wait hook from spiceai/datafusion#258
Apache Ballista (spiceai/datafusion-ballista, v55)Fork revision 432924cd, which adds task-slot lending from spiceai/datafusion-ballista#76

The upstream versions of DataFusion, Arrow, Iceberg, and Ballista do not change.

Upgradingโ€‹

An upgrade from rc.2 needs no configuration change. TLS connections that rustls verifies now fail if the server's certificate chain passes through a CA that Mozilla no longer trusts. See TLS Certificate Verification.

To upgrade to v2.4.0-rc.3 once the release artifacts are available, use one of the following methods:

CLI:

spice upgrade v2.4.0-rc.3

Homebrew:

brew upgrade spiceai/spiceai/spice

Homebrew follows the version published in the tap. Use the explicit CLI version or Docker tag to select this release candidate.

Docker:

Pull the spiceai/spiceai:2.4.0-rc.3 image:

docker pull spiceai/spiceai:2.4.0-rc.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.3

AWS Marketplace:

Spice is available in the AWS Marketplace. Marketplace availability follows its published versions.

Cookbook Updatesโ€‹

The Spice Cookbook provides recipes to help you get started with Spice.

Contributorsโ€‹

What's Changedโ€‹

Changelogโ€‹

  • fix(iceberg): find the committed Hadoop metadata version by @phillipleblanc in #15142
  • feat(iceberg): append refresh follows Iceberg snapshots so late-arriving rows are not missed by @sgrebnov in #15147
  • Drive accelerator refresh writes on the refresh runtime by @sgrebnov in #15262
  • fix(models): answer ai_* decisions from Bedrock models instead of failing them as refusals by @krinart in #15270
  • fix(cache): coalesce Cayenne cache replacements so caching mode keeps up with sustained misses by @krinart in #15253
  • feat(cluster): lend a distributed task's slot while it waits on an HTTP source by @peasee in #15123
  • fix(cayenne): evict a folded snapshot's cached listing once its superseded files are removed by @bjchambers in #15303
  • fix(cache): delete replaced Cayenne cache entries by primary key, not by scanning the table by @krinart in #15312
  • fix(cayenne): list snapshot directories under the listing fence by @bjchambers in #15304
  • fix(cayenne): checkpoint before a durable CDC write only when a source callback is installed by @phillipleblanc in #15315
  • fix(iceberg): read tables whose last commit is schema-only (ADD/RENAME COLUMN) by @lukekim in #15334
  • fix(runtime): skip the refresh_append_overlap warning for Iceberg append datasets by @sgrebnov in #15340
  • fix(spiced): raise the soft open file limit to the hard limit at startup by @bjchambers in #15339
  • fix(iceberg): keyed DuckDB/SQLite append refreshes read by time so late rows don't overwrite newer versions by @sgrebnov in #15346
  • fix(acceleration): restore the strict high-water mark in append dedupe by @sgrebnov in #15358
  • fix(deps): bump rustls-webpki to 0.103.16 and webpki-roots to 1.0.10 by @phillipleblanc in #15374
  • fix(observability): export a pinned query's execution log records under its trace id by @krinart in #15363
  • fix(caching): let an empty origin response replace a cached HTTP entry by @krinart in #15365

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.4.0-rc.2...v2.4.0-rc.3

Spice v2.4.0-rc.2 (Oct 15, 2026)

ยท 28 min read
Jack Eadie
Member of Technical Staff at Spice AI

Spice v2.4.0-rc.2 is now available! ๐Ÿ”ฅ

Spice v2.4.0-rc.2 is the second release candidate of v2.4.0 for early testing and feedback. It introduces a new OpenAI-compatible POST /v1/decisions API backed by an OpenAI decision model, TypeSafe Jev, or any other LLM in the Spicepod that can answer decisions. Five new SQL functions bring the same AI classification, scoring, and decisions into SQL.

This release also adds Iceberg reads without a catalog and improves acceleration snapshots. Cayenne gains runtime-wide adaptive tuning and applies acceleration configuration changes on restart.

Highlights in this release:

  • AI Decisions API and decisions in SQL: An OpenAI-compatible /v1/decisions API, plus five SQL functions for filtering, classifying, and scoring data with models.
  • Iceberg without a catalog: Read a table directly from its root directory or pin a metadata file to read a fixed version.
  • Shared runtime state: Startup validation and a common location for cluster state, rate control, cache warmup, and acceleration snapshots.
  • SQL results cache: Cache entries expire by TTL and remain available across data refreshes.
  • Cayenne acceleration: Improvements to primary-key loading, secondary indexes, and configuration updates.
  • OpenTelemetry log export: Send runtime logs to any OTLP endpoint as structured log records, and choose which failed queries are logged.

These notes cover changes since v2.4.0-rc.1. Review Breaking Changes before upgrading; upgrades from earlier versions also require the rc.1 migration guidance.

What's New in v2.4.0-rc.2โ€‹

AI Decisions API and decisions in SQLโ€‹

Spice now supports POST /v1/decisions, which is compatible with the OpenAI Decisions API. A decision request asks typed questions about an input. Each answer comes back with its probabilities and a confidence.

Any model in the Spicepod can answer a decision request:

ModelExample fromProbabilities
OpenAI decision modelopenai:gpt-6-lunaThe estimates of the model.
TypeSafe decision modeltypesafe:jevCalibrated by the model.
Any other LLMopenai:gpt-4o-mini, anthropic:claude-haiku-4-5The estimates of the chat model.

OpenAI decision models use the native OpenAI Decisions API. gpt-6-luna also supports chat and /v1/nsql. TypeSafe Jev returns calibrated probabilities and answers decisions only. Other LLMs answer through chat completions and return their own uncalibrated estimates.

models:
- from: openai:gpt-6-luna
name: luna
params:
openai_api_key: ${secrets:OPENAI_API_KEY}
- from: typesafe:jev
name: jev
params:
typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Five new SQL functions make the same decisions available in queries:

FunctionResult
ai_if(input, condition)BOOLEAN: true when the probability of "yes" exceeds 0.5.
ai_probability(input, condition)DOUBLE: a probability from 0 to 1.
ai_classify(input, labels)VARCHAR: one of the supplied labels.
ai_score(input, instructions, levels)DOUBLE: a probability-weighted score.
ai_decide(input, questions)STRUCT: answers, probabilities, and confidence for a set of questions.
SELECT
id,
ai_classify(body, ['billing', 'technical', 'account'], model => 'support_model' ) AS team,
ai_probability(body, 'The customer is requesting a refund', model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI functions for more examples.

Iceberg Without a Catalogโ€‹

Read an Iceberg table directly from object storage or the local filesystem, without configuring a catalog. Set from to the table root using file://, s3://, s3a://, gs://, or gcs://:

datasets:
- from: iceberg:s3://my-bucket/warehouse/db/taxis
name: taxis
params:
iceberg_s3_region: us-east-1

Spice resolves the metadata file on each query and refresh. It supports metadata/version-hint.text as well as a fixed version (a path relative to the table root or a full URL):

datasets:
- from: iceberg:s3://my-bucket/warehouse/db/taxis
name: taxis
params:
iceberg_s3_region: us-east-1
iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Fixed tables are read-only. The earlier metadata_path parameter remains supported with a deprecation warning; see Breaking Changes.

Shared Runtime Stateโ€‹

runtime.state provides a shared, object-store location for cluster state, rate control, SQL results-cache warmup, and acceleration snapshots. Each feature uses its own subdirectory within the provided location.

Acceleration snapshots now default to the snapshots/ subdirectory when runtime.state is configured.

runtime:
state:
location: s3://my-bucket/spice/

datasets:
- from: postgres:orders
name: orders
acceleration:
enabled: true
engine: cayenne
mode: file
snapshots: enabled

A dataset can override the snapshot location, its settings, and the bootstrap behavior on its own acceleration:

datasets:
- from: postgres:events
name: events
acceleration:
snapshots: enabled
snapshots_location: s3://archive-bucket/snapshots/
snapshots_params:
s3_region: eu-west-1
snapshots_bootstrap_on_failure_behavior: fallback

Other changes to shared state:

  • Snapshot queues: An SQS queue that covers the whole runtime.state prefix no longer causes an ERROR log for each scheduler heartbeat. The snapshot reader now deletes messages for the other state folders without a log.

SQL Results Cacheโ€‹

The SQL results cache now expires entries by TTL, independent of acceleration refreshes, CDC batches, and retention passes no longer invalidate cached results.

Review item_ttl and stale_while_revalidate_ttl when upgrading: a successful refresh no longer makes cached results reflect the refreshed data immediately. See Breaking Changes.

Cayenne Accelerationโ€‹

Adaptive tuning is configured once for the runtime. Enable it with runtime.adaptive_tuning and set targets under runtime.params:

runtime:
adaptive_tuning: enabled
params:
target_freshness: 30s

Supported targets are target_replication_lag, target_freshness, target_query_latency, target_convergence_window, and target_qph. Adaptive tuning defaults to disabled. See Breaking Changes and the runtime YAML documentation.

Other Cayenne improvements include:

  • Primary-key loading: Keyed tables build their primary-key filter during the initial load and bulk-load without per-row routing or a whole-table fold.
  • Position deletes: cayenne_deletion_mode: auto can select position deletes for small primary-key tables on local storage on Linux systems with at least eight physical cores.
  • Secondary indexes: Index runs are persisted, indexes remain complete across acceleration snapshots, and CDC maintains indexes for each operation type. Hash joins read each file once when narrowing a scan through an index.
  • Partition handling: Cayenne rejects stored data with an incompatible partition_by, protects keyed appends to partitioned tables during publication, and prepares every partition after a refresh.
  • Storage and query correctness: Cayenne reclaims snapshot directories superseded by a reload, corrects pruning for NaN values, and fixes deletion-index bounds and reload fencing.
  • Snapshot storage growth: A writer with snapshots_trigger: refresh_complete kept one more copy of the table for each refresh. Each snapshot held file deletions, so Cayenne skipped each cleanup pass. Cayenne now runs the cleanup when a snapshot releases its hold. In a 10-minute test with a refresh every 60 seconds, each dataset kept 1 snapshot directory instead of 10.
  • Configuration changes on restart: A change to write settings only, such as cayenne_sort_columns or cayenne_compression_strategy, applies to the existing acceleration. A change to primary_key or partition_by rebuilds the acceleration from the source only under specific conditions. See Cayenne Configuration Updates.

OpenTelemetry Log Exportโ€‹

Spice can now send its logs to an OpenTelemetry Collector, or any OTLP endpoint, as structured OTLP log records. Logs use the same otel_exporter that already sends metrics. A new runtime.query.log_errors setting chooses which failed queries are logged.

runtime:
telemetry:
otel_exporter:
endpoint: otel-collector:4317
signals: [metrics, logs] # new; default [metrics]
query:
log_errors: execution # new; none (default) | execution | all
  • Structured records: Each log line becomes an OTLP log record with its severity, the message as the body, typed fields as attributes, and the logging module as the instrumentation scope. Records share the resource attributes of exported metrics.
  • Trace correlation: A record logged during a query carries the query's trace ID, the same ID returned in spice-trace-id and stored in runtime.task_history.
  • Delivery: Records are batched on a dedicated export runtime, separate from query and HTTP traffic, and flushed at shutdown so the final error and Goodbye! reach the endpoint. For OTLP/HTTP endpoints, logs are sent to <base>/v1/logs.
  • Cluster executors: Executors export their logs with the scheduler's telemetry settings, including headers resolved from the scheduler's secrets.

runtime.query.log_errors controls which failed queries are logged:

ValueWhat is logged
none (default)Existing behavior: a query refused for memory at WARN, and every other failure at DEBUG
executionAlso queries that failed while running, at WARN
allAlso queries rejected before running (syntax and planning errors), at INFO

Each record reads Query failed: <error>, with error_code and datasets fields and the query's trace ID. The SQL text is never logged.

Unknown-column errors are now reported with the correct error_code instead of InternalError, in logs, runtime.task_history, and the query_failures metric.

See OpenTelemetry Logs Export, Exporting logs, and runtime.query.log_errors.

Fixes and Improvementsโ€‹

AI APIs and Modelsโ€‹

  • Responses API: /v1/responses executes Spice tools and honors tool_recursion_limit when streaming. It returns Spice tool calls as mcp_call items in output, allowing clients that manage their own conversation history to replay them.
  • Model controls: /v1/nsql accepts reasoning_effort, and rerank() uses evaluation models with a strategy selected for each query.
  • Anthropic: Streaming handles responses that begin with a thinking block, empty text blocks are dropped, and anthropic_workspace_id is available as a model parameter.
  • OpenAI: Models with an unknown service tier can load and serve requests.
  • Luna: gpt-6-luna and its snapshots support chat, /v1/nsql, and native decisions.

Accelerationโ€‹

  • Caching point lookups: Lookups on a refresh_mode: caching dataset failed with HTTP 400. This occurred when Cayenne accelerated the dataset, the primary key covered the request columns, and the table held two protected snapshots. The point-lookup plan did not combine the partitions of the Cayenne scan. Lookups now return the cached row or fetch it from the source.

Connectorsโ€‹

  • MySQL timestamps: Timestamp literals in queries that the runtime sends to MySQL now keep their fractional seconds.

Dependency Updatesโ€‹

DependencyUpdate
Tursov0.8.2

Breaking Changesโ€‹

Shared State Directory Layout and Startup Checksโ€‹

State written at the runtime.state root by rc.1 is no longer read from that location. Move existing state objects into the appropriate subdirectories before upgrading to retain them:

StateSubdirectory
Cluster schedulercluster/
Rate controlrate-control/
SQL results-cache warmupresults-cache/

Azure locations (abfs:// and abfss://) need one more step. In rc.1, the runtime ignored the path in an Azure URL and wrote state at the container root. For example, with abfs://container/spice/, rc.1 wrote cluster.json at the container root. This release writes spice/cluster/cluster.json. Move Azure state from the container root into the configured path and then into the feature subdirectory.

An explicit runtime.scheduler.state_location retains its earlier layout. Without migration, Spice starts without the state previously stored at the shared root.

The configured store must accept writes and conditional writes. Spice does not become ready until startup validation succeeds and refuses stores with a newer state format. Check storage permissions and compatibility before upgrading.

SQL Results Cache Freshnessโ€‹

A refresh no longer invalidates SQL results-cache entries. Queries can return data from before the latest refresh until item_ttl expires, plus stale_while_revalidate_ttl when configured. Reduce item_ttl if results need to follow source refreshes more closely.

Cayenne Tuning Configurationโ€‹

Replace the Cayenne tuning parameters with runtime settings that apply across datasets and catalogs:

Previous parameterReplacement
cayenne_tuning: adaptiveruntime.adaptive_tuning: enabled
cayenne_tuning: autoruntime.adaptive_tuning: disabled
cayenne_goal_replication_lagruntime.params.target_replication_lag
cayenne_goal_freshnessruntime.params.target_freshness
cayenne_goal_query_latencyruntime.params.target_query_latency
cayenne_goal_convergence_windowruntime.params.target_convergence_window
cayenne_goal_qphruntime.params.target_qph

runtime.adaptive_tuning accepts only enabled or disabled. Other values, including true, fail the Spicepod load.

Pinned Iceberg Tablesโ€‹

Replace the deprecated metadata_path parameter with iceberg_metadata_path. The two settings resolve paths differently. metadata_path uses its value as given. iceberg_metadata_path resolves a value without a URL scheme relative to the table root in from. Convert an absolute local path to a full file:// URL or to a path relative to the table root.

Datasets pinned to a .json metadata file are read-only and fail to load with access: read_write. Remove that setting from pinned datasets before upgrading.

Cayenne Configuration Updatesโ€‹

A restart now applies changed settings to an existing Cayenne acceleration. The result depends on the setting and the dataset:

  • Write settings only: A change to only cayenne_sort_columns, cayenne_cluster_by, cayenne_compression_strategy, cayenne_delta_encoding, or cayenne_shard_key_columns applies to the existing acceleration. Stored rows keep their layout until a refresh or compaction rewrites them.
  • Identity settings: A change to primary_key, partition_by, cayenne_pk_conflict_detection from none to auto, or the data location rebuilds the acceleration from the source. The runtime rebuilds only a dataset with refresh_mode: full, mode: file or file_update, and write_mode: write_through. The source must not be sink:, and must be readable with the schema of the dataset.
  • All other datasets: The acceleration keeps all of its stored settings. This includes write settings that changed in the same restart. The runtime logs a warning that names the change and the fix.

While the runtime cannot read the source, a dataset that qualifies for a rebuild keeps its existing acceleration. A partition_by change or a conflict-detection change cannot use the stored rows, so it fails the dataset instead. Revert the setting, or restart when the source is readable.

A restart alone does not migrate a dataset that does not meet these conditions. Read the startup warning for the dataset before you delete stored data. Allow time for a rebuild when the conditions apply.

Upgradingโ€‹

Once release artifacts are available, select v2.4.0-rc.2 explicitly with the CLI, Docker tag, or Helm chart version.

CLIโ€‹

spice upgrade v2.4.0-rc.2

Dockerโ€‹

docker pull spiceai/spiceai:2.4.0-rc.2

See Docker Hub for available tags.

Helmโ€‹

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.2

Homebrew and AWS Marketplaceโ€‹

Homebrew and AWS Marketplace follow their published versions. To select this release candidate, use an explicit version through the CLI or Docker.

Spice is also available in the AWS Marketplace.

Contributorsโ€‹

Changelogโ€‹

See the full changelog from v2.4.0-rc.1 to v2.4.0-rc.2.

  • feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
  • fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588
  • fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart in #14851
  • fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice in #14792
  • fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-duckdb[file]-2gib.yaml by @github-actions in #14850
  • Update openapi.json by @github-actions in #14858
  • test: strengthen assertions and enforce sign-off coverage by @lukekim in #14740
  • Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart in #14888
  • fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice in #14884
  • fix(anthropic): drop empty text blocks from assistant messages by @Jeadie in #14834
  • fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim in #14882
  • test(sqlite): guard the Date32 literal cast through the SQLite acceleration's own scan (fixes #14491) by @grokspice in #14703
  • fix(cayenne): read each file once when a hash join narrows an indexed table's scan by @lukekim in #14934
  • fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc in #14946
  • fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc in #14939
  • fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim in #14944
  • fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim in #14955
  • fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc in #14942
  • fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc in #14960
  • feat(spicepod): publish pull request reviews by @lukekim in #14964
  • fix(ci): bound the REPL expect-script waits on silence, not total response time (fixes #13711) by @grokspice in #14963
  • fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc in #14957
  • fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc in #14956
  • fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim in #14798
  • fix(ci): provision DuckLake for the ducklake[catalog] schema test, which has never passed by @grokspice in #14952
  • fix(llms): stream Anthropic responses that open with a thinking block (fixes #14909) by @krinart in #14920
  • fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov in #14968
  • feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim in #14852
  • fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart in #14972
  • fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart in #14927
  • feat(cayenne): always persist secondary index runs by @sgrebnov in #14979
  • fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers in #14919
  • feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim in #14838
  • fix(rrf): keep named arguments on nested search calls by @phillipleblanc in #14975
  • fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart in #14932
  • fix(vortex): keep temporal casts Vortex can't evaluate out of the scan by @sgrebnov in #14985
  • fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers in #14928
  • test(huggingface): unblock the merge queue: read imdb's parquet conversion through @~parquet and fail fast on a vanished Hub pin by @grokspice in #14998
  • test(cayenne): link the integration tests into one test binary by @bjchambers in #14976
  • test(vortex): compile the integration tests into a binary the sign-off gate runs by @bjchambers in #14987
  • fix(http): reject a request_path that URL parsing would rewrite by @bjchambers in #14965
  • fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc in #14953
  • Add anthropic_workspace_id param for Anthropic models by @Jeadie in #14829
  • fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim in #14783
  • test: run federated source tests on every supported LTS release line by @lukekim in #14936
  • fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius in #14791
  • fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim in #14806
  • fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim in #15009
  • Improve types methods in runtime-rate-control by @Jeadie in #14573
  • fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart in #14930
  • Update openapi.json by @github-actions in #14908
  • test(cayenne): run the heap-measuring tests in one isolated test binary by @bjchambers in #15010
  • test(vortex): run vortex-datafusion's integration tests in the sign-off gate by @bjchambers in #15001
  • fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc in #15019
  • Update openapi.json by @github-actions in #15018
  • fix(ci): keep the OpenAPI regeneration PR current instead of skipping while one is open by @phillipleblanc in #15023
  • fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim in #14993
  • fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers in #14891
  • chore: Update Turso crate to 0.8.2 by @spiceemma in #15022
  • feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee in #14844
  • ci: apply whole-binary nextest overrides to the module test sets they select by @bjchambers in #15011
  • fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc in #15028
  • test(data_components): compile the integration tests into one test binary by @bjchambers in #14991
  • fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc in #15053
  • fix(runtime): give each runtime.state feature its own folder by @phillipleblanc in #15036
  • test(data_components): make the HTTP provider snapshots independent of the machine by @bjchambers in #15000
  • fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc in #15054
  • test(runtime): drive the cdc_cayenne tests through Cayenne's change sink by @bjchambers in #15003
  • ci(object_store_occ): run the WAL E2E suite on self-hosted runners against disk, MinIO, RustFS and spiceio by @lukekim in #15031
  • test(llms): compile the credential-free tests into the integration binary by @bjchambers in #14990
  • fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov in #15068
  • ci(setup-minio): name the missing secret instead of failing in mc's argument parser by @grokspice in #15073
  • ci: run Substrait Mode B on the Linux runners that can reach the trunk spiced builds by @grokspice in #15074
  • feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie in #14782
  • build(deps): bump the github-actions-dependencies group across 5 directories with 7 updates by @dependabot in #15027
  • fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov in #15069
  • feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov in #15081
  • Fixes for MCP server by @Jeadie in #15041
  • fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice in #15056
  • fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice in #15037
  • perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers in #15072
  • fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart in #14931
  • fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart in #14937
  • fix(decisions): Fixes for /v1/decisions by @Jeadie in #15040
  • perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers in #15076
  • fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers in #15077
  • test(runtime,spiced): compile integration tests into declared binaries, run the ones no CI job ran, and remove the uncalled supersede_initial_load by @bjchambers in #15005
  • fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim in #15083
  • ci(e2e): don't run rust-cache on the macOS build, whose save deletes cargo from sibling runners by @grokspice in #15100
  • feat(testoperator): record per-query search metrics by @Jeadie in #15090
  • fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc in #15096
  • feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie in #14340
  • ci: run merge-queue Mode A TPC-H and LLM integration tests on spiceai-macos-large by @lukekim in #15116
  • docs(cayenne): the per-file fold merges no source files, but may roll by @bjchambers in #15101
  • build(deps): bump async-openai to the fork's merge of upstream main by @krinart in #14980
  • feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov in #15095
  • fix(runtime): fail recovery of an unsaved running query by @phillipleblanc in #15092
  • perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc in #15099
  • test(cayenne): guard per-file statistics sharing for listings outside a scan view by @bjchambers in #15103
  • feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers in #15033
  • fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers in #14140
  • Change codeql runner from spiceai-macos to spiceai-macos-large by @lukekim in #15120
  • fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim in #15106
  • fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim in #15108
  • fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers in #15111
  • fix: Update Search integration test snapshots by @github-actions in #15050
  • test(iceberg): read the Glue fixture unpinned and check the picked metadata file by @sgrebnov in #15115
  • test(cayenne): differential and plan coverage for secondary indexes on partitioned accelerations by @bjchambers in #15113
  • test: answer results-cache misses from the acceleration while the source is down by @bjchambers in #15112
  • ci(lint): fail on test files no target compiles, and document the integration test layout by @bjchambers in #15109
  • feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov in #15117
  • fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov in #15110
  • fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart in #14977
  • feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim in #15042
  • fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim in #15105
  • fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc in #15128
  • fix(test): wait for pkd_ctl to load before reading its bootstrap row in the write-back pk_conflict test by @grokspice in #15102
  • fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart in #15126
  • test(mysql): skip the SQL results cache in the binlog replication end-to-end tests by @grokspice in #15141
  • feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc in #15138
  • feat(acceleration): apply changed acceleration settings on restart by @lukekim in #15094
  • fix(models): enable Luna chat and NSQL support by @krinart in #15166
  • fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc in #15177
  • fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov in #15179
  • feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim in #15176

Spice v2.4.0-rc.1 (Oct 8, 2026)

ยท 64 min read
Sergei Grebnov
Member of Technical Staff at Spice AI

Spice v2.4.0-rc.1 is now available! ๐Ÿ”ฅ

Spice v2.4.0-rc.1 is the first release candidate for v2.4.0. It adds performance improvements, S3 event-driven ingestion, SQL results-cache warmup, and adaptive HTTP rate controls. The release also upgrades to DataFusion v55, Ballista v55, Arrow v59, Vortex v0.86, Iceberg v0.11, and Turso v0.81.

Highlights in v2.4.0-rc.1 include:

What's New in v2.4.0-rc.1โ€‹

Performance & Query Engineโ€‹

This release upgrades Apache DataFusion to the v55.2.0 dependency line and Apache Arrow to v59.3.0. It also upgrades Vortex to v0.86.1, Apache Iceberg to the v0.11.0 fork, and Apache Ballista to v55.

Apache DataFusion v55โ€‹

The DataFusion v55 release adds the following improvements:

  • Sort pushdown and TopK pruning: Parquet scans reevaluate each unread row group as the threshold for ORDER BY ... LIMIT tightens. They skip groups that cannot contribute to the result. TopK pruning also supports multiple sort columns.
  • Join planning: The optimizer converts eligible inner joins to semi joins and removes redundant sides of outer joins. It also orders filter predicates by estimated cost.
  • Aggregation and expressions: Multi-column GROUP BY uses column-oriented storage for all supported key types, such as fixed-size binary UUIDs. More string functions preserve dictionary encoding, and IN lists use specialized paths for small integer types.
  • Parquet reads: Scans skip nested fields that the declared schema does not contain. They also skip page-index reads when a file has no page index.
  • Spill handling: Sorts bound the number of streams in a merge. If memory is insufficient, they spill the largest stream again in smaller batches.
  • SQL diagnostics and functions: EXPLAIN accepts PostgreSQL-style options and FORMAT pgjson. New array functions cover element-wise addition, subtraction, scaling, sums, and averages.

Spice carries these changes through its query plans and preserves statistics across plan wrappers. Cayenne keeps Vortex scans below 10 MiB unsplit to avoid repeated footer reads. See #14612.

Vortex v0.79.0 to v0.86.1โ€‹

Spice v2.3.2 used the Vortex v0.79.0 fork. This upgrade covers the full upstream range from v0.79.0 through v0.86.1, not only the v0.86 changes:

  • Types and arithmetic: Vortex adds native Map arrays, Arrow map conversion, and operations and compression for maps. It also adds union arrays and decimal addition, subtraction, multiplication, and division.
  • Row selection: Piecewise-sequence indices represent contiguous selections without an expanded index for every row. Specialized paths handle chunked arrays, fixed-size lists, variable-length lists, and binary values.
  • Scans and expressions: Layout scans gain a physical plan and expression optimization. Filters can pass through scalar functions with multiple arguments. Filters on wide lists restrict child elements to the selected range, and constant masks can resolve from metadata.
  • Compression: Binary arrays support FSST compression with variable-length offsets. OnPair becomes a stable encoding for reads and gains a storage-backed dictionary. Vortex can convert run-end arrays of lists and decimals.
  • File metadata: Files can store custom metadata. Readers cache decoded type descriptions, and file-format editions define the supported encodings and types.
  • Memory and execution: Builders append nested values in batches, preserve list views, and propagate buffer allocators. Expression rewrites retain unchanged nodes, and row functions support batch execution.
  • Correctness and validation: NULL handling changes cover dictionary predicates, BETWEEN bounds, and empty arrays. Readers add validation for footer offsets, compression metadata, and array indices. Other corrections cover nested scalar hashes, decimal operations, and variable-length binary selections above 4 GiB.

See the complete Vortex v0.79.0 to v0.86.1 changelog for every upstream change. Changes to standalone Vortex bindings and GPU execution do not imply new Spice features.

Apache Iceberg v0.11.0โ€‹

Spice updates its Iceberg reader and catalog integrations from v0.10.1 to the v0.11.0 fork for DataFusion 55 and Arrow 59. The upstream v0.11 changes extend reads and catalog compatibility:

  • Iceberg v3 reads: The reader applies deletion vectors from Puffin files and carries row identifiers and sequence numbers through scans.
  • Metadata-only scans: Metadata-only projections do not require data-column reads. Manifest reads reuse partition types, and positional-delete processing buffers runs instead of allocating a key per row.
  • REST catalogs: Clients negotiate server-advertised endpoints and use session-scoped OAuth2 authentication.

The upgrade also aligns Iceberg storage with OpenDAL v0.58. See #14771.

Apache Ballista v55โ€‹

Spice.ai Enterprise feature. See the Enterprise documentation.

The Ballista v55 upgrade adds virtual-core resource accounting for distributed tasks. A protocol-version handshake detects incompatible schedulers and executors. Cancellation identifies tasks by their task IDs, and task-state records use an append-only model.

Schedulers and executors must run the same version. Upgrade all cluster components together.

S3 Event-Driven Ingestionโ€‹

S3 listing datasets can use refresh_mode: changes with S3 event notifications delivered through SQS:

datasets:
- from: s3://my-bucket/events/
name: events
params:
file_format: parquet
s3_region: us-east-1
s3_auth: iam_role
s3_changes_queue_url: ${ secrets:events_queue_url }
acceleration:
enabled: true
engine: cayenne
mode: file
refresh_mode: changes

New object notifications append the object's rows. A periodic listing backfill covers missed or expired notifications. Each dataset needs its own queue and permissions to read and delete SQS messages, alongside its S3 read/list permissions.

This is object ingestion: removal notifications are ignored by default. Set s3_on_object_removed: rebuild to rebuild the entire prefix when an object is removed. An overwrite of an already applied object key is not ingested again; use new object keys for incoming data. See #14121.

SQL Results-Cache Warmup and Shared Fetchesโ€‹

The SQL results cache can persist query plan shapes and replay them after a dataset's first full or append refresh:

runtime:
caching:
sql_results:
enabled: true
warmup: on_first_refresh

For example, run these queries against an accelerated orders dataset with warmup enabled:

SELECT id, status FROM orders WHERE id = 1;
SELECT id, status FROM orders WHERE id = 2;

Spice records one query shape because only the equality-filter value differs. After a restart and the dataset's first full or append refresh, warmup reruns that shape with distinct id values from the refreshed dataset, filling the cache before the dataset becomes ready. Keep the local .spice/data directory across restarts, or configure runtime.state.location, to retain recorded query shapes.

Warmup replays up to ten distinct recorded query shapes and tries up to 1,024 distinct filter-value combinations per shape, stopping when the cache is full. A dataset stays not ready until warmup completes. Later refreshes do not repeat warmup. The feature requires the default plan-based cache key; cache_key_type: sql is incompatible. See #14178.

Concurrent cache-miss fetches for the same request now share a source fetch. SQL results caching also includes changes for tables updated during query execution and for stale results served while revalidation runs. HTTP dataset caching supports RFC 5861 stale-if-error handling. See #14142, #14710, #14708, and #14134.

SQL, search, and embedding caches now use Spice's sharded cache backend. Existing engine: moka and engine: pingora values are accepted for configuration compatibility but no longer select a backend.

Adaptive HTTP Rate Controlsโ€‹

HTTP rate controls now adapt admission to upstream failures while staying within configured request limits. rate_control_acquire_timeout bounds how long a request waits for capacity and defaults to the connector's client timeout. rate_control_failure_threshold and rate_control_window control the response to upstream failures.

In Spice.ai Enterprise, instances that share a runtime.state.location coordinate per-second and per-minute limits through shared state; OSS instances keep these limits in memory independently. Concurrency limits remain local to each instance. Components sharing an upstream origin must use matching rate-control settings. See HTTP rate-control documentation and #14143.

ORC Files and Object Metadata Queriesโ€‹

Listing connectors support file_format: orc. Object-store listing also uses predicates on metadata columns, including _last_modified, to narrow eligible objects. Queries that select only partition or metadata columns can use those values without reading file contents. See #14075, #14265, #14303, and #14116.

Hugging Face Datasetsโ€‹

The new Hugging Face data connector queries and accelerates datasets from the Hugging Face Hub. It supports Parquet, CSV, TSV, JSON, and ORC files. Public datasets need no credentials:

datasets:
- from: hf://datasets/stanfordnlp/imdb/plain_text/
name: imdb
acceleration:
enabled: true

The location format is hf://datasets/<owner>/<dataset>[@<revision>][/<path>]. A path can select a file, a folder, or a glob. A revision can name a branch, a tag, or a commit. Use @~parquet to select the Hub's automatic Parquet conversion.

Set hf_token for private or gated datasets. Set hf_endpoint for a Hub mirror or proxy. Each scan reads one commit. Refreshes follow the selected branch, but the dataset keeps its registered schema until reload. See #14877.

Automatic Primary-Key Handlingโ€‹

Cayenne keeps one row per primary_key without requiring an on_conflict policy. When a dataset sets time_column, the row with the newest time wins; without it, the last arrival wins. This applies to full and append refreshes as well as writes. Existing explicit conflict policies remain accepted during the deprecation period.

datasets:
- from: s3://my-bucket/orders/
name: orders
time_column: updated_at
params:
file_format: parquet
acceleration:
enabled: true
engine: cayenne
mode: file
primary_key: id

For a read-write dataset whose writes should stay in its acceleration, set acceleration.write_mode: acceleration. Its source need not support writes. This mode cannot be combined with a dataset that refreshes by changes.

Cayenne secondary indexes also support dynamic join filters, and their write handling covers inserts, updates, deletes, and refreshes. See #14726, #14282, and #14593.

PostgreSQL and MySQL replication, and MongoDB change streams, rejected Cayenne datasets that omitted on_conflict. Their validation still required an explicit upsert policy. These sources now accept Cayenne datasets with primary_key alone. See #14880.

For file-mode Cayenne datasets, append refreshes failed with a configuration that combined primary_key, time_column, and retention_sql. The refresh selected a version-resolution path that did not support retention. The refresh now resolves each key's newest version before Cayenne applies retention. See #14878.

Cayenne maintained aggregates now share one compact index of per-key contributions across views. Rebuilds capture concurrent writes and apply them after the scan. The aggregate budget uses 10% of a bounded query pool without the former 512 MiB cap. An unbounded pool retains the 512 MiB budget. See #14762.

Read Datasets from Published Snapshotsโ€‹

Spice.ai Enterprise feature. See the Enterprise documentation.

A dataset can now read published acceleration snapshots directly, without configuring the original source connector:

datasets:
- from: s3://my-bucket/spice/snapshots/orders/
name: orders
params:
file_format: snapshot
s3_region: us-east-1

Spice reads the snapshot metadata to select the engine, restores the published data, and checks for newer snapshots. The dataset is read-only. Snapshot-mode readers can also use S3 notifications delivered through SQS, with periodic checks retained for missed notifications. Each reader process needs its own queue.

This release includes changes to snapshot retries, slow-connection bootstrap, publication metadata, and coordination between snapshot archiving and Cayenne maintenance. Cayenne datasets with a datalake tier cannot create acceleration snapshots. See snapshot documentation, #14529, and #14335.

Connector and Protocol Updatesโ€‹

  • Connector status: ADBC, Databricks Spark Connect and SQL Warehouse, FlightSQL, Glue, HTTP/HTTPS, Iceberg, Localpod, and MongoDB are now Stable data connectors.
  • MCP: support for specification version 2026-07-28, alongside the earlier protocol era. See #14043.
  • GitHub: nested GraphQL pagination and rate-limit pacing updates; the default concurrency limit is now four. See #14179 and #14431.
  • GitHub nested pages: Scans could return incomplete reviews or comments because pagination accepted a short page as complete. The connector now rejects incomplete connections and repeated cursors. It retries a failed nested page without another fetch of the outer page. Datasets with the same token also share the REST quota. See #14862.
  • Iceberg REST: Clients could read an empty dataset because the catalog synthesized metadata without snapshots. The catalog now returns the source metadata for Iceberg datasets that Spice reads unchanged. Other datasets return 400 BadRequestException. Clients need their own storage credentials. See #14588 and Breaking Changes.

Other Fixesโ€‹

The release includes fixes in the following areas; the linked PRs provide details of the changes:

  • Cayenne queries and writes: NULL-aware NOT IN, maintained aggregates, dynamic filters, partition-filter forwarding, memory-mode DML and retention, and primary-key handling across CDC checkpoints. See #14429, #14761, #14370, #14047, and #14344.
  • Startup and reloads: retry datasets with unavailable sources, serve existing accelerations during source outages, and invalidate cached plans and results on catalog replacement or dataset unload. See #14623, #14624, #13914, and #14365.
  • Federation: local evaluation of casts and functions whose source semantics differ, plus filter pushdown changes for DynamoDB, Cosmos DB, and MongoDB. See #14484, #14601, and #14419.
  • HTTP and GraphQL: response-status handling during refresh, retry-budget handling, non-JSON gateway responses, and URL redaction in HTTP errors. See #13538, #14313, #14781, and #14490.
  • Search: deletion of obsolete Elasticsearch chunks, non-finite embedding handling, and source-scan coordination during full-text refresh. See #13960, #13902, and #14663.
  • Models and tools: tool-call-only assistant turns, required tool choices, streaming tool-use completion, model-load diagnostics, and propagation of the API-key principal into MCP tool calls. See #14232, #14460, #14548, and #14828.
  • CDC shutdown and reconnects: source-position recording before accelerations close, and MySQL shared-stream reconnect handling. See #14702 and #14751.
  • SQL weekdays: date_part('dow') aligns with EXTRACT(dow), with Sunday represented as zero. See #14796.
  • Cayenne schema statistics: Decimal bounds could retain an old scale after schema evolution because maintenance published statistics from the previous schema. Cayenne now rejects statistics from an obsolete schema and keeps row counts conservative. See #14856.
  • Vector search: vector_search planning failed after the DataFusion 55 upgrade because a second optimization pass tried to reorder a join with a dynamic filter. The planner now preserves that join's input order. See #14857.

Default accelerator: Datasets and views that enable acceleration without engine now use Cayenne. Explicit engine settings keep their behavior. Storage still defaults to memory. Set mode: file for persistent acceleration. See Breaking Changes for migration guidance.

Dependency Updatesโ€‹

Dependency / ComponentVersion
DataFusionv55.2.0
Apache Arrowv59.3.0
Vortexv0.86.1
Apache Icebergv0.11.0
Apache Ballistav55.0.0
Tursov0.8.1
ADBCv0.24
Rust toolchainv1.98.1

Contributorsโ€‹

Breaking Changesโ€‹

Cayenne is the default accelerator on supported platforms. A dataset or view that omits acceleration.engine switches from Arrow to Cayenne. To retain Arrow, set engine: arrow explicitly before upgrading. Windows keeps Arrow as its default. Persistent datasets should continue to name their engine and use mode: file.

on_conflict is deprecated and scheduled for removal in v3.0. Cayenne automatically keeps one row per primary key, choosing the newest time_column value when configured, or the last arrival otherwise. Existing explicit policies remain supported during the deprecation period. Review those policies before removing them, especially drop or policies that reject conflicting rows.

on_conflict no longer routes writes to the acceleration. For read-write datasets whose writes should stay in the acceleration, use:

acceleration:
enabled: true
engine: cayenne
write_mode: acceleration

This setting cannot be used with refresh_mode: changes, including a connector's default change-stream mode. The default write_through and write_back modes require a writable source.

Cache engine selection is retired. engine: moka and engine: pingora remain accepted but are ignored. Remove the field and use caching_policy to select eviction behavior.

GitHub connector default concurrency is four. Review explicit concurrency settings if your deployment relied on the previous default.

HTTP rate-control waits are bounded by default. Requests waiting for rate-control capacity now time out after the connector's client timeout. Set rate_control_acquire_timeout to a suitable duration, or 0 to retain the previous unbounded wait behavior.

Cayenne acceleration snapshots are unavailable for datalake-tier datasets. Review snapshot settings on datasets using cayenne_datalake_location; this release disables snapshotting that configuration.

Iceberg REST no longer synthesizes metadata for unsupported datasets. GET /v1/namespaces/{namespace}/tables/{table} returns 400 BadRequestException for accelerated datasets, views, and other datasets that Spice does not read unchanged from Iceberg. If a client used this endpoint for schema discovery, use SQL DESCRIBE or information_schema.columns instead. Query these datasets through /v1/sql or Arrow Flight SQL. For eligible Iceberg datasets, clients read the source metadata and need their own storage access. See the Get a table API.

Cookbook Updatesโ€‹

The Spice Cookbook provides recipes to help you get started with Spice.

Upgradingโ€‹

To upgrade to v2.4.0-rc.1 once the release artifacts are available, use one of the following methods:

CLI:

spice upgrade v2.4.0-rc.1

Docker:

Pull the spiceai/spiceai:2.4.0-rc.1 image:

docker pull spiceai/spiceai:2.4.0-rc.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.1

AWS Marketplace:

Spice is available in the AWS Marketplace. Marketplace availability follows its published versions.

What's Changedโ€‹

Changelogโ€‹

  • fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
  • fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
  • fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
  • fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
  • fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
  • fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
  • docs: require a reviewed Enhancement before any user-facing surface changes by @lukekim in #13970
  • ci: upgrade spiceio setup action to v0.9.0 by @lukekim in #13971
  • fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
  • feat(hash-index): verify the bloom filter's block index with Verus by @lukekim in #13777
  • build(deps-dev): bump js-yaml by @dependabot in #13989
  • docs: release notes for v2.3.0 by @bjchambers in #13999
  • fix(ci): drop the dangling substrait-compliance submodule pointer by @bjchambers in #14002
  • fix(cayenne): release the keyset bytes an abandoned PK checkout accounted (fixes #13668) by @grokspice in #13925
  • fix(caching): keep a declared key from disabling eviction and stranding stale rows (fixes #13976) by @bjchambers in #13992
  • Add Substrait compliance harness (IBM TPC-H Mode A + FlightSQL Mode B stub) by @lukekim in #13879
  • ci: skip DynamoDB TPC-H benches in OSS testoperator dispatch by @phillipleblanc in #14016
  • docs: update security support and roadmap after v2.3.0 by @phillipleblanc in #14024
  • chore: post v2.3.0 release housekeeping by @bjchambers in #13969
  • test(adbc): guard BigQuery corpus offline and in release gate by @phillipleblanc in #14017
  • fix(duckdb): deny the regexp built-ins DuckDB cannot answer faithfully (fixes #13809) by @claudespice in #13871
  • fix: Update tpch benchmark snapshots for federated/adbc[bigquery].yaml by @app/github-actions in #13984
  • fix(search): drop the chunks a shortened row no longer produces from a chunked index (refs #13717) by @claudespice in #13960
  • Reduce Cayenne allocations during primary-key validation and filtering by @lukekim in #14009
  • test(forks): guard seven fork patches that had no repo-side test by @krinart in #13996
  • fix(deps): bump arrow-rs to correctly-rounded Decimalโ†’Float cast (closes #13978) by @Jeadie in #14012
  • perf(vortex): defer projection setup on filtered scans until the filter resolves by @bjchambers in #14035
  • endgame: include spiceai/skills versioned release by @lukekim in #14031
  • fix(caching): partition doomed entries at the survivor cutoff so eviction converges (closes #13994) by @Jeadie in #14021
  • fix(deps): bump arrow-rs fork pin for Decimal->Float rounding fix by @Jeadie in #14049
  • Fix subqueries with use_source acceleration by @phillipleblanc in #14022
  • fix(cayenne): make DELETE, UPDATE and INSERT work on a mode: memory acceleration (fixes #12008) by @bjchambers in #14047
  • fix: clarify OpenDAL S3 retry warnings by @lukekim in #14040
  • test(s3): run the parquet-overwrite fixtures on RustFS by @bjchambers in #14067
  • feat(cayenne): materialize multi-reference CTEs on the query path by @lukekim in #13918
  • perf(vortex): answer a constant IN list by probing a set, and falsify it by interval by @bjchambers in #14061
  • perf(vortex): skip a scan split whose zones cannot satisfy the filter by @peasee in #14064
  • fix(arrow): report an exact row count from the indexed point-lookup scan by @krinart in #13972
  • fix(cayenne): apply sort_columns with refresh_mode: full by @peasee in #14063
  • fix(smb): pad an empty CREATE buffer so Samba lists the share root (fixes #13293) by @grokspice in #14050
  • perf(cache): key the logical-plan cache on SQL text, not parameter values by @bjchambers in #14069
  • fix: harden HuggingFace E2E chat against slow Metal generation by @lukekim in #14072
  • fix(cayenne): move accelerator filesystem I/O off Tokio workers by @lukekim in #14073
  • test(chbench): enable CTE materialization and IVM on mysql/postgres adaptive HTAP by @lukekim in #14070
  • ci: run Substrait Mode A TPC-H on pull requests and the merge queue by @lukekim in #14071
  • feat(mcp): support MCP specification 2026-07-28 (dual-era) by @lukekim in #14043
  • fix(ci): call a linker that died of a signal an infrastructure failure, not a check failure (fixes #13614) by @grokspice in #14044
  • fix: Provide temporary directory in docker images by @Jeadie in #14089
  • fix: restore OSS installer, CLI and test workflow coverage by @phillipleblanc in #14025
  • Delete v2.2.0.md by @Jeadie in #14095
  • docs(release): add v2.3.1 release notes by @phillipleblanc in #14094
  • fix(test): allow DELETE in the CORS allow-methods assertion by @claudespice in #14098
  • fix(ci): stop install-protoc unzipping into a shared ~/.local by @lukekim in #14097
  • feat(connectors): add ORC listing format via in-repo FileFormat by @lukekim in #14075
  • fix(cayenne): run snapshot bootstrap check before opening the metastore by @Jeadie in #14093
  • feat(cache): verify the results-cache namespace prefix with Verus by @lukekim in #14074
  • feat(cloud-connect): add a GetDatasets command that answers the /v1/datasets document (refs #13369) by @grokspice in #14051
  • Suppress Cayenne startup logs when no Cayenne dataset is configured by @Jeadie in #14042
  • fix(cache): re-bind parameter values when revalidating a stale result (fixes #14099) by @bjchambers in #14100
  • fix(cluster): support distributed HTTP scans by @phillipleblanc in #14108
  • fix(bigquery): keep ILIKE evaluation local by @phillipleblanc in #14110
  • docs: update security support for v2.3.1 by @phillipleblanc in #14117
  • feat(cayenne): reuse ScanView until write, lag only for read-only CDC by @lukekim in #14055
  • fix(deps): remediate open Dependabot alerts by @phillipleblanc in #14111
  • fix(testoperator): validate results in every scale factor 1 TPC-H, TPC-DS and ClickBench benchmark by @lukekim in #14119
  • fix(cayenne): reject ambiguous metastore paths by @phillipleblanc in #14130
  • perf: serve results-cache hits where the request arrives and cut per-hit overhead by @lukekim in #14103
  • test(runtime): record query previews in the management export test by @lukekim in #14155
  • ci: upgrade spiceio setup action to v0.11.0 by @lukekim in #14152
  • feat(caching): Make caching_stale_if_error RFC-5861 compliant (with stale-if-error header) by @Jeadie in #14134
  • perf(runtime-table): defer cache-eviction key extraction to entries a delete actually names by @Jeadie in #14138
  • fix(runtime): report the acceleration.ready_state deprecation once per component (fixes #13749) by @claudespice in #14006
  • fix(runtime): write the inferred Arrow sort order under the prefixed key its validation accepts (fixes #14023) by @claudespice in #14032
  • fix(connectors): Fix JSON/Orca files using metadata columns by @Jeadie in #14115
  • feat(cayenne): build secondary indexes from indexes in file and memory mode by @phillipleblanc in #14149
  • fix(cayenne): round-trip decimal, binary, and time stats and drop them on scale change by @lukekim in #14139
  • feat(cayenne): cluster warm and datalake tiers, and write full refreshes as key-range files by @lukekim in #14124
  • fix(cayenne): compile the cold-tier pruning test and backtick a doc literal by @lukekim in #14175
  • fix(cayenne): make the crates own targets lint and compile by @phillipleblanc in #14200
  • test(forks): guard five more fork patches, and drop a row that is not fork state by @krinart in #14015
  • perf(cache): promote encoded SQL results to raw after the second decode by @lukekim in #14199
  • fix(turso): build a dictionary column directly so a dictionary over a list, map or boolean value reads back (fixes #13033) by @grokspice in #14181
  • fix(ci): probe the macOS toolchain before reaching for brew in the release builds by @grokspice in #14203
  • fix(ci): skip Metal kernel precompilation in the macOS release build by @grokspice in #14204
  • fix(github): paginate nested GraphQL connections and pace to GitHub's rate limits by @lukekim in #14179
  • fix(vortex): stop an IN list holding a NULL from panicking the scan by @krinart in #14163
  • bench(cayenne): use std::hint::black_box in the clustering bench by @lukekim in #14129
  • fix(runtime-table): stop rebuilding SessionContext on every cache fetch by @Jeadie in #14141
  • test(chbench): cluster order_line, oorder and customer on the adaptive HTAP arms by @lukekim in #14192
  • fix(runtime): count a first load as still loading in the Dataset load summary (fixes #13974) by @claudespice in #14020
  • fix(duckdb): push regexp_count down again at a rendering that counts as the kernel does (fixes #13870) by @claudespice in #14153
  • build: lint and test the sign-off under the same profile as the merge queue by @lukekim in #14180
  • ci: require the Verus proofs in the merge queue as one check by @lukekim in #14189
  • ci: run the longest macOS jobs on their own runner pool by @lukekim in #14229
  • fix(cayenne): build the DELETE sink inside the execution-time write lock (fixes #13828) by @claudespice in #14218
  • build(deps): bump the github-actions-dependencies group across 1 directory with 7 updates by @dependabot in #14231
  • fix(cluster): decide what a Flight message carries by its IPC header, not its body length (refs #13737) by @claudespice in #14212
  • ci: run Mode A TPC-H on merge queue and trunk/release push only by @lukekim in #14247
  • chore(deps): bump spiceai/duckdb-rs to 76655d2f by @lukekim in #14246
  • ci: stop exporting empty AWS and DuckLake endpoints to the schema test by @phillipleblanc in #14194
  • fix(runtime): reload a localpod dataset when the dataset it reads through is reloaded (fixes #3288) by @claudespice in #14208
  • build(deps): bump the aws-sdk group with 3 updates by @dependabot in #14255
  • fix(ci): resolve Homebrew prefix when brew is the spice flock wrapper by @lukekim in #14210
  • chore(deps): raise the datafusion-table-providers pin to include the NUMERIC result-column fix by @phillipleblanc in #14254
  • ci: align the DuckLake bootstrap with the embedded DuckDB, wait for Databricks startup, and stop dispatching legs that cannot pass by @phillipleblanc in #14250
  • fix(runtime): install the Spice function deny-list on the PostgreSQL catalog connector (refs #13664) by @claudespice in #14225
  • perf(cayenne): share inline-cache view entries by Arc instead of cloning them per scan by @krinart in #14191
  • build(deps): bump aws-actions/configure-aws-credentials by @dependabot in #14256
  • ci: stop dispatching the indexed turso TPC-H SF1 tests by @phillipleblanc in #14252
  • fix(catalog): keep the tables registered under an existing schema by @phillipleblanc in #14193
  • feat(cache): Spice sharded cache as the sole LruCache engine by @lukekim in #14206
  • ci: lint GitHub Actions definitions with actionlint, and fix the 73 findings it surfaced by @grokspice in #14223
  • perf(cache): tighten the Raw SQL results-cache serve path by @lukekim in #14205
  • ci: stop triggering the CUDA build on pull requests by @lukekim in #14267
  • ci: run CodeQL on pull requests and the merge queue by @lukekim in #14269
  • Release 2.3.2 release notes by @krinart in #14271
  • chore: make AGENTS.md the canonical agent instructions by @lukekim in #14237
  • ci: run CodeQL Analyze on spiceai-dev-runners by @lukekim in #14281
  • feat: TypeSafe Jev System One evaluation provider by @lukekim in #14215
  • fix(cayenne): tag file statistics bounds as the column's Arrow type (fixes #14280) by @phillipleblanc in #14283
  • ci: install spiceio when the runner has no gh (refs #14233) by @lukekim in #14288
  • test(forks): 11 repo guards by @krinart in #14261
  • docs: add Spice.ai in Action manuscript and companion labs by @lukekim in #13965
  • test(duckdb): add an integration test for the index CTE materialization by @sgrebnov in #13885
  • fix(cayenne): coalesce inline writes into one batch by @sgrebnov in #14279
  • release: Update SECURITY.md and endgame template after 2.3.2 by @peasee in #14293
  • fix(cayenne): count each Arrow allocation once in the inline-cache gauge by @krinart in #14272
  • feat(s3): SQS event-driven changes for refresh_mode: changes by @lukekim in #14121
  • feat(caching): single-flight coalesce concurrent cache-miss fetches by @Jeadie in #14142
  • fix(caching): make caching_stale_if_error detect transient HTTP failures on real schemas by @krinart in #14161
  • Use Cayenne secondary indexes for dynamic join filters by @phillipleblanc in #14282
  • fix(ci): compare DynamoDB sets without relying on element order by @bjchambers in #14328
  • docs(release): Remove QA analytics step from endgame by @peasee in #14329
  • ci: run CodeQL on the merge queue and trunk, not on pull requests by @lukekim in #14290
  • docs(cayenne): reposition the reference, add query serving, re-audit against trunk by @lukekim in #14338
  • docs(endgame): update versioned docs release steps by @ewgenius in #14277
  • test(forks): guard the ballista per-task file-scan restriction by @krinart in #14292
  • fix(cli): surface the full error chain for spice chat connection failures by @krinart in #14289
  • fix(ci): degrade the incomplete-sign-off handler when the runner has no gh (refs #14234) by @claudespice in #14304
  • fix(runtime-table): serialize a direct write against acceleration snapshot creation (fixes #13548) by @claudespice in #14310
  • fix(duckdb): screen regexp_like and regexp_replace as regexp_count is screened (fixes #14148) by @claudespice in #14321
  • fix(http): honor retry budget without an extra origin request by @phillipleblanc in #14313
  • Fix SchemaCastScanExec's schema conversion in fn partition_statistics by @Jeadie in #14258
  • fix(cluster): recognise a Flight keepalive by its empty envelope, not by what its header declares (fixes #13737) by @claudespice in #14327
  • perf(http): defer zero-TTL acceleration lookup until origin failure by @phillipleblanc in #14302
  • fix(cayenne): keep a key visible when it is re-inserted over a stale-insert tombstone by @sgrebnov in #14312
  • perf(cayenne): read only key and filter columns in filtered key deletes by @sgrebnov in #14374
  • fix(cayenne): let small protected-snapshot merges run during a long large-tier merge by @sgrebnov in #14296
  • perf(cayenne): serve primary-key lookups on a freshly loaded table in ~1 ms by @lukekim in #14314
  • fix(cayenne): skip min/max statistics for nested columns (fixes #14368) by @sgrebnov in #14392
  • test(caching): cover SchemaCastScanExec statistics projection by name, retype, and SQL filter by @Jeadie in #14146
  • fix(turso): keep a quantified comparison out of Turso SQL (fixes #14041) by @grokspice in #14393
  • fix(udfs): declare the local_embed dev-dependency the embed tests need (fixes #13092) by @claudespice in #14377
  • perf(cayenne): batch metastore manifest rewrites, upsert in place, and run every write on one writer connection by @lukekim in #14369
  • fix(runtime-table): ignore zero-row batches in stale fallback by @phillipleblanc in #14331
  • Prune object-store file listing by _last_modified predicates by @Jeadie in #14265
  • Reading only partition or metadata columns needlessly scans all file contents by @Jeadie in #14116
  • fix(duckdb): keep a concat over a binary operand out of the federated plan (fixes #13915) by @claudespice in #14333
  • fix(ci): keep the sign-off attribution inside GitHub's 140-character status cap (fixes #14076) by @grokspice in #14390
  • fix(ci): expose a present-but-unlinked cc tool on macOS runners instead of routing it through brew install (fixes #13479) by @grokspice in #14387
  • ci: re-measure integration.yml's job bounds after the archive consolidation (fixes #13429) by @grokspice in #14388
  • fix(ci): start DuckLake's local MinIO from an image that is still published by @grokspice in #14399
  • test(runtime-table): make the metric-scraping refresh tests pass under cargo test by @claudespice in #14381
  • fix(federation): keep a correlated subquery predicate above a join of two sources (refs #8220) by @claudespice in #14372
  • fix(caching): snapshot staleness before the origin fetch for stale_if_error by @Jeadie in #14263
  • perf(snapshots): skip unchanged snapshot metadata with a conditional GET by @sgrebnov in #14409
  • fix(search): prune the rest of a key group from an Elasticsearch chunked index (refs #13717) by @claudespice in #14320
  • fix(bench): derive MySQL's empty-field NULL handling from the column type (refs #13152) by @claudespice in #14345
  • fix(graphql): debit a LIMIT by the rows a page returned, not the declared page size (fixes #14308) by @claudespice in #14353
  • fix(ci): fit retention_oom's retry budget inside its workflow step, and guard the coupling (fixes #13512) by @grokspice in #14389
  • docs: say plainly what Spice is, refresh the README for v2.3, and promote connector statuses by @lukekim in #14410
  • fix(ci): pin, checksum and retry the oha download in the E2E graceful-shutdown jobs by @grokspice in #14418
  • fix(snapshots): resolve snapshot entries relative to the metadata location (#14425) by @sgrebnov in #14426
  • Lower the GitHub connector default concurrency limit to 4 by @lukekim in #14431
  • build(deps): bump nvidia/cuda in the docker-dependencies group by @dependabot in #14439
  • build(deps): bump the aws-sdk group with 3 updates by @dependabot in #14440
  • build(deps): bump the github-actions-dependencies group across 1 directory with 5 updates by @dependabot in #14441
  • test(cayenne): bound the refused-build guard by builds, not by the host's speed by @grokspice in #14424
  • fix(cache): don't report an invalidation cancelled by runtime shutdown as a failure by @grokspice in #14417
  • ci: run remote sign-off on the spiceai-macos pool by @lukekim in #14449
  • ci: run CodeQL Analyze on spiceai-macos by @lukekim in #14442
  • fix(runtime): keep the built-in date_part so both weekday spellings agree (fixes #13920) by @claudespice in #14154
  • fix(cayenne): include the in-memory CDC tier when an overwrite or a retention pass covers the whole table by @lukekim in #14428
  • fix: keep NOT IN null-aware through the join reorder and the Cayenne sort-merge rewrite by @lukekim in #14429
  • fix(cayenne): discard a compaction whose snapshot an overwrite replaced mid-pass by @lukekim in #14432
  • fix(cayenne): stop maintained views, Vortex IN lists and dynamic-filter sharing from returning wrong rows by @lukekim in #14427
  • fix(cayenne): hide spilled rows on CDC upsert fallback by @bjchambers in #14416
  • fix(ci): skip the integration, ADBC, chDB and E2E gate jobs on pull requests instead of passing them (fixes #13841) by @grokspice in #14454
  • fix(cayenne): judge a filtered key delete's captured sources by the index captured with them (refs #13913) by @claudespice in #14455
  • fix(ci): clear the macos-15 image's [email protected] symlink before installing MySQL by @lukekim in #14489
  • ci: produce CodeQL SARIF in the Analyze job on spiceai-macos by @lukekim in #14500
  • fix(cayenne): correctness fixes for the goal-driven adaptive controller, with a closed-loop simulation harness by @lukekim in #14443
  • fix(cdc): keep the newest source commit timestamp on a coalesced change batch by @lukekim in #14463
  • fix(cayenne): draw a sequence for a current-snapshot append so per-key OCC can order it (fixes #13685) by @claudespice in #14360
  • fix(http): name a configured model's load failure instead of reporting it not found (fixes #13303) by @claudespice in #14395
  • fix(duckdb): keep inferred source indexes off change-stream accelerations so upserts commit under concurrent reads (refs #13929) by @claudespice in #14396
  • fix(duckdb): keep a text cast over a binary operand out of the federated plan (fixes #14355) by @claudespice in #14448
  • fix(llms): honor tool_choice required and allowed_tools on mistral.rs-hosted models instead of panicking (fixes #14230) by @claudespice in #14460
  • test(runtime): run the load-error counter test in its own process (fixes #13085) by @claudespice in #14462
  • fix(runtime): stop a replaced dataset configuration's load from registering over the new one (fixes #1458) by @claudespice in #14367
  • fix(install): stop asking for sudo on a first install into a fresh HOME (fixes #14445) by @claudespice in #14495
  • perf(cayenne): serve primary-key point lookups in half the time by @lukekim in #14433
  • fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning by @lukekim in #14430
  • fix(runtime): size every internal DataFusion session from the CPU budget by @bjchambers in #14412
  • fix(runtime): serve nested, zoned and half-float columns from the Iceberg catalog API (fixes #4815) by @claudespice in #14480
  • fix(acceleration): keep cached results when a snapshot refresh finds no newer snapshot by @sgrebnov in #14497
  • fix(runtime): sleep cron tests to the next boundary, not one that already fired (fixes #13759) by @claudespice in #14474
  • fix(ci): derive every lint-rust guard make runs, whatever its separator or recipe layout (fixes #13783) by @claudespice in #14475
  • fix(cayenne): stop the small-file compaction of a position-mode PK table from deadlocking on its own write lock (fixes #14420) by @claudespice in #14481
  • fix(runtime-table): report refresh bytes for the rows each batch holds by @Jeadie in #14469
  • fix(spark,databricks): keep Spice-only functions out of SQL sent to Spark Connect and Databricks SQL Warehouse (refs #13664) by @claudespice in #14498
  • fix(vortex): size a cached footer by what it retains, not its serialized bytes (fixes #12917) by @claudespice in #14502
  • fix(cayenne,telemetry): Register accelerated sink dataset immediately from existing acceleration by @peasee in #13955
  • Update spicepod.yml by @Jeadie in #14533
  • fix(cayenne): keep a rewrite's count inexact when it retains a late protected snapshot (fixes #14383) by @claudespice in #14385
  • fix: Update tpch benchmark snapshots for accelerated/on_zero_results/file[parquet]-cayenne[file]-on_zero_results.yaml by @app/github-actions in #14408
  • build(rust): upgrade toolchain to 1.98.1 by @lukekim in #14560
  • fix(postgres): release a shared slot's hold on a table its publication cannot drop (fixes #13032) by @claudespice in #14527
  • fix(cayenne): size the build side before sort-merging a non-Cayenne outer join by @krinart in #14520
  • Update DF Upgrade template by @krinart in #14526
  • fix(ci): wait for the refresh to invalidate the results cache, not a fixed 3s by @grokspice in #14598
  • fix(deps): move the DataFusion pin past the four unparser fixes, and guard each of them (fixes #13022) by @grokspice in #14570
  • fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does by @lukekim in #14419
  • feat(snapshots): serve a dataset from published snapshots with from: s3://โ€ฆ and file_format: snapshot by @lukekim in #14529
  • perf(cayenne): keep the per-shard PK index across checkpoint flushes and back off futile bakes; fix q17 and two wrong-results races (fixes #14235) by @lukekim in #14555
  • test(chbench): lower the MySQL adaptive pods' CDC linger to 500 ms by @lukekim in #14615
  • fix(graphql): show the parse-failure bytes in JSON decode previews by @Jeadie in #14530
  • Defer source-first cache fallback planning by @phillipleblanc in #14556
  • perf(cayenne): order whole-table rewrites per scan partition, then merge by @bjchambers in #14488
  • fix(ci): tolerate the parquet-rename race's third DuckDB error in the E2E log scan by @grokspice in #14541
  • fix(graphql): retry an inferred credential refusal instead of failing the refresh by @Jeadie in #14539
  • fix(cayenne): serve a widened table's files from their persisted statistics (refs #13829) by @claudespice in #14316
  • fix(acceleration): checkpoint DuckDB's write-ahead log before a snapshot copies its file (fixes #13912) by @claudespice in #14325
  • fix(ci): let E2E cleanup run when the job failed before creating its working directory by @grokspice in #14553
  • Replace unmaintained backoff with a workspace crate by @phillipleblanc in #14621
  • test: read Expect stand-ins with the installed shell by @phillipleblanc in #14634
  • fix(models): apply a tool_choice that forces a call to one round of the tool-use loop (fixes #14459) by @claudespice in #14635
  • fix(runtime): don't end a tool-use stream on a stale tool_calls finish (fixes #13309) by @claudespice in #14548
  • fix(smb): name listed locations under the share so directory datasets load (fixes #14060) by @claudespice in #14549
  • fix(models): name a configured model's load failure in ai() and streaming nsql (fixes #14394) by @claudespice in #14632
  • test(chbench): remove the mysql-cayenne[file]-adaptive-split HTAP arm by @sgrebnov in #14620
  • fix(cayenne): record sharded CDC keys in the table-wide PK index by @lukekim in #14603
  • fix(sqlite): keep TRY_CAST and every cast SQLite answers differently out of the federated plan (fixes #14398) by @claudespice in #14496
  • ci: move remaining Ubuntu 22.04 references to 24.04 by @lukekim in #14677
  • fix(snapshots): retry a failed snapshot attempt with backoff by @sgrebnov in #14676
  • fix(snapshots): large snapshots no longer fail bootstrap on slow connection by @sgrebnov in #14571
  • fix(refresh): start a refresh's source scan only when the sink reads it, so the full-text index keeps its rows (fixes #14619) by @claudespice in #14663
  • test(cayenne): compare suite answers cell by cell against SQLite and chDB by @lukekim in #14473
  • Evaluate any chat model via /v1/evaluate by @lukekim in #14568
  • ci: quarantine the management API integration schedule until its dev OAuth client is reactivated (refs #12376) by @grokspice in #14684
  • ci: stop scheduled Testoperator Ballista benchmarks by @phillipleblanc in #14682
  • fix(cayenne): stop snapshotting datasets with a datalake tier, whose restored copies deleted each other's files by @lukekim in #14583
  • fix(cayenne): count each Arrow allocation once for the mem-tier limit and checkpoint write sizing by @krinart in #14675
  • fix(ci): classify a sign-off that reached its step budget as signalled, so an expiry publishes no verdict (fixes #13843) by @grokspice in #14685
  • fix(runtime): retry a dataset whose source is unreachable at startup (fixes #14609) by @bjchambers in #14623
  • fix(cayenne): archive only the snapshot directories a dataset snapshot references (fixes #14605) by @sgrebnov in #14627
  • ci: route trunk-push macOS builds to the standard pool by @phillipleblanc in #14645
  • ci: install strip for the retention OOM regression test by @phillipleblanc in #14701
  • fix(smb): serve every share on a host from the one store registered for it (fixes #14550) by @grokspice in #14689
  • fix(runtime): discard cached plans and results when a dataset is unloaded (refs #14251) by @claudespice in #14365
  • feat(snapshots): reload snapshot-mode datasets from S3 event notifications (SQS) by @lukekim in #14335
  • fix(cayenne): report runs per size tier when protected-snapshot compaction declines (fixes #13622) by @claudespice in #14476
  • fix(cayenne): re-run a post-write compaction pass a concurrent append asked for (fixes #13906) by @claudespice in #14479
  • fix(http): keep the request URL out of HTTP connector errors (fixes #13534) by @claudespice in #14490
  • fix(runtime): load every localpod dataset that reads from one parent at startup (fixes #13087) by @claudespice in #14532
  • fix(build): share one cargo metadata helper across the lint guards, so a broken cargo is never a violation (fixes #13121) by @claudespice in #14534
  • fix(duckdb): push regexp_replace down again for a one-digit group reference, and refresh the Postgres ClickBench q35 plan (refs #13966) by @claudespice in #14552
  • fix(duckdb): keep a cast into binary out of the federated plan (refs #14397) by @claudespice in #14633
  • fix(runtime): keep finished async query jobs finished, and delete expired ones by @lukekim in #14585
  • fix(federation): keep DataFusion's cast built-ins out of every federated plan (fixes #14444) by @claudespice in #14484
  • test(cayenne): cover a swapped join filter through the sort-merge rewrite end to end (refs #14235) by @claudespice in #14551
  • fix(runtime): decide every Spark/built-in function collision by name, and refuse an undecided one (fixes #14361) by @grokspice in #14400
  • fix(ci): run every bin target's unit tests in the sign-off gate, and give spice connect its own --cloud-region refusal (fixes #13426) by @grokspice in #14406
  • test(data_components): compile the federation unparser guards in every scoped test run (fixes #13625) by @grokspice in #14447
  • fix(runtime-component): skip an inferred Cayenne index on a floating-point column instead of failing the load (fixes #14590) by @grokspice in #14599
  • fix(cache): serve stale cached results during frequent table updates (stale_while_revalidate_ttl) by @sgrebnov in #14708
  • Update Turso to 0.8.1 and retry write conflicts a metastore statement raises by @lukekim in #14680
  • fix(snapshots): stop a snapshot dataset's load when a reload replaces it by @lukekim in #14673
  • fix(cayenne): hand non-partition filters to every partition scan (fixes #12959) by @claudespice in #14370
  • fix(postgres-accel): resolve secret references in the sidecar connection parameters (fixes #13296) by @claudespice in #14544
  • fix(cayenne): drain the in-memory CDC tier before a full rewrite scans it (fixes #14450) by @claudespice in #14486
  • feat(caching): warm SQL results cache on first refresh from persisted plan shapes by @lukekim in #14178
  • fix(ci): retry only the container startup in the zero-retry MySQL CDC tests by @grokspice in #14727
  • feat(key-index): immutable secondary index runs over compound Arrow keys by @bjchambers in #14592
  • fix(federation): keep a fractional-to-integer cast out of the plans pushed to DuckDB, PostgreSQL, MySQL and BigQuery (fixes #14482) by @grokspice in #14601
  • fix(testoperator): give accelerated bench configs a 300s ready_wait and name unready datasets on timeout (fixes #13973) by @claudespice in #14728
  • fix(federation): keep arrow_typeof and its plan-introspection siblings local on every backend (fixes #14334) by @grokspice in #14695
  • test(bench): refresh the tpch_q16 explain snapshots for the null-aware NOT IN plan (fixes #13977) by @claudespice in #14731
  • ci: build trunk-push macOS legs on spiceai-macos-large again, keeping one-at-a-time coalescing by @grokspice in #14735
  • ci(verus): derive the verified crates from cargo metadata; pin vstd once by @bjchambers in #14709
  • docs: raise the test and evidence bar: differential first, exact assertions, performance always measured by @lukekim in #14723
  • test(vortex): wait for a dropped segment cache to be freed before asserting it is gone (fixes #13295) by @claudespice in #14470
  • ci(integration): report each integration part's own result in its required check by @grokspice in #14743
  • fix(cayenne): back off futile bakes under a violated query goal too by @lukekim in #14736
  • feat(object_store_occ): add transactional WAL and MVCC snapshots by @lukekim in #14732
  • fix(snapshots): bootstrap readers from late snapshot publications by @phillipleblanc in #14468
  • fix(cayenne): apply retention_sql to a mode: memory acceleration (fixes #14045) by @claudespice in #14344
  • fix(duckdb, sqlite): keep the first copy of a key a write repeats under on_conflict: drop (fixes #14629) by @claudespice in #14748
  • fix(cache): release a removed dataset's memory when the plan cache discards its plans (fixes #14251) by @claudespice in #14760
  • build(deps): bump the aws-sdk group with 2 updates by @dependabot in #14764
  • fix(llms): explain an Anthropic model's refusal of a sampling control instead of passing the bare 400 through (fixes #13564) by @grokspice in #14700
  • build(deps-dev): bump dompurify by @dependabot in #14613
  • fix(snapshots): allow cayenne_file_path and cayenne_metadata_dir on file_format: snapshot datasets (fixes #14696) by @sgrebnov in #14698
  • test(snapshots): make snapshot integration tests robust under parallel CI runs by @sgrebnov in #14765
  • feat(cayenne): keep the secondary index current across every write by @bjchambers in #14593
  • fix: Update tpch benchmark snapshots for accelerated/file[parquet]-duckdb[memory].yaml by @app/github-actions in #14674
  • Isolate Docker integration fixtures across concurrent processes by @phillipleblanc in #14639
  • ci: run Substrait Mode A TPC-H at SF 1 on spiceai-macos in every merge-queue entry by @lukekim in #14522
  • ci: stop repeating scheduled benchmarks: one source per accelerator, hosted sources weekly, a release commit once by @lukekim in #14681
  • build(deps): bump nvidia/cuda by @dependabot in #14763
  • fix(http): stop a non-2xx response body replacing an accelerated table's rows (fixes #13515) by @claudespice in #13538
  • fix(chat-api): answer a tool-call-only assistant turn instead of panicking (fixes #13207) by @grokspice in #14232
  • fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492) by @claudespice in #14670
  • fix(spiceai, duckdb, sharepoint): name the Spicepod key a missing-parameter error asks for (refs #14446) by @claudespice in #14769
  • Add table-bound ChangeSink ownership and backends by @phillipleblanc in #14704
  • fix(ci): serialize rustup installs on shared macOS runners by @lukekim in #14717
  • fix(cayenne): keep the per-shard PK index when the table-wide index is discarded by @lukekim in #14604
  • Upgrade to DataFusion 55.1 and Arrow 59.3 by @krinart in #14612
  • Update openapi.json by @app/github-actions in #14742
  • build(deps): bump rustls from 0.23.40 to 0.23.45 by @dependabot in #14773
  • Upgrade to iceberg-rust to 0.11.0 and DF 55.2 by @sgrebnov in #14771
  • fix(cayenne): write an append into an empty unkeyed table as one load by @phillipleblanc in #14784
  • fix(mysql_replication): don't re-send a live member's delivered commits after a reconnect by @lukekim in #14751
  • ci(codeql): don't fail the SARIF upload when the merge queue already deleted its ref by @grokspice in #14793
  • fix(acceleration): accept time_format iso8601 when the time_column is already a timestamp by @lukekim in #14777
  • fix(cache): store SQL results that go stale mid-query by @lukekim in #14710
  • fix(cayenne): serve filtered and global maintained aggregates by @lukekim in #14761
  • perf(cayenne): build the checkpoint's tombstone union after releasing the capture locks by @lukekim in #14759
  • ci: keep no artifacts or caches from PR and merge-queue checks, and run gating macOS jobs on spiceai-macos-large by @lukekim in #14804
  • fix(cayenne): keep maintenance from deleting files a snapshot is archiving by @sgrebnov in #14789
  • fix(snapshots): never overwrite shared snapshot metadata, and publish more than once to a file:// location by @lukekim in #14582
  • perf(cdc): build deferred change rows in the reader while the apply runs by @lukekim in #14746
  • fix: Update tpch benchmark snapshots for accelerated/on_zero_results/file[parquet]-cayenne[file]-on_zero_results.yaml by @app/github-actions in #14786
  • test(testoperator): add a cold-start time-to-ready regression test by @phillipleblanc in #14785
  • test(testoperator): make append tests more robust by @sgrebnov in #14817
  • fix(runtime,data_components): wait for change-data-capture sources to record their positions before shutdown closes the accelerations (fixes #14523) by @grokspice in #14702
  • fix(graphql): detect non-JSON responses and treat gateway errors as retryable by @lukekim in #14781
  • fix(runtime): require keys for s3_auth: key, honor gs:// state location params, and leave newer rate-control state alone by @lukekim in #14584
  • build(deps): bump hickory-resolver from 0.26.1 to 0.26.2 by @dependabot in #14774
  • fix(listing): skip zero-byte objects (S3 folder markers) in format-selected listings by @sgrebnov in #14822
  • Avoid unnecessary repartitioning in indexed dynamic-filter joins by @bjchambers in #14799
  • feat(cayenne): keep one row per primary key automatically and deprecate on_conflict by @bjchambers in #14726
  • fix(object-store): stop reading a response body at its first error by @phillipleblanc in #14831
  • fix(ci): use byte-order collation in the benchmark Postgres container (refs #14815) by @sgrebnov in #14836
  • fix(listing): keep a partition predicate as a residual filter on the _location fast path by @grokspice in #14790
  • fix(ci): search only the unixodbc keg, not all of Homebrew's lib, in the macOS release builds by @grokspice in #14846
  • fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC (fixes #14753) by @claudespice in #14840
  • chore: pin the fork at spiceai/datafusion#249 and guard the DataFusion fixes backported from 54 by @krinart in #14800
  • Revert "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492) (#14670)" by @krinart in #14825
  • perf(runtime): skip EnsureRequirements while planning a point lookup by @lukekim in #14807
  • fix(testoperator): report CH-benCH queries with no rows on either side as vacuous, not as matches by @lukekim in #14810
  • feat(acceleration): make Cayenne the default acceleration engine by @phillipleblanc in #14837
  • fix(cayenne): keep a join's LIMIT when the oversized-join rewrite makes it a sort-merge join by @lukekim in #14805
  • test(bigquery): exclude subtrees an empty join build side never runs from the corpus job count (fixes #14848) by @sgrebnov in #14849
  • fix(cayenne): fail contradictory write settings once, and order NULL times below every time by @bjchambers in #14847
  • fix(runtime): serve an existing acceleration while its source is unavailable (fixes #14610) by @bjchambers in #14624
  • Prune object-store file listing by metadata columns (#14264) by @Jeadie in #14303
  • fix(vortex): fold partition values into the file-pruning predicate; test null-equal joins under mode:file by @lukekim in #14797
  • feat(rate-control): adaptive, bounded and cluster-coordinated HTTP rate controls by @Jeadie in #14143
  • fix(sql): align date_part('dow') with EXTRACT(dow) Sunday=0 by @lukekim in #14796
  • fix(runtime): carry API-key principal into MCP tools/call by @lukekim in #14828
  • perf(cayenne): hold maintained-aggregate retraction state in a compact shared index by @lukekim in #14762
  • fix(deps): keep a hash join's order once it has a dynamic filter, so vector_search plans again by @Jeadie in #14857
  • fix(cayenne): fence schema statistics and control maintenance tests by @bjchambers in #14856
  • fix(github): retry nested GraphQL pages in place and fail incomplete nested connections by @sgrebnov in #14862
  • fix(cayenne): load an append dataset that has retention_sql and orders versions by time by @phillipleblanc in #14878
  • fix(cdc): Cayenne replication needs only a primary key, not on_conflict by @phillipleblanc in #14880
  • feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
  • fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.2...v2.4.0-rc.1

Spice v2.3.2 (Sep 22, 2026)

ยท 12 min read
William Croxson
Member of Technical Staff at Spice AI

Spice v2.3.2 is now available! โšก

Spice v2.3.2 makes point lookups and repeated queries faster with Spice Cayenne. SQL results-cache improvements apply to every cached query, regardless of its data source or accelerator. Across two benchmark rounds, the time for a small cache hit fell by 50-52% over HTTP and 44-48% over Flight SQL. For small cache hits, CPU time per server request fell by 58-61% over HTTP and 49-51% over Flight SQL.

Highlights in v2.3.2 include:

  • Indexed Point Lookups โ€” Cayenne now honors indexes, so a lookup reads the matching rows instead of scanning
  • Cayenne Data Layout โ€” cayenne_cluster_by groups related rows together across storage tiers
  • Faster Queries After a Full Refresh โ€” refreshed tables are written so that filtered queries read fewer files
  • Faster Cached Responses โ€” these improvements apply to every query in the SQL results cache, regardless of its data source or accelerator. For small results, cache-hit time fell by 44-52%
  • Faster Repeat Queries โ€” Cayenne reuses its prepared view of a table until the data changes
  • Localpod Fix โ€” a localpod dataset no longer serves data its source has replaced
  • Catalog Fix โ€” tables no longer go missing when datasets load at the same time
  • Safer Cayenne Configuration โ€” an ambiguous metastore configuration is reported instead of appearing as lost data
  • Distributed HTTP Queries โ€” async distributed queries can read HTTP datasets

What's New in v2.3.2โ€‹

Faster Point Lookups with Cayenne Indexesโ€‹

Every other accelerator already accepted indexes, and Cayenne logged that it ignored them. Cayenne now builds an index for each entry, in both mode: file and mode: memory:

acceleration:
engine: cayenne
mode: file # or memory
indexes:
'(TenantId, ServiceId)': enabled

A query that pins every column of an index to a value, such as WHERE TenantId = 7 AND ServiceId = 'a', now reads the matching rows directly. Without an index, Cayenne can only skip files whose minimum and maximum values rule the key out, which rarely helps when related rows are spread across the table โ€” a lookup on a 5.4M-row test dataset had to open about half its files for nearly every key.

Indexes only reduce what a query reads, so results are identical either way. A query that uses a range, an IN list, an OR, or a cast on the indexed column reads the table as before. Index definitions are not stored with the table, so adding or removing one takes effect the next time the dataset loads. Floating-point columns cannot be indexed and are reported at load time.

EXPLAIN shows whether a query used an index, and the cayenne_lookup_index_probe_total metric counts lookups by outcome.

Control How Cayenne Lays Out Dataโ€‹

cayenne_cluster_by stores rows with similar values near each other, so a filtered query reads fewer files. It applies to every storage tier, replacing cayenne_datalake_clustering_columns, which affected only the coldest tier.

acceleration:
engine: cayenne
params:
cayenne_cluster_by: 'tenant_id, event_time'

CREATE TABLE ... CLUSTER BY (column, ...) is also supported, and several cases it previously rejected โ€” a single column in parentheses, and names whose capitalization differs from the column definition โ€” now work. A column name that does not exist is reported when the dataset loads rather than failing later.

Faster Queries After a Full Refreshโ€‹

A full refresh previously spread each key across every file it wrote, so a filtered query had to open all of them even when it wanted a single row. A refreshed table is now written so that each file holds a distinct range of the data, and queries filtering on that range read only the files that can match.

This needs no configuration and applies to any dataset whose accelerated table is replaced by a refresh. Datasets that already set cayenne_sort_columns or cayenne_cluster_by keep their existing layout, and the very first load is unchanged.

Faster Cached Query Responsesโ€‹

Before this release, the runtime planned a query before it checked the SQL results cache. It also repeated other work that a cache hit does not need. The runtime now checks the cache first and returns cached answers directly. These improvements apply to every cached query, regardless of its data source or accelerator.

Across rounds of benchmarking queries that returned a seven-row GROUP BY result, cache-hit time over HTTP fell by 50-52%. Over Flight SQL, cache-hit time fell by 44-48%. In one HTTP round, cache-hit time fell from 63.8 ยตs to 30.8 ยตs. In one round over Flight SQL, it fell from 55.3 ยตs to 30.9 ยตs.

With task history enabled, the server used 125 ยตs of CPU time for a small HTTP hit before the change. It used 49 ยตs after the change. For a small result over Flight SQL, the server used 330 ยตs before the change. It used 168 ยตs after the change. These changes reduced CPU time by 61% and 49%, respectively.

For a wide result over HTTP, the server used 241 ยตs of CPU time before the change. It used 116 ยตs after the change. This is a 52% reduction. Wide results did not reduce the runtime's cache-hit time in those runs. Some measurements were slower.

Raw cache entries now share each stored batch with streaming JSON HTTP and Flight SQL responses. Buffered HTTP formats โ€” including CSV, plain text, vnd.* envelopes, and JSON with union columns โ€” still clone batches while encoding.

A stream benchmark with eight 20-column batches fell from 1.8722 ยตs to 226.23 ns. This is an 8.3x speedup. One 200-column batch fell from 2.0190 ยตs to 131.94 ns. This is a 15x speedup. These figures measure the construction and full consumption of the stream. They do not measure query latency from start to finish.

For compressed entries, the first read keeps the entry compressed. The second read promotes it to raw data when the raw size fits the cache. Later reads avoid decompression. In a cache benchmark, a later read took 17.1-17.8 ยตs across three payload sizes. A decode took 39.1-154.8 ยตs. Cached answers, expiry times, and cache settings are unchanged.

Faster Repeat Queries on Cayenneโ€‹

Cayenne prepares a view of a table's current contents before it can answer a query. It now reuses that preparation until the data actually changes, instead of rebuilding it, so repeated queries against a table that is not being written return faster and stop querying the metastore entirely once warm.

Datasets fed by continuous change data capture โ€” a cdc: or debezium: source, or refresh_mode: changes โ€” reuse the preparation for up to one second so a burst of incoming changes can share it. Datasets that accept writes always see their own writes immediately. No configuration is required.

Better Pruning for Decimal, Binary, and Time Columnsโ€‹

Cayenne did not record minimum and maximum values for decimal, binary, time, and 16-bit float columns, so queries filtering on them could not skip files and had to read more data than necessary. Those columns now carry the same statistics as every other type, and clustering on a decimal column works as intended.

After a change that widens a decimal column's scale, Cayenne discards the affected statistics and rebuilds them, so queries read a little more until that completes. Results are unaffected.

Localpod Datasets Stay in Sync with Their Sourceโ€‹

A localpod dataset reads through another dataset in the same Spicepod. When the source dataset was reloaded โ€” because its configuration changed โ€” the localpod dataset kept reading the replaced copy, so it answered with data the source no longer had, and both copies kept refreshing.

A localpod dataset now reloads whenever the dataset it reads through does, including through several levels of chaining, and its cached results are cleared at the same time. This also works when the source is named with its full path, such as localpod:spice.public.parent. Fixes #3288.

Catalog Tables No Longer Go Missingโ€‹

When two datasets in the same schema loaded at the same time, one could be silently discarded, and every later query against it failed with Table not found. Both datasets reported that they had loaded successfully, and which one went missing varied between restarts. Datasets that share a schema now always both register.

Cayenne Detects an Ambiguous Metastore at Startupโ€‹

Cayenne datasets that each set a different cayenne_file_path, without a shared cayenne_metadata_dir, could open the wrong metadata directory after a restart. Cayenne then started up empty even though the data was still on disk, which looks like a total loss of accelerated data.

This configuration is now rejected at startup, naming each dataset and path involved and linking to the documentation. If a Spicepod uses several Cayenne data paths, set the same cayenne_metadata_dir on each dataset before upgrading.

Distributed Queries over HTTP Datasetsโ€‹

An async distributed query submitted to /v1/queries failed if it read an unaccelerated HTTP dataset. These queries now run, with credentials, headers, and pagination behaving as they do for a non-distributed query. Fixes #14104.

Other Fixesโ€‹

  • Partitioned datasets: a full refresh of a partitioned dataset only replaced the partitions the new data reached, so rows deleted at the source stayed queryable, and a refresh that returned no rows changed nothing. Every partition is now replaced.
  • Iceberg write-through: a write to a partitioned dataset could deadlock against a refresh running at the same time, leaving both waiting.
  • BigQuery: a case-insensitive LIKE could fail the query or return the wrong rows, because BigQuery has no ILIKE. Spice now evaluates it locally. Ordinary LIKE is unchanged.
  • PostgreSQL: timestamps written back to PostgreSQL lost everything below the second. Microseconds are now preserved.

Dependency Updatesโ€‹

No crate versions changed in this release. DataFusion remains at v54.1.0, Arrow remains at v58.3.0, and Vortex remains at v0.79.0.

Spice updates two fork revisions: datafusion-table-providers for the BigQuery and PostgreSQL fixes above, and duckdb-rs so the bundled DuckDB builds against the macOS 27 SDK.

Contributorsโ€‹

Breaking Changesโ€‹

cayenne_datalake_clustering_columns is replaced by cayenne_cluster_by. Rename the parameter before upgrading. The new one groups data on every storage tier, not only the coldest:

acceleration:
engine: cayenne
params:
cayenne_cluster_by: 'tenant_id, event_time'

A dataset that sets both cayenne_sort_columns and a cluster key is now rejected when it loads. Remove cayenne_sort_columns to keep the cluster key.

Cayenne datasets using several data paths must share a metastore. If your Cayenne datasets set different cayenne_file_path values, set the same cayenne_metadata_dir on each one before upgrading. Spice now refuses to start on this configuration instead of risking an empty-looking acceleration:

acceleration:
engine: cayenne
params:
cayenne_file_path: /mnt/a/cayenne
cayenne_metadata_dir: /mnt/shared/cayenne-metadata

Cookbook Updatesโ€‹

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgradingโ€‹

To upgrade to v2.3.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.3.2 image:

docker pull spiceai/spiceai:2.3.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.2

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changedโ€‹

Changelogโ€‹

  • fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
  • feat(cayenne): reuse ScanView until write, lag only for read-only CDC by @lukekim in #14055
  • perf: serve results-cache hits where the request arrives and cut per-hit overhead by @lukekim in #14103
  • fix(cluster): support distributed HTTP scans by @phillipleblanc in #14108
  • fix(bigquery): keep ILIKE evaluation local by @phillipleblanc in #14110
  • feat(cayenne): cluster warm and datalake tiers, and write full refreshes as key-range files by @lukekim in #14124
  • fix(cayenne): reject ambiguous metastore paths by @phillipleblanc in #14130
  • fix(cayenne): round-trip decimal, binary, and time stats and drop them on scale change by @lukekim in #14139
  • feat(cayenne): build secondary indexes from indexes in file and memory mode by @phillipleblanc in #14149
  • fix(catalog): keep the tables registered under an existing schema by @phillipleblanc in #14193
  • ci: stop exporting empty AWS and DuckLake endpoints to the schema test by @phillipleblanc in #14194
  • perf(cache): promote encoded SQL results to raw after the second decode by @lukekim in #14199
  • perf(cache): tighten the Raw SQL results-cache serve path by @lukekim in #14205
  • fix(runtime): reload a localpod dataset when the dataset it reads through is reloaded (fixes #3288) by @claudespice in #14208

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.1...v2.3.2

Spice v2.3.1 (Sep 15, 2026)

ยท 8 min read
Phillip LeBlanc
Co-Founder and CTO of Spice AI

Spice v2.3.1 is now available! ๐Ÿ› ๏ธ

Spice v2.3.1 is a patch release that improves performance of filtered Vortex scans, cache eviction, and query planning. It also improves installation reliability, subquery fallback, numeric conversion, and Cayenne writes.

Highlights in v2.3.1 include:

What's New in v2.3.1โ€‹

Cayenne Applies sort_columns During Full Refreshesโ€‹

Cayenne previously applied sort_columns only during compaction. A dataset with refresh_mode: full replaces its table, so it did not run that compaction. Cayenne now sorts the replacement stream before it writes the refreshed table, and the ordered files let min/max statistics exclude unrelated files from selective queries.

In the reported TPC-H SF1 benchmark, point-lookup p50 latency improved by 16%, and the same query used 37% less CPU. The sort adds work to each full refresh, so only datasets that configure sort_columns pay this cost. See Full Refresh for full-refresh configuration.

Faster Vortex Scansโ€‹

Vortex now waits for a pushed-down filter before it prepares projected column readers, so a split with no matching rows does not prepare those readers. Selective queries in the reported benchmarks read 45% to 58% fewer bytes, and bandwidth-constrained storage showed the largest latency improvements.

Vortex also checks each scan split against its zone statistics, and it returns an empty stream when the split cannot contain a matching row. In the reported TPC-H SF1 benchmark, point-lookup p50 latency improved by 12%, and CPU use per query fell by 53%. Nonselective scans do not use these shortcuts and remain unchanged within the reported run-to-run variation.

Large IN Lists Run Fasterโ€‹

Vortex previously evaluated a constant IN list once for every row and every list item, so runtime grew with both dimensions. Vortex now builds a set once and probes it in one pass, and it uses value intervals to reject files before a scan.

A 32,768-value list over 1,048,576 rows fell from 44.57 seconds to 225 milliseconds in the reported benchmark. Small lists keep their existing evaluation path.

Parameterized Queries Reuse Logical Plansโ€‹

The logical-plan cache previously included each bound parameter value in its key, so each value tuple created another entry for the same SQL text. The cache now stores an unbound plan for each SQL statement, and Spice binds the current values after it reads the plan. The SQL results cache still includes parameter values in its keys, so a result only matches a query with the same values.

The reported point-lookup benchmark improved p50 latency by 12.4% and p99 latency by 31.7%, and parameterized traffic creates fewer duplicate entries. Indexed primary-key lookups now report an exact row count to the query planner, so DataFusion can avoid an unnecessary repartition for a single-row probe.

Caching Accelerations Evict Reliablyโ€‹

A refresh_mode: caching dataset can use size and row-count limits, and two defects could let the cache exceed these limits. A declared primary_key could expand the eviction key to the entire stored row, and an index marked as unique had the same fault. A Map column in the expanded key then made every sweep fail. Spice now keeps the group key limited to the request columns, and a refreshed entry also removes rows that disappeared from its latest response.

Many responses can share the same one-second fetch timestamp. The previous eviction predicate could then remove only 512 entries when this tie crossed the survivor boundary. The eviction sweep now combines a timestamp range with named boundary entries. The range removes the older bulk, so eviction converges under sustained ingestion.

Federation and Runtime Improvementsโ€‹

  • DuckDB and SQLite accelerations now handle more subqueries with on_zero_results: use_source. This includes mixed fallback policies and ANY or ALL comparisons.
  • Decimal-to-float casts now preserve the expected rounded value after a federated aggregate. For example, a PostgreSQL average of 47.5 returns 47.5.
  • Cayenne uses asynchronous filesystem calls from its asynchronous code. A slow data directory does not block the Tokio runtime at these call sites.

Installer and Container Reliabilityโ€‹

The runtime installer now retries unsuccessful HTTP responses and invalid archives, and it accepts an archive only when it contains a regular spiced executable. The installer sends a configured token to GitHub release requests, and it does not send that token to runtime download hosts. On native Windows, spice upgrade now rejects the unsupported runtime upgrade instead of continuing with a partial upgrade.

The scratch-based standard and release images now provide a writable temporary directory at /app/tmp, which supports operations such as acceleration snapshot uploads.

Cayenne Memory-Mode DML Works as Expectedโ€‹

A Cayenne acceleration uses mode: memory by default. Before this release, filtered DML statements did not change its in-memory rows as requested.

  • A filtered DELETE reported zero affected rows and left each matching row in place.
  • An UPDATE could create a duplicate live row for the same primary key.
  • An INSERT over an existing primary key could keep both versions of the row.

DELETE, UPDATE, and INSERT now update the in-memory tier. The append path also enforces each declared primary_key according to its on_conflict setting. The SQL DML reference lists the supported statements.

Dependency Updatesโ€‹

No dependency changes its version in this release. DataFusion remains at v54.1.0, Arrow remains at v58.3.0, and Vortex remains at v0.79.0. Spice updates the Arrow and DataFusion fork revisions for the Decimal-to-Float rounding fix and the scan improvements described above.

Contributorsโ€‹

Breaking Changesโ€‹

No breaking changes.

Cookbook Updatesโ€‹

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgradingโ€‹

To upgrade to v2.3.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.3.1 image:

docker pull spiceai/spiceai:2.3.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changedโ€‹

Changelogโ€‹

  • fix(arrow): report an exact row count from the indexed point-lookup scan by @krinart in #13972
  • fix(caching): keep a declared key from disabling eviction and stranding stale rows (fixes #13976) by @bjchambers in #13992
  • fix(deps): bump arrow-rs to correctly-rounded Decimalโ†’Float cast (closes #13978) by @Jeadie in #14012
  • fix(caching): partition doomed entries at the survivor cutoff so eviction converges (closes #13994) by @Jeadie in #14021
  • Fix subqueries with use_source acceleration by @phillipleblanc in #14022
  • fix: restore OSS installer, CLI and test workflow coverage by @phillipleblanc in #14025
  • perf(vortex): defer projection setup on filtered scans until the filter resolves by @bjchambers in #14035
  • fix(cayenne): make DELETE, UPDATE and INSERT work on a mode: memory acceleration (fixes #12008) by @bjchambers in #14047
  • fix(deps): bump arrow-rs fork pin for Decimal->Float rounding fix by @Jeadie in #14049
  • perf(vortex): answer a constant IN list by probing a set, and falsify it by interval by @bjchambers in #14061
  • fix(cayenne): apply sort_columns with refresh_mode: full by @peasee in #14063
  • perf(vortex): skip a scan split whose zones cannot satisfy the filter by @peasee in #14064
  • perf(cache): key the logical-plan cache on SQL text, not parameter values by @bjchambers in #14069
  • fix(cayenne): move accelerator filesystem I/O off Tokio workers by @lukekim in #14073
  • fix: Provide temporary directory in docker images by @Jeadie in #14089

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.0...v2.3.1

Spice v2.3.0 (Sep 10, 2026)

ยท 42 min read
Phillip LeBlanc
Co-Founder and CTO of Spice AI

Spice v2.3.0 brings performance improvements, broader query federation, and expanded data connector capabilities. The release improves cache reuse and reduces memory use for cached SQL results. It also extends BigQuery support and adds GitHub review, release, and repository data for SQL analysis. Google models now use Vertex AI, so deployments with credentials for Google AI Studio require migration.

Highlights in v2.3.0 include:

What's New in v2.3.0โ€‹

SQL Federation Improvementsโ€‹

This release improves SQL translation and function handling for accelerated and federated datasets. The following bug fixes cover string functions, NULL handling, correlated subqueries, and timezone declarations.

  • A trim call failed on DuckDB, SQLite, and MySQL because DataFusion emitted its canonical name, btrim. DuckDB now receives trim, with an explicit space argument for the one-argument form. The native SQLite and MySQL paths evaluate btrim locally.
  • DuckDB federation now lower-cases to_hex output to match local evaluation.
  • DuckDB federation now decodes sha256 output into a 32-byte digest instead of a hexadecimal string.
  • concat uses || on DuckDB to consistently propagate NULL arguments.
  • inner_product now returns NULL for an undefined dot product on DuckDB for consistency.
  • Operations that DuckDB cannot handle (such as regex with U or R flags) are executed locally.
  • User-defined functions registered after startup now execute locally.
  • Catalog connectors now use the same list of local Spice-only functions as the data connectors.
  • Two EXISTS shapes emitted SQL which evaluated the correlation over the whole relation, so the bound selected nothing. A semi or mark join then reported a match on a row the plan never read. Both shapes now refuse pushdown and run locally.
  • DuckDB labels a TIMESTAMPTZ column with the connection's own timezone. The connector built its pool and never pinned that setting. A dataset's schema therefore carried the host timezone, and the same query returned different rows on different machines. A connector session is now pinned to UTC.

BigQuery Federationโ€‹

BigQuery federation runs more query shapes as one remote job.

  • Temporal expressions, recursive CTEs, and integer division now keep their results and stay in one federated statement. These shapes previously failed remotely, split into several queries, or produced wrong cohort boundaries.
  • Three more statement shapes now run. The first is a grouped query that projects a wrapped form of its grouping expression. The second is a query whose federated tables all sit inside a correlated subquery. The third is any aggregate window function.
  • A query that reads BigQuery tables from several datasets of one project now runs as one BigQuery query. It also no longer returns rows from the wrong dataset when two datasets hold a table of the same name.
  • A dataset that stores JSON in a STRING column now pushes down scalar json_as_text expressions and json_get(...) IS NULL checks.
  • The built-in date_trunc is preserved, and the dialect and the federation policy now agree on which aggregate and window calls are eligible.
  • regexp_match null checks federate safely.
  • A distinct union now renders as UNION DISTINCT. BigQuery rejects a bare UNION, so such a query failed outright. UNION ALL is unchanged.
  • A numbering function such as ROW_NUMBER no longer carries a window frame, which BigQuery rejects. An aggregate window function keeps its frame.
  • Percentile functions and grouping keys now render in the form BigQuery accepts. A reported 29-statement workload that failed on these shapes now runs in full.
  • array_element translates a non-negative integer literal index that fits in Int64 as SAFE_ORDINAL. Other indexes evaluate locally. This preserves DataFusion's end-relative semantics for negative indexes.

Cancellation: A BigQuery query whose client goes away now stops. The BigQuery job ends as cancelled, and the pooled connection returns immediately. Before this release the query ran to completion, and the connection stayed busy for its whole duration. Enough cancellations left an application unable to query at all.

The Caching Accelerator Accepts Explicit Limitsโ€‹

A refresh_mode: caching accelerator had nothing bounding what it held. Retention was derived from caching_ttl plus caching_stale_while_revalidate_ttl, and only when caching_stale_if_error was disabled. A dataset that set caching_stale_if_error therefore got no policy at all, and nothing was ever evicted. Nothing capped the acceleration by size or by count either.

Two settings now bound the acceleration. Each one refuses an unparseable value rather than falling back to a default:

  • caching_max_size โ€” a byte budget, such as 512MiB.
  • caching_max_items โ€” a row budget.

caching_ttl is also accepted as caching_item_ttl, which is the spelling the SQL results, search results, and embeddings caches use. Eviction is entry-granular. A cached response can span several rows, so the runtime ranks entries by their oldest page and removes all of an entry's rows together. The storage schema is unchanged, and no existing acceleration needs a rebuild.

datasets:
- from: https://api.example.com/v1/items
name: items
acceleration:
enabled: true
engine: duckdb
refresh_mode: caching
primary_key: '(request_query, request_path)'
params:
caching_ttl: 5m
caching_max_size: 512MiB
caching_max_items: 50000

caching_stale_if_error now fires on the failure it exists for. It keyed off a fetch that returned an error. The HTTP connector reports a failing origin as a successful fetch whose rows carry a 429 or 5xx status once it exhausts max_retries. An operator who enabled the setting received the origin's error body instead of the cached response.

A caching accelerator that has nothing bounding it now says so at startup.

SQL Results Cache Improvementsโ€‹

Stale results remain available across a refresh when configured. An acceleration refresh evicted every dependent SQL results cache entry, and any successful refresh counted as a change. A refresh_mode: full update still flushed the whole per-table cache. For a workload with consistently high QPS, each refresh turned a population of cached results into simultaneous synchronous misses.

When stale_while_revalidate_ttl is configured, an invalidation now marks dependent entries stale as of the refresh instead of evicting them. Inside the stale window the runtime serves the previous result with Results-Cache-Status: STALE and starts one background revalidation per key. Past the stale window the request is a miss, exactly as before. With no stale window configured, invalidation stays hard.

Cache accounting covers more retained memory. A cache with a million empty results reported 0.09 GiB against 1.85 GiB of retained memory. Its max_size accounting omitted parts of each entry. The cache now shares schemas and input-table sets, copies foreign buffers, and accounts for per-buffer allocation overhead.

In the reported benchmark, each entry retained 1%โ€“80% less memory across 20 combinations of result shape and source. The benchmark ran on macOS/arm64 with snmalloc. Reported size ranged from 0.65x to 1.73x of retained memory after the change. The lowest ratio before the change was 0.31x. These figures compare the fix with its merge base, not v2.2.1, and are not guarantees for every workload.

This release also corrects memory accounting for the search results and embeddings caches.

Pingora cache engine: Table invalidation read every entry with a destructive get, so an invalidation promoted every key in the cache. Scan order replaced recency, and each visited key became a momentary miss to concurrent readers. A read also served a hit destructively, so a second reader reported a miss for a key the cache holds. Both reads are now non-destructive.

GitHub Data Connector: Review, Release, and Repository Tablesโ€‹

The GitHub Data Connector adds eight tables, 17 columns on pulls, and repository identity on every row. An application can now answer a code-review question in SQL. Before this release, pulls.reviews_count was a bare integer with no state and no reviewer, and pulls.review_comments recorded only inline comments.

PathRows
github.com/{owner}/{repo}/reviewsOne per pull request review, with state, author, submitted_at, and commit_sha
github.com/{owner}/{repo}/review_threadsOne per resolvable thread, with is_resolved, is_outdated, path, and resolved_by
github.com/{owner}/{repo}/releasesOne per release, with total_download_count and assets_count
github.com/{owner}/{repo}/release_assetsOne per asset, with download_count, size, and content_type
github.com/{owner}/{repo}/milestonesOne per milestone, with due_on and progress_percentage
github.com/{owner}/{repo}/repoOne row of repository metadata
github.com/{owner}/reposEvery repository an owner has
github.com/{login}/userThe public profile of one login

pulls adds is_draft, mergeable, merge_state_status, review_decision, status_check_rollup, merge_queue_state, merge_queue_position, merged_by, closed_by, base_ref, head_ref, head_sha, milestone_id, milestone_title, closing_issues_references, closing_issues_count, and reactions_count. issues adds state_reason, closed_by, reactions_count, type, and type_color. Every table returns a repo and an owner column, so a multi-repository UNION ALL keeps its rows apart.

The connector also asks GitHub for a narrower pull request page. A 100-node page now exceeds GitHub's per-request compute budget on a large repository. GitHub rejects that page with Resource limits for this query exceeded and returns every node as null, so the dataset never loaded.

Google Models Move to Vertex AIโ€‹

A from: google chat or embedding model now authenticates as a GCP service account against Vertex AI. Spice no longer accepts a Google AI Studio API key. See Breaking Changes for the migration.

models:
- from: google:gemini-2.5-pro
name: gemini
params:
google_project: my-project
google_location: us-central1
google_service_account_path: /etc/spice/gcp-sa.json

Other AI model fixes in this release:

  • An Anthropic model configured without an explicit model id now resolves. The default named claude-3-5-sonnet-latest, which Anthropic has retired, so every such request failed.
  • An Anthropic model now refuses an OpenAI top_logprobs request instead of translating it to top_k. The two fields are unrelated. top_logprobs reports log probabilities and top_k narrows sampling, so the translation silently changed the model's output.
  • Anthropic streaming failures and provider refusals from openai, xai, and spiceai are now classified from the provider's typed error fields. Each path searched the rendered error text for 401, 429, or rate, and then replaced the provider's own detail with a fixed string.
  • A from: huggingface: chat model reads hf_token again. The parameter moved to the prefix huggingface, so hf_token was warned about as unknown and a gated repository was downloaded anonymously.
  • An Amazon Bedrock model now names the credential AWS rejected instead of reporting unhandled error.
  • The text-embeddings-inference model-load path runs its filesystem and tokenizer work on a blocking thread. That work ran on a Tokio worker thread during model registration, so it could starve /health.

Search Improvements and Bug Fixesโ€‹

This release fixes bugs in search result limits, index updates, and deletes.

  • A result set larger than the requested limit: vector_search(tbl, 'query', 10) against an Elasticsearch-backed index now respects the requested limit.
  • Index writes and deletes kept in sync with the table: Previously, writes for rows with repeated primary keys, no chunks, or non-embeddable chunks could leave the previous index entry. These outdated index entries are now deleted.
  • A chunked Elasticsearch delete: The delete filtered on the key columns, and a string key was left to Elasticsearch dynamic mapping as an analyzed text field. The delete now filters on a field that can match the key exactly.
  • A partial Elasticsearch delete: _delete_by_query returns 2xx when the request ran, and it reports per-document failures and version conflicts in the body. Spice discarded that body, so a delete could leave documents behind and still report success.
  • Full-text index encoding: A full-text upsert is a delete followed by an insert, so both halves must encode the primary key the same way. They disagreed for Float32, Float16, and Binary keys, and both the old and the new row stayed in the index.

Acceleration and Refreshโ€‹

  • LIMIT on a partitioned scan: PartitionTableProvider::scan passed the scan limit as the skip argument rather than the fetch argument. LIMIT 10 over a three-row partitioned dataset returned zero rows.
  • acceleration.enabled: false: A dataset or a view can set enabled: false and leave the rest of the block in place. The runtime read every other setting, accepted it, and then ignored it. The component reported healthy and served federated queries. The runtime now names the settings it discards.
  • ready_state on a view: A view's acceleration.ready_state was accepted by the schema and by the parser, and then never applied. The identical key on a dataset was honoured. A view now resolves the key the way a dataset does.
  • A retention policy that cannot start: A dataset that set retention_check_enabled: true, a retention_period, and a time_column but no retention_check_interval got no retention task and no diagnostic. The builder now reports the refusal.
  • A refresh completion that arrives early: A completion published before a caller registered its wait was dropped with no record, and the caller waited for a refresh that had already happened. The signal is now level-triggered.
  • A refresh completion from the wrong refresh: A waiter was satisfied by the next completion recorded on the table, whichever refresh produced it. A refresh that was already running could therefore release a caller. Completions are now correlated with the refresh that a caller triggered.
  • A table replaced during a refresh: Two callers acted on a completion for a table that had since been removed or rebuilt. The runtime now re-resolves the table after the refresh lands and before it acts on the completion.
  • A schema repair on a checkpoint: Writing a checkpoint's schema also wrote its refresh timestamp, so a schema repair told the scheduler the data was fresh. An overdue dataset then waited a full refresh_check_interval. A schema repair now leaves the freshness clock alone.
  • A recorded snapshot schema: A snapshot's recorded schema is a foreign declaration, and a Map that declares its entries field nullable is a declaration no accelerator can hold. The restore path now conforms that declaration to the Arrow map layout.
  • A source row durable write-back could not confirm: The delivery worker read a missing point-scan row as a deletion. A short visibility gap in the accelerator therefore deleted that row from the source of record. The worker now withholds a key it cannot read and retries it on a later pass. A delivery cursor advances only after the pass succeeds. Write-back also refuses a configuration it cannot uphold, which is a behavior change. See Breaking Changes.

Arrow and Storage Bug Fixesโ€‹

  • Decimal128 on write paths: Three conversions could produce a plausible wrong number instead of an error. The sum of two in-range halves wrapped to a large negative decimal at scale 38. A float-to-int cast saturated to i128::MAX, and NaN became 0. The declared precision was never checked, so a value needing more digits than the column declares was appended anyway. All three were reachable from Debezium decimal ingestion, where the input is source-controlled. The runtime now validates the destination precision once, where every input form converges.

  • An Iceberg DELETE an equality key cannot express: An Iceberg delete writes an equality delete file, which removes rows whose key columns equal the given values. That statement matches the user's WHERE only when the condition reads key columns alone. A condition on a float or a nested column removed rows that did not match. Spice now refuses the statement and names the offending column.

  • A Parquet object overwritten mid-scan: A listing-table Parquet scan decoded two object generations as one file. The scan now pins one generation through a version id or an If-Match header. dataset_acceleration_refresh_errors carries reason=object_generation_changed|parquet_decode|other, so an expected overwrite is distinguishable from corruption.

  • An Arrow relabel: relabel_array_data carries an array's values across a type change untouched, and only field names and nested nullability flags may differ. Nothing enforced that contract. It now refuses three kinds of target:

    • a target that changes what the buffers mean
    • a target that declares away nulls the array still holds
    • a target whose same-typed sibling fields are reordered

    The third kind produced silent column-value transposition on the Delta Lake column-mapping path.

  • Arrow MAP columns: The Arrow map layout forbids a nullable entries field, and MapArray::try_new refuses one. Nothing enforced it at decode. A producer that declared it that way handed over a column that decoded cleanly and then failed in the first kernel that rebuilt it. The runtime now normalizes entries nullability at every Arrow decode point. A Databricks SQL Warehouse MAP column, which declares entries nullable, no longer panics the runtime. A Cayenne accelerator that has already persisted the non-conforming declaration is now repaired.

  • A nested nullability difference: try_cast_to decided its fast paths with Schema::contains, which permits a nested field's nullability to differ. RecordBatch requires the two types to be identical. The shared entry point now aligns the difference instead of publishing it.

  • A retired Vortex file: Retiring a Cayenne file released its Vortex segments and left its footer in DataFusion's file-metadata cache, which has no TTL. A file opened during retirement could also repopulate the path that retirement had just cleared. The retirement drain had no ceiling. A stalled put therefore held every caller of the invalidation, and the delete sink is one of them. All three faults are fixed.

  • A Cayenne teardown that deleted a shared metastore: Recreation of a Cayenne dataset could delete a shared metastore and leave other datasets unavailable after restart. The guard checked only the metastore named by that dataset's settings, not catalogs inside its data directory. Open file handles hid the loss until restart. The runtime now scans that directory before catalog changes and again before deletion. It refuses recreation if it finds a Cayenne metastore or cannot safely resolve the configured paths.

Observability and Operationsโ€‹

  • OpenTelemetry resource attributes: The OTLP ingest path parsed resource attributes such as service.name and service.instance.id and then dropped them. Data points from two processes were therefore indistinguishable once written. Resource attributes now reach the metric data points. The same change closes four ingest races that dropped data with no error. One of those races let a write publish through a table provider that a schema evolution had already replaced.
  • A panicking query: A query whose execution panicked was sometimes returned as an empty HTTP 200 success, which no client can tell apart from "no rows matched". This happened in 20 of 60 identical runs on trunk. A panicking query is now always an error.
  • runtime.cpu.cores above the container's ceiling: runtime.cpu.cores was the one CPU entitlement setting taken raw rather than clamped. A pod configured with runtime.cpu.cores: 6 under resources.limits.cpu: 2 sized every derived pool for six cores and was then throttled. The runtime now warns and names both readings. It does not clamp, because an operator may size the runtime for a node the pod has not reached yet.
  • HTTP latency: The HTTP server sets TCP_NODELAY, which lowers the latency of a small response body. The Flight SQL server already set it.
  • MCP tools in runtime.task_history: A proxied MCP tool call was recorded under two different task values depending on the entry point, so one logical tool split across two rows. Grouping by task gave wrong per-tool counts. Both entry points now use the encoded name.
  • A discarded Flight batch: The runtime reported a data_loss count that counted a message by its body length. A batch whose body is empty still carries rows, so the count was wrong. The runtime now reads the IPC header.
  • A hot reload that changes functions or catalogs: A cached logical plan embeds the ScalarUDF and the TableSource it was planned against. A hot reload that redefined a SQL function or replaced a catalog left those plans in place. The same SQL then kept answering from the replaced component. The plan cache is installed unconditionally with a one-hour TTL, so no caching configuration was needed to hit this. Both handlers now discard the affected plans.
  • Cloud Connect metrics cadence: A Cloud Connect instance exports metrics every 10 seconds rather than every 30, so a chart drawn from the control stream resolves at 10 seconds. The payload is a snapshot of cumulative totals, so this changes chart resolution and not what is recorded.

Other Improvements and Bug Fixesโ€‹

  • Glue catalog: A discovered Glue table that Spice cannot read, such as an ORC or Avro table, was absent from the catalog with nothing said about it. The catalog connector now reports each such table and the reason.
  • Databricks: The connector accepts Unity Catalog streaming tables and views. It also forwards the runtime's spark feature.
  • DuckDB index materialization: The DuckDB intermediate index materialization rule reads a table's index list before it rewrites a scan into a materialized CTE. That list had been empty, so the rule never fired and an indexed column never narrowed a scan.
  • Vortex list_length pushdown: DataFusion array_length(expr) and array_length(expr, 1) now convert to Vortex list_length and push into the scan. A list length is computed from offsets, and element values are not materialized.
  • ScyllaDB: The ScyllaDB Data Connector is out of the default build, alongside ODBC. make install-scylladb or --features scylladb builds it. A Spicepod that names scylladb: on a build without it now says the build lacks the connector rather than offering the closest registered name. The connector also declines a physical sort that CQL cannot serve.
  • Turso: The accelerator refuses a stored list whose encoding predates the version marker rather than reading it under the current encoding.
  • CLI: spice run resolves spiced beside the CLI before it reaches for the managed install, and never from PATH. A Spice Cloud project listing is attributed to the organization it was requested for. spice query and spice nsql analyze keep the API key on its origin across a redirect. A Cloud Connect managed instance no longer warns about the default pods watcher on every spice run.
  • MCP tools: A renamed tool forwards strict() and as_mcp_proxy() to the tool it wraps. runtime-tools declares what its mcp feature needs.
  • Connector registries on shutdown: The runtime no longer clears stateless connector registries on shutdown.

Dependency Updatesโ€‹

Compared with v2.2.1, this release changes the following versions:

Dependency / Componentv2.2.1v2.3.0
iceberg-rustv0.10.0v0.10.1
Rust toolchainv1.96.1v1.97.1

DataFusion remains at v54.1.0 and Arrow remains at v58.3.0. Spice updates their fork revisions for the federation, Parquet scan, and cache fixes described above.

Contributorsโ€‹

Breaking Changesโ€‹

Google models authenticate against Vertex AI. A from: google chat or embedding model no longer accepts google_api_key. Every such model now authenticates as a GCP service account.

Update each from: google model and embedding to set google_project, google_location, and exactly one credential setting.

Before:

models:
- from: google:gemini-2.5-pro
name: gemini
params:
google_api_key: ${secrets:google_api_key}

After:

models:
- from: google:gemini-2.5-pro
name: gemini
params:
google_project: my-project
google_location: us-central1
google_service_account_path: /etc/spice/gcp-sa.json
SettingDescription
google_projectThe GCP project id. Required.
google_locationThe GCP region, such as us-central1, or global. Required.
google_service_account_pathThe path to a GCP service account JSON key file.
google_service_account_keyA GCP service account JSON key as a string.
google_application_default_credentialsRead the key path from the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Set exactly one of the three credential settings.

Two behavior changes to note before you upgrade:

  • Durable write-back rejects unsafe settings and operations. Before you upgrade, set mode: file, remove retention settings, and declare a single-column primary_key for each durable write-back dataset. The runtime rejects unsupported settings at load time. Submit writes inside a transaction as one BEGIN; ...; COMMIT; request. Write-back datasets reject DELETE and TRUNCATE when you issue those statements.
  • The ScyllaDB Data Connector is out of the default build. Build with --features scylladb, or run make install-scylladb, to keep it. ODBC already worked this way.

Cookbook Updatesโ€‹

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgradingโ€‹

To upgrade to v2.3.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.3.0 image:

docker pull spiceai/spiceai:2.3.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changedโ€‹

Changelogโ€‹

  • fix(search): surface an Elasticsearch delete that only partially applied (fixes #12364) by @claudespice in #12720
  • fix(ci): bound every integration job, so a wedged one cannot hold the queue (refs #12718) by @grokspice in #12721
  • feat(tools): add pdf-parse tool to compare liteparse and pdf-inspector by @Jeadie in #12807
  • fix(turso): refuse a stored list whose encoding predates the version marker (fixes #12632) by @grokspice in #12837
  • Use TypedParams for reranker parameters by @Jeadie in #13046
  • fix(search): correctness, async-safety, and performance fixes across the search subsystem by @Jeadie in #13065
  • Decide full-text CDC-attachment at construction, not after by @Jeadie in #13075
  • fix(scylladb): decline the physical sort pushdown CQL cannot serve (fixes #10775) by @claudespice in #13107
  • fix(cache): stop Pingora table invalidation from promoting every key it reads (fixes #12674) by @claudespice in #13117
  • fix(cayenne): stop a failed statistics read publishing a partial count as exact (fixes #13010) by @claudespice in #13125
  • fix(cayenne): refuse a widened CDC batch a partitioned acceleration cannot apply (fixes #13051) by @claudespice in #13133
  • fix(catalogs): report the Glue tables Spice cannot read instead of dropping them silently (fixes #13102) by @claudespice in #13149
  • fix(ci): gate every tracked Rust source tree in the merge queue's change filter (fixes #13120) by @claudespice in #13151
  • fix(cache): bound max_size on the memory an entry holds, not its array bytes (fixes #12931) by @claudespice in #13154
  • fix(cayenne): honour the sort-merge row floor on the memory-gated path (fixes #12958) by @claudespice in #13157
  • fix(cache): serve a Pingora hit without hiding it from a concurrent reader (fixes #12987) by @claudespice in #13158
  • fix(vortex): stop a file opened mid-retirement from repopulating the cleared path (fixes #12963) by @claudespice in #13161
  • fix(vortex): evict a retired file's footer with its segments (fixes #12953) by @claudespice in #13162
  • feat(google): switch from Google AI Studio to Vertex AI by @krinart in #13210
  • Run TEI candle model-load filesystem and tokenizer work on a blocking thread by @Jeadie in #13223
  • fix(correctness): validate Decimal128 precision and overflow on write paths by @lukekim in #13224
  • fix(runtime): stop clearing stateless connector registries on shutdown by @Jeadie in #13225
  • fix(search): use one encoding path for full-text index writes and deletes by @Jeadie in #13226
  • fix(docker): install ARM64 linker toolchain by @Jeadie in #13266
  • ci: build the runtime integration archives with one feature set by @bjchambers in #13270
  • ci: archive every integration test target in one invocation by @bjchambers in #13273
  • fix(sql): bump the datafusion pin so a bounded EXISTS refuses instead of returning wrong rows (fixes #13277) by @claudespice in #13280
  • ci: have the compiler-cache action own its credentials by @bjchambers in #13282
  • fix(cpu-budget): warn when a configured cores value exceeds the container's real CPU ceiling (fixes #13275) by @claudespice in #13284
  • fix(databricks): publish a MAP column with the non-nullable entries field Arrow requires (fixes #7307) by @claudespice in #13288
  • fix(search): bound a search result set by the requested limit, not just the index read (fixes #13274) by @claudespice in #13290
  • fix(vortex): bound the segment-cache retirement drain so a stuck put cannot hold a delete open (fixes #12964) by @claudespice in #13300
  • Reach the accelerator contract without going through the runtime by @bjchambers in #13304
  • fix(postgres): rebuild the datasets already streaming when a join replaces their replication slot (fixes #13229) by @claudespice in #13313
  • fix(cli): resolve spiced beside the CLI before the managed install, and never from PATH by @claudespice in #13317
  • fix(iceberg): refuse a delete an equality key cannot express by @lukekim in #13322
  • Stop the accelerator engines reaching up into the runtime by @bjchambers in #13324
  • fix(benchmarks): open snapshot-update PRs against the dispatch branch, not always trunk by @krinart in #13335
  • chore(deps): bump datafusion-table-providers for NUMERIC scale fidelity by @lukekim in #13349
  • feat(cloud-connect): export metrics every 10s by @phillipleblanc in #13353
  • Move the accelerator engines into their own crates by @bjchambers in #13354
  • perf(cayenne): shard the encode by what the write is, not by what the table declares by @lukekim in #13356
  • fix(cli): attribute a project listing to the org it was requested for by @lukekim in #13357
  • Enable Oracle TPC-H result validation, fix the loader that trimmed leading spaces by @sgrebnov in #13360
  • Readme: Change PG catalog status from Alpha to Beta by @sgrebnov in #13361
  • Upgrade iceberg-rust to v0.10.1 by @krinart in #13365
  • perf(cayenne): cut a rewrite's shards at equal row mass, not equal width by @lukekim in #13373
  • fix(deps): bump datafusion-table-providers for AVG/division NUMERIC scale rounding by @krinart in #13387
  • fix(databricks): forward runtime's spark feature to the databricks connector by @sgrebnov in #13389
  • Move the Cayenne accelerator into its own crate by @bjchambers in #13391
  • fix(testoperator): compare exact decimals on their mantissas, not through f64 by @bjchambers in #13407
  • Configure an accelerator engine through its constructor instead of a published global by @bjchambers in #13409
  • fix(opentelemetry): merge resource attributes into metric data points and close OTLP-to-sink ingest races by @peasee in #13412
  • fix(arrow): refuse a relabel that changes what the array's buffers mean (fixes #13423) by @claudespice in #13435
  • fix(tools): record one task_history task per proxied MCP tool (fixes #13338) by @claudespice in #13437
  • fix(release): read release notes from a file and trim them to GitHub's body limit by @sgrebnov in #13457
  • fix(ci): probe the cc toolchain before reaching for brew in setup-cc by @claudespice in #13466
  • fix(cayenne): refuse a teardown that would delete a metastore no parameter names (fixes #13436) by @claudespice in #13471
  • fix(spiced): stop warning about the default pods watcher by @phillipleblanc in #13494
  • fix(arrow): align a nested nullability difference instead of advertising it (fixes #13285) by @claudespice in #13496
  • fix(acceleration): make the refresh-completion signal level-triggered (fixes #13086) by @claudespice in #13505
  • fix(acceleration): apply a partitioned scan's LIMIT as a fetch, not a skip by @vatsalp2008 in #13507
  • fix(ci): stage the retention OOM test binary without its debug info by @claudespice in #13511
  • fix(ci): let sign-off run from a worktree nested inside the checkout, and add --skip-targeted by @bjchambers in #13520
  • Authenticate the stargazers dataset with a PAT and drop qa_analytics by @lukekim in #13529
  • fix(ci): isolate sccache per job on shared self-hosted Macs by @lukekim in #13531
  • chore(scylladb): take the ScyllaDB connector out of the default build by @lukekim in #13532
  • feat(github): add review, release, milestone, user and repo tables, plus repo/owner columns by @lukekim in #13545
  • fix(ci): match the allowed refresh-task warning at its current module path by @claudespice in #13547
  • fix(flight): normalize MAP entries nullability at every Arrow decode point (fixes #13495) by @claudespice in #13550
  • fix(anthropic): default to a model Anthropic still serves (fixes #13557) by @claudespice in #13563
  • fix(postgres): rebuild an emptied CDC acceleration rather than resume its surviving position (refs #13546) by @claudespice in #13566
  • chore(spicepod): accelerate GitHub datasets with Cayenne instead of DuckDB by @lukekim in #13571
  • fix(arrow): refuse a relabel that declares away nulls the array still holds (fixes #13433) by @claudespice in #13585
  • fix(cayenne): refuse a catalog whose data directory would hold its metastore (fixes #13105) by @claudespice in #13593
  • fix(ci): exempt every App account from the assignee gate, not just Dependabot (fixes #13115) by @grokspice in #13594
  • fix(mysql): assign the binlog dump session's net_write_timeout floor as an integer literal (fixes #13307) by @grokspice in #13595
  • fix(ci): let the E2E macOS build share the fleet's Cargo home (fixes #13299) by @grokspice in #13596
  • fix(ci): call a test binary the runner cannot load an infrastructure failure (fixes #13518) by @grokspice in #13597
  • fix(spicepod): say which acceleration settings enabled: false discards (fixes #13514) by @grokspice in #13602
  • feat(caching): bound a caching accelerator by size, count and entry lifetime (closes #13525) by @bjchambers in #13604
  • fix(ci): catch a stale Cargo.lock before the merge queue, not after a 55-minute build (fixes #13598) by @grokspice in #13606
  • fix(bedrock): say which credential AWS rejected instead of "unhandled error" (refs #12396) by @claudespice in #13616
  • feat(caching): serve results stale after an acceleration refresh instead of evicting them by @krinart in #13618
  • fix(write-back): never delete a source row for a key the accelerator did not return by @phillipleblanc in #13638
  • fix(cayenne): fold a staged append's unpublished keys into the PK-keyset rebuild (fixes #13639) by @claudespice in #13644
  • fix(tools): forward strict() and as_mcp_proxy() from a renamed tool (fixes #13443) by @claudespice in #13649
  • fix(delta_lake): order a column-mapping relabel target the way the scan reads it (fixes #13434) by @claudespice in #13655
  • fix(ci): raise the e2e Linux build bound above the worst legitimate run (fixes #13674) by @grokspice in #13675
  • fix: comment out refresh_append_overlap in the sample spicepod by @lukekim in #13680
  • fix(anthropic): refuse a log-probability request instead of narrowing sampling (fixes #13581) by @claudespice in #13682
  • fix(cayenne): record a pipelined non-conflict staged append's primary keys (fixes #13642) by @claudespice in #13686
  • fix(schema): stop an illegal Arrow Map entries declaration from being stored or compared (fixes #13549) by @claudespice in #13695
  • build(rust): upgrade toolchain to 1.97.1 by @lukekim in #13696
  • fix(ci): stop a CI git push from blocking forever on a credential prompt (fixes #13701) by @claudespice in #13702
  • fix(search): remove the vector a rejected write left behind (fixes #13504) by @claudespice in #13705
  • fix(acceleration): stop a refresh already running from answering a later waiter (refs #13544) by @claudespice in #13709
  • fix(search): remove a chunked row's stale chunks when its text goes away (refs #13704) by @claudespice in #13716
  • fix(catalogs): install the Spice function deny-list on the SQL catalog connectors (refs #13664) by @claudespice in #13731
  • fix(runtime-tools): declare what the mcp feature actually needs (fixes #13648) by @grokspice in #13733
  • fix(runtime): re-resolve a table after its refresh lands, before acting on the completion (fixes #13603) by @claudespice in #13735
  • fix(flight): count a discarded batch by its IPC header, not its body length (fixes #13636) by @grokspice in #13736
  • docs(makefile): say what SPICED_DATA_FEATURES actually is (fixes #13678) by @grokspice in #13738
  • fix(anthropic): classify a streaming failure by Anthropic's error type, not its message text (fixes #13562) by @claudespice in #13748
  • fix(views): apply a view's acceleration.ready_state instead of dropping it (fixes #13615) by @claudespice in #13750
  • fix(ci): resolve a Python 3.11+ interpreter for the lint-rust guards (refs #13754) by @grokspice in #13755
  • fix(bigquery): emit valid pushed-down SQL by @phillipleblanc in #13768
  • fix(bigquery): safely federate regexp_match null checks by @krinart in #13771
  • feat(hash-index): verify the bloom filter's block index with Verus by @lukekim in #13777
  • fix(adbc): federate a BigQuery statement spanning datasets as one query by @phillipleblanc in #13780
  • fix(adbc): cancel an abandoned query, stop the BigQuery job, free the connection by @phillipleblanc in #13782
  • fix(snapshot): conform a recorded snapshot schema to the Arrow map layout (fixes #13694) by @claudespice in #13786
  • fix(cayenne): materialize the in-memory CDC tier before a scanning DELETE by @lukekim in #13798
  • fix(caching): say when a caching accelerator has nothing bounding it (fixes #13525) by @claudespice in #13805
  • fix(bigquery): run three federated statement shapes BigQuery was refusing by @phillipleblanc in #13812
  • fix(duckdb): rewrite DataFusion's btrim to DuckDB's trim (fixes #13794) by @claudespice in #13821
  • fix(federation): stop pushing btrim to SQLite and MySQL, which have no btrim (fixes #13840) by @claudespice in #13823
  • fix(ci): give the throughput workflow the postgres fixtures bench provisions by @krinart in #13830
  • fix(acceleration,cayenne): Resolve quoted columns in keys, name which primary key columns are null by @peasee in #13845
  • fix: pin listing-table Parquet reads to one object generation by @phillipleblanc in #13847
  • fix(duckdb): lower-case the hex digits a federated to_hex gets back (fixes #13818) by @claudespice in #13852
  • fix(bigquery): carry the merged unparser fixes through the dialect wrapper by @phillipleblanc in #13853
  • fix(databricks): allow Unity Catalog streaming tables and views through the table-type check by @krinart in #13855
  • fix(acceleration): report a retention policy that cannot start instead of silently building none (fixes #13804) by @claudespice in #13857
  • fix(search): evict a key whose deciding row the index rejected (fixes #13848) by @claudespice in #13859
  • fix(duckdb): Restore intermediate index materialization optimization by @sgrebnov in #13864
  • fix(federation): refuse a user function the deny-list snapshot was built before (fixes #13726) by @claudespice in #13868
  • fix(duckdb): decode a federated sha256 back to the digest's bytes (fixes #13850) by @claudespice in #13869
  • fix(runtime): Add TCP_NODELAY to HTTP server by @peasee in #13874
  • fix(query): surface a panicking query as an error, never an empty success (fixes #13876) by @claudespice in #13878
  • fix(bigquery): keep the built-in date_trunc, forward two dialect renderings, repin the unparser by @phillipleblanc in #13882
  • fix(llms): classify a provider refusal from its typed fields, not its message (refs #13747) by @claudespice in #13884
  • fix(vortex): keep control-byte field names unescaped in physical schema by @lukekim in #13886
  • perf(vortex): push DataFusion array_length down as Vortex list_length by @lukekim in #13888
  • fix(duckdb): render a federated concat as || so a NULL argument propagates (fixes #13849) by @claudespice in #13889
  • fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
  • fix(duckdb): screen a federated inner_product so a non-finite result is NULL (fixes #13787) by @claudespice in #13895
  • refactor(postgres): report an acceleration re-read as a refresh, not a bespoke metric by @bjchambers in #13896
  • fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
  • fix(duckdb): pin a connector's DuckDB session to UTC so a dataset's schema does not carry the host timezone (fixes #13899) by @claudespice in #13903
  • fix(cayenne): keep a file's statistics the same whichever source serves them (refs #13829) by @claudespice in #13904
  • fix(bigquery): preserve results and federation for temporal and recursive queries by @bjchambers in #13905
  • Make the SQL results cache hold what it says it holds by @bjchambers in #13908
  • fix(duckdb): keep a call the dialect cannot render out of the federated plan (fixes #13900) by @claudespice in #13909
  • fix(functions): discard cached plans when a hot reload changes the function set (refs #13873) by @claudespice in #13911
  • fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
  • fix(cli): keep the API key on its origin in the SDK-built query client (fixes #12502) by @grokspice in #13923
  • fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
  • ci: align default and ODBC build features by @phillipleblanc in #13933
  • fix(github): bound the pull request page to GitHub's per-request compute budget (refs #13762) by @grokspice in #13938
  • fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
  • fix(ci): preserve Cargo discovery markers during runner disk sweeps by @phillipleblanc in #13940
  • fix(bigquery): push down JSON scalar text and null checks by @phillipleblanc in #13944
  • fix(models): read the HuggingFace chat token as hf_token again (fixes #13932) by @claudespice in #13946
  • fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
  • fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
  • ci: upgrade spiceio setup action to v0.9.0 by @lukekim in #13971

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.2.1...v2.3.0

Spice v2.2.1 (Sep 2, 2026)

ยท 15 min read
Ben Chambers
Member of Technical Staff at Spice AI

Spice v2.2.1 is now available! ๐Ÿ› ๏ธ

Spice v2.2.1 is a patch release that improves accelerated data management, query federation, and search. Spice Cayenne now reclaims the storage that upserts and compaction leave behind, and new metrics report what each maintenance pass did. Retention policies apply to more Cayenne datasets. Federated queries push more work down to BigQuery. DuckDB and HTTP datasets keep their memory bounded. Vector search with HuggingFace embedding models returns more relevant results.

What's New in v2.2.1โ€‹

Cayenne Reclaims Storage Automaticallyโ€‹

A Spice Cayenne acceleration now reclaims the space that a write or a compaction supersedes. Each maintenance pass removes four kinds of state that earlier releases kept:

  • tombstones for superseded upsert rows
  • snapshot directories that a later commit replaced
  • manifest and catalog rows for snapshots that compaction merged away
  • deletion vectors that a compaction orphaned

Cayenne also releases the metastore pages that a pass frees. On a table with frequent refreshes or CDC upserts, disk and metastore usage now stay bounded. No configuration change is required.

Retention Policies Apply to More Cayenne Datasetsโ€‹

Cayenne now applies retention_sql on datasets with refresh_mode: full and on CDC-accelerated datasets with refresh_mode: changes. Before this release, only the append refresh path ran the retention pass. A full refresh reloaded every source row, and the rows that a policy had removed came back.

A retention_sql predicate that calls now() also works now. Cayenne prepared the predicate once when the table opened, at a point where DataFusion cannot evaluate now(), so the pass failed and removed nothing. Cayenne now resolves now() at the start of each pass. See Data Refresh for the retention settings.

datasets:
- from: postgres:events
name: events
acceleration:
engine: cayenne
retention_sql: DELETE FROM events WHERE created_at < now() - INTERVAL '7 days'

Cayenne also accepts refresh_append_overlap now. It warns when a dataset sets retention_sql together with mode: memory, or sets indexes, because it cannot apply either setting.

Cayenne Maintenance and Storage Metricsโ€‹

Cayenne now reports its maintenance decisions and its storage footprint as metrics on the cayenne meter. The runtime serves them at /metrics when spiced starts with --metrics <addr>. An operator can now tell a pass that found nothing from a pass that declined. The metrics also show where disk and metastore space goes. See Metrics for the metrics endpoint.

  • cayenne_compaction_outcome_total and cayenne_maintenance_outcome_total count each pass by outcome, such as committed, no_op, or declined_<reason>.
  • cayenne_compaction_trigger_total names the threshold that requested a pass.
  • cayenne_maintenance_reclaimed_files_total, cayenne_maintenance_reclaimed_bytes_total, and cayenne_maintenance_reclaimed_rows_total count what each pass reclaimed.
  • cayenne_storage_files, cayenne_storage_bytes, and cayenne_storage_rows report the live footprint by storage tier.
  • cayenne_data_dir_files, cayenne_data_dir_bytes, and cayenne_data_dir_snapshot_dirs report what the data directory holds on disk.
  • cayenne_metastore_db_bytes, cayenne_metastore_wal_bytes, and cayenne_metastore_table_rows report the size of the metastore.

Query Federation and SQL Improvementsโ€‹

  • ADBC federation: The ADBC Data Connector again keeps Spice-only functions such as json_get_str out of the SQL it sends to a source, and it honours query_federation: disabled. The v2.2.0 move of the connector into its own crate dropped both settings. A query with a Spice-only function against BigQuery then failed with an unknown-function error.
  • JSON functions in BigQuery: BigQuery federation now translates json_get_int and json_get_float into BigQuery SQL. A filter or a join on a JSON value runs in BigQuery instead of in Spice.
  • Negative JSON numbers: json_get_int and json_get_float now read a negative JSON number. Every earlier release returned NULL for one.
  • Joins between described datasets: An INNER or FULL join between two datasets that both set a description now plans. DataFusion carries the description as schema metadata, and the join failed with a schema mismatch error. This release updates DataFusion with the fix.

Cayenne Query and Delete Fixesโ€‹

  • Predicate deletes on upsert tables: A DELETE with a non-primary-key predicate on a Cayenne upsert table now removes only the rows that match. The delete matched the predicate against superseded row versions as well as live rows. A key whose old version matched therefore lost its live row too. The delete now applies the same visibility rules as a query.
  • Date filters on Cayenne data files: A pushed-down CAST(date_col AS TIMESTAMP) comparison failed the scan for a Date32 column. For a Date64 column, the same comparison skipped files that held matching rows, so the query returned fewer rows. The scan cast the file's date statistic through the wrong storage type. It now converts the statistic before it prunes on it.
  • Primary keys after an interrupted index rebuild: A rebuild of the primary-key index that failed or was cancelled left the index disabled for the life of the process. A later upsert could then write a second live row for a declared primary key, and every write rebuilt the whole keyset. Cayenne now closes the rebuild window on every exit path.
  • Row counts after a commit: A distributed COUNT(*) that Cayenne answered from its maintained row count could miss the rows of a commit that had just become visible. Cayenne now claims a commit's row-count delta before it publishes the rows.
  • Point lookups on CDC datasets: A primary-key lookup with an ORDER BY on the same key failed to plan on a CDC-accelerated Cayenne dataset after its first transactional commit. An example is WHERE id = 7 ORDER BY id. The runtime rejected the query and returned an error. The scan now forwards its constant columns across the schema cast, and the query plans again.
  • Arrow Map columns in file mode: A Cayenne accelerator with mode: file now stores Arrow Map columns, such as the response_headers column of an HTTP dataset. Every write of such a column failed, and the acceleration stayed empty.

DuckDB and HTTP Datasets Keep Memory Boundedโ€‹

DuckDB: The bundled DuckDB moves from v1.5.5 to v1.4.4. On v1.5.5, a DuckDB-accelerated dataset that refreshed through an ON CONFLICT ... DO UPDATE upsert grew its memory by about 6 MiB per refresh. The row count stayed steady while the memory grew, and the write eventually failed with an out-of-memory error. The DuckDB project tracks the regression as duckdb/duckdb#25162, and it is present in every v1.5.x release. v1.4.4 holds a steady footprint. Queries that run during a file-mode DuckDB swap also now recover when the swap invalidates their statements. The configured database path stays attached across the swap.

HTTP: The HTTP Data Connector now bounds its response cache. The cache kept every fetched response for the life of the process. An API-proxy workload with request-shaped keys therefore grew by about 1 MB per distinct request.

The cache now holds at most response_cache_max_size_bytes per dataset, 64 MiB by default, and evicts the oldest entries first. It expires an entry by the origin's max-age and Age headers, and it refuses to store a response that sends no-store or no-cache. The new response_cache_fallback_ttl setting keeps a response from an origin that sends no Cache-Control header at all. Two new metrics, response_cache_size_bytes and response_cache_items_count, report occupancy per dataset.

datasets:
- from: https://api.example.com/v1/items
name: items
params:
response_cache_max_size_bytes: 16777216
response_cache_fallback_ttl: 5m

More Relevant Vector Search with HuggingFace Embeddingsโ€‹

Vector search with a HuggingFace sentence-transformer embedding model, such as all-MiniLM-L6-v2, now returns relevant results. The runtime kept the fixed padding that some models declare in tokenizer.json, so it padded every input to 128 tokens. The padding then dominated the vector of a short input. The runtime now clears that padding and lets the embedding layer own the attention mask. On the MTEB SciFact benchmark with all-MiniLM-L6-v2, nDCG@10 rises from 0.018 to 0.640 (published: 0.645). Full-text search and model2vec static models were not affected.

A search filter on a timezone-aware timestamp column now matches the exact instant when the runtime pushes it down to a DuckDB-backed search index. The filter compared whole milliseconds, so a filter inside a millisecond returned different rows than requested.

Change Data Capture and Write-Backโ€‹

  • PostgreSQL write-back echo suppression: A CDC-accelerated dataset that writes back to its PostgreSQL source now applies each write once. The logical-replication stream returns every write-back transaction as a change. Spice now records the transaction ID of each delivery and drops the matching changes from the stream. Spice drops a replay after a slot reconnect the same way.
  • Write-back for BIGINT primary keys: Durable write-back now delivers for a table whose primary key is a single BIGINT column. Such a table acknowledged writes but never delivered them to the source.
  • MySQL rebuild after a purged binlog: A MySQL acceleration with mysql_replication_invalid_checkpoint_behavior: restart now rebuilds when the source has purged its binlog position, even when mysql_replication_initial_snapshot: disabled is set. It previously resumed from the current head and kept the rows that the source had deleted in the gap.
  • Append refresh on Cayenne: An append refresh on a Cayenne dataset resumes correctly again. The runtime read the append watermark through a timestamp cast that the Cayenne scan could not evaluate. The read failed, and the dataset stopped appending. The runtime now reads the raw column and normalizes the value in memory.

Other Improvementsโ€‹

  • Startup logs: The log events that the runtime emits while it builds now reach the configured log output. These include the DuckDB memory budget and the query memory-limit defaults. Earlier releases dropped them at every log level.
  • Spice Cloud CLI: The CLI now reconciles its stored credential state and active organization for Spice Cloud after login.
  • Cayenne write concurrency: A Cayenne write that requires a single writer now runs serially even when cayenne_write_concurrency is raised.

Contributorsโ€‹

Breaking Changesโ€‹

No breaking changes.

One behavior change to note before you upgrade:

  • DuckDB moves from v1.5.5 to v1.4.4 to keep the memory of DuckDB-accelerated datasets bounded under upsert refreshes (duckdb/duckdb#25162). SQL syntax and functions that DuckDB added in v1.5 are not available to DuckDB-accelerated datasets.

Cookbook Updatesโ€‹

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgradingโ€‹

To upgrade to v2.2.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.2.1 image:

docker pull spiceai/spiceai:2.2.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.2.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changedโ€‹

Changelogโ€‹

  • Make a pinned single partition actually mean one writer by @lukekim in #13325
  • feat(postgres): outstanding write-back xid registry for CDC echo suppression by @Jeadie in #13350
  • fix(llms): clear tokenizer padding/truncation so TEI owns masking by @Jeadie in #13416
  • fix(http): bound the connector's response cache and honour no-store (fixes #13460) by @bjchambers in #13462
  • fix(search): render a DuckDB search filter against the table schema (fixes #13144) by @claudespice in #13489
  • fix(duckdb): bump datafusion-table-providers for the attachment-race fixes by @sgrebnov in #13491
  • fix(cli): reconcile Spice Cloud credential state by @phillipleblanc in #13500
  • fix(mysql): rebuild a purged-position acceleration whatever the snapshot mode says (fixes #13024) by @claudespice in #13501
  • fix(cayenne): decode write-back markers for a single Int64 primary key by @phillipleblanc in #13502
  • fix(deps): bump DataFusion to pick up the join schema-metadata fix by @krinart in #13526
  • fix(cayenne): give each cayenne_catalog test a table root of its own (fixes #13527) by @claudespice in #13535
  • fix(acceleration): read the append watermark without casting the time column by @sgrebnov in #13539
  • test(postgres): cover CDC echo suppression through a real delivery transaction by @phillipleblanc in #13553
  • test(federation): guard the derived-table output names a pin bump landed unguarded (refs #12751) by @claudespice in #13555
  • fix(runtime): carry a scan's constant columns across the schema cast by @phillipleblanc in #13565
  • fix(cayenne): store Arrow Map columns in a file-mode accelerator (fixes #13524) by @bjchambers in #13567
  • fix(cayenne): stop a predicate DELETE destroying live rows on an upsert table by @lukekim in #13574
  • fix(logging): deliver the runtime build's events to the installed subscriber by @sgrebnov in #13587
  • fix(adbc): restore the federation deny-list and query_federation wiring by @peasee in #13590
  • feat(cayenne): observability for maintenance decisions and footprint growth by @bjchambers in #13619
  • test(forks): guard the Spice patches carried on our upstream forks by @bjchambers in #13623
  • test(cayenne): settle the tier before asserting why a subset pass declined by @lukekim in #13632
  • fix(cayenne): close the PK-index checkout window on every exit path (fixes #13267) by @claudespice in #13641
  • fix(cayenne): reclaim inline tombstones, and return the freed metastore pages by @lukekim in #13651
  • fix(cayenne): delete merged-away manifest rows at protected-snapshot compaction by @Jeadie in #13658
  • fix(cayenne): prune old-snapshot catalog rows at subset small-file compaction by @Jeadie in #13659
  • fix(cayenne): delete emptied protected snapshot manifest rows in retention cleanup by @Jeadie in #13660
  • fix(cayenne): schedule the orphaned deletion-vector sweep from every publication that orphans one by @phillipleblanc in #13663
  • fix(cayenne): reclaim superseded snapshot dirs against live state by @peasee in #13670
  • Translate the JSON extraction functions into BigQuery SQL, and read negative JSON numbers by @phillipleblanc in #13672
  • test(forks): guard the four BigQuery dialect unparser fixes the datafusion pin carries by @lukekim in #13691
  • fix(cayenne): claim a commit's live-row delta before it publishes the rows (fixes #13036) by @claudespice in #13720
  • Downgrade DuckDB to v1.4.4 by @sgrebnov in #13742
  • fix(vortex): convert a date statistic into a timestamp before pruning on it (fixes #13624) by @lukekim in #13745
  • fix(cayenne): resolve now() in retention predicates once per pass by @lukekim in #13769

**Full Changelog: https://github.com/spiceai/spiceai/compare/v2.2.0...v2.2.1