Spice v2.3.0 brings performance improvements, broader query federation, and expanded data connector capabilities. The release improves cache reuse and reduces memory use for cached SQL results. It also extends BigQuery support and adds GitHub review, release, and repository data for SQL analysis. Google models now use Vertex AI, so deployments with credentials for Google AI Studio require migration.

Highlights in v2.3.0 include:

• SQL Federation Improvements — bug fixes for function translation, NULL handling, and timezone declarations
• BigQuery Federation — temporal expressions, recursive CTEs, correlated subqueries, and window aggregates now run as one BigQuery job
• Caching Accelerator Limits — caching_max_size and caching_max_items bound what a refresh_mode: caching accelerator holds
• Stale-While-Revalidate Across a Refresh — configured stale-while-revalidate windows retain cached results across acceleration refreshes
• GitHub Review Data — eight new tables, 17 new pulls columns, and a repo / owner column on every row
• Google Models on Vertex AI — a from: google model authenticates as a GCP service account

What's New in v2.3.0​

SQL Federation Improvements​

This release improves SQL translation and function handling for accelerated and federated datasets. The following bug fixes cover string functions, NULL handling, correlated subqueries, and timezone declarations.

• A trim call failed on DuckDB, SQLite, and MySQL because DataFusion emitted its canonical name, btrim. DuckDB now receives trim, with an explicit space argument for the one-argument form. The native SQLite and MySQL paths evaluate btrim locally.
• DuckDB federation now lower-cases to_hex output to match local evaluation.
• DuckDB federation now decodes sha256 output into a 32-byte digest instead of a hexadecimal string.
• concat uses || on DuckDB to consistently propagate NULL arguments.
• inner_product now returns NULL for an undefined dot product on DuckDB for consistency.
• Operations that DuckDB cannot handle (such as regex with U or R flags) are executed locally.
• User-defined functions registered after startup now execute locally.
• Catalog connectors now use the same list of local Spice-only functions as the data connectors.
• Two EXISTS shapes emitted SQL which evaluated the correlation over the whole relation, so the bound selected nothing. A semi or mark join then reported a match on a row the plan never read. Both shapes now refuse pushdown and run locally.
• DuckDB labels a TIMESTAMPTZ column with the connection's own timezone. The connector built its pool and never pinned that setting. A dataset's schema therefore carried the host timezone, and the same query returned different rows on different machines. A connector session is now pinned to UTC.

BigQuery Federation​

BigQuery federation runs more query shapes as one remote job.

• Temporal expressions, recursive CTEs, and integer division now keep their results and stay in one federated statement. These shapes previously failed remotely, split into several queries, or produced wrong cohort boundaries.
• Three more statement shapes now run. The first is a grouped query that projects a wrapped form of its grouping expression. The second is a query whose federated tables all sit inside a correlated subquery. The third is any aggregate window function.
• A query that reads BigQuery tables from several datasets of one project now runs as one BigQuery query. It also no longer returns rows from the wrong dataset when two datasets hold a table of the same name.
• A dataset that stores JSON in a STRING column now pushes down scalar json_as_text expressions and json_get(...) IS NULL checks.
• The built-in date_trunc is preserved, and the dialect and the federation policy now agree on which aggregate and window calls are eligible.
• regexp_match null checks federate safely.
• A distinct union now renders as UNION DISTINCT. BigQuery rejects a bare UNION, so such a query failed outright. UNION ALL is unchanged.
• A numbering function such as ROW_NUMBER no longer carries a window frame, which BigQuery rejects. An aggregate window function keeps its frame.
• Percentile functions and grouping keys now render in the form BigQuery accepts. A reported 29-statement workload that failed on these shapes now runs in full.
• array_element translates a non-negative integer literal index that fits in Int64 as SAFE_ORDINAL. Other indexes evaluate locally. This preserves DataFusion's end-relative semantics for negative indexes.

Cancellation: A BigQuery query whose client goes away now stops. The BigQuery job ends as cancelled, and the pooled connection returns immediately. Before this release the query ran to completion, and the connection stayed busy for its whole duration. Enough cancellations left an application unable to query at all.

The Caching Accelerator Accepts Explicit Limits​

A refresh_mode: caching accelerator had nothing bounding what it held. Retention was derived from caching_ttl plus caching_stale_while_revalidate_ttl, and only when caching_stale_if_error was disabled. A dataset that set caching_stale_if_error therefore got no policy at all, and nothing was ever evicted. Nothing capped the acceleration by size or by count either.

Two settings now bound the acceleration. Each one refuses an unparseable value rather than falling back to a default:

• caching_max_size — a byte budget, such as 512MiB.
• caching_max_items — a row budget.

caching_ttl is also accepted as caching_item_ttl, which is the spelling the SQL results, search results, and embeddings caches use. Eviction is entry-granular. A cached response can span several rows, so the runtime ranks entries by their oldest page and removes all of an entry's rows together. The storage schema is unchanged, and no existing acceleration needs a rebuild.

datasets:
  - from: https://api.example.com/v1/items
    name: items
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: caching
      primary_key: '(request_query, request_path)'
      params:
        caching_ttl: 5m
        caching_max_size: 512MiB
        caching_max_items: 50000

caching_stale_if_error now fires on the failure it exists for. It keyed off a fetch that returned an error. The HTTP connector reports a failing origin as a successful fetch whose rows carry a 429 or 5xx status once it exhausts max_retries. An operator who enabled the setting received the origin's error body instead of the cached response.

A caching accelerator that has nothing bounding it now says so at startup.

SQL Results Cache Improvements​

Stale results remain available across a refresh when configured. An acceleration refresh evicted every dependent SQL results cache entry, and any successful refresh counted as a change. A refresh_mode: full update still flushed the whole per-table cache. For a workload with consistently high QPS, each refresh turned a population of cached results into simultaneous synchronous misses.

When stale_while_revalidate_ttl is configured, an invalidation now marks dependent entries stale as of the refresh instead of evicting them. Inside the stale window the runtime serves the previous result with Results-Cache-Status: STALE and starts one background revalidation per key. Past the stale window the request is a miss, exactly as before. With no stale window configured, invalidation stays hard.

Cache accounting covers more retained memory. A cache with a million empty results reported 0.09 GiB against 1.85 GiB of retained memory. Its max_size accounting omitted parts of each entry. The cache now shares schemas and input-table sets, copies foreign buffers, and accounts for per-buffer allocation overhead.

In the reported benchmark, each entry retained 1%–80% less memory across 20 combinations of result shape and source. The benchmark ran on macOS/arm64 with snmalloc. Reported size ranged from 0.65x to 1.73x of retained memory after the change. The lowest ratio before the change was 0.31x. These figures compare the fix with its merge base, not v2.2.1, and are not guarantees for every workload.

This release also corrects memory accounting for the search results and embeddings caches.

Pingora cache engine: Table invalidation read every entry with a destructive get, so an invalidation promoted every key in the cache. Scan order replaced recency, and each visited key became a momentary miss to concurrent readers. A read also served a hit destructively, so a second reader reported a miss for a key the cache holds. Both reads are now non-destructive.

GitHub Data Connector: Review, Release, and Repository Tables​

The GitHub Data Connector adds eight tables, 17 columns on pulls, and repository identity on every row. An application can now answer a code-review question in SQL. Before this release, pulls.reviews_count was a bare integer with no state and no reviewer, and pulls.review_comments recorded only inline comments.

pulls adds is_draft, mergeable, merge_state_status, review_decision, status_check_rollup, merge_queue_state, merge_queue_position, merged_by, closed_by, base_ref, head_ref, head_sha, milestone_id, milestone_title, closing_issues_references, closing_issues_count, and reactions_count. issues adds state_reason, closed_by, reactions_count, type, and type_color. Every table returns a repo and an owner column, so a multi-repository UNION ALL keeps its rows apart.

The connector also asks GitHub for a narrower pull request page. A 100-node page now exceeds GitHub's per-request compute budget on a large repository. GitHub rejects that page with Resource limits for this query exceeded and returns every node as null, so the dataset never loaded.

Google Models Move to Vertex AI​

A from: google chat or embedding model now authenticates as a GCP service account against Vertex AI. Spice no longer accepts a Google AI Studio API key. See Breaking Changes for the migration.

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_project: my-project
      google_location: us-central1
      google_service_account_path: /etc/spice/gcp-sa.json

Other AI model fixes in this release:

• An Anthropic model configured without an explicit model id now resolves. The default named claude-3-5-sonnet-latest, which Anthropic has retired, so every such request failed.
• An Anthropic model now refuses an OpenAI top_logprobs request instead of translating it to top_k. The two fields are unrelated. top_logprobs reports log probabilities and top_k narrows sampling, so the translation silently changed the model's output.
• Anthropic streaming failures and provider refusals from openai, xai, and spiceai are now classified from the provider's typed error fields. Each path searched the rendered error text for 401, 429, or rate, and then replaced the provider's own detail with a fixed string.
• A from: huggingface: chat model reads hf_token again. The parameter moved to the prefix huggingface, so hf_token was warned about as unknown and a gated repository was downloaded anonymously.
• An Amazon Bedrock model now names the credential AWS rejected instead of reporting unhandled error.
• The text-embeddings-inference model-load path runs its filesystem and tokenizer work on a blocking thread. That work ran on a Tokio worker thread during model registration, so it could starve /health.

Search Improvements and Bug Fixes​

This release fixes bugs in search result limits, index updates, and deletes.

• A result set larger than the requested limit: vector_search(tbl, 'query', 10) against an Elasticsearch-backed index now respects the requested limit.
• Index writes and deletes kept in sync with the table: Previously, writes for rows with repeated primary keys, no chunks, or non-embeddable chunks could leave the previous index entry. These outdated index entries are now deleted.
• A chunked Elasticsearch delete: The delete filtered on the key columns, and a string key was left to Elasticsearch dynamic mapping as an analyzed text field. The delete now filters on a field that can match the key exactly.
• A partial Elasticsearch delete: _delete_by_query returns 2xx when the request ran, and it reports per-document failures and version conflicts in the body. Spice discarded that body, so a delete could leave documents behind and still report success.
• Full-text index encoding: A full-text upsert is a delete followed by an insert, so both halves must encode the primary key the same way. They disagreed for Float32, Float16, and Binary keys, and both the old and the new row stayed in the index.

Acceleration and Refresh​

• LIMIT on a partitioned scan: PartitionTableProvider::scan passed the scan limit as the skip argument rather than the fetch argument. LIMIT 10 over a three-row partitioned dataset returned zero rows.
• acceleration.enabled: false: A dataset or a view can set enabled: false and leave the rest of the block in place. The runtime read every other setting, accepted it, and then ignored it. The component reported healthy and served federated queries. The runtime now names the settings it discards.
• ready_state on a view: A view's acceleration.ready_state was accepted by the schema and by the parser, and then never applied. The identical key on a dataset was honoured. A view now resolves the key the way a dataset does.
• A retention policy that cannot start: A dataset that set retention_check_enabled: true, a retention_period, and a time_column but no retention_check_interval got no retention task and no diagnostic. The builder now reports the refusal.
• A refresh completion that arrives early: A completion published before a caller registered its wait was dropped with no record, and the caller waited for a refresh that had already happened. The signal is now level-triggered.
• A refresh completion from the wrong refresh: A waiter was satisfied by the next completion recorded on the table, whichever refresh produced it. A refresh that was already running could therefore release a caller. Completions are now correlated with the refresh that a caller triggered.
• A table replaced during a refresh: Two callers acted on a completion for a table that had since been removed or rebuilt. The runtime now re-resolves the table after the refresh lands and before it acts on the completion.
• A schema repair on a checkpoint: Writing a checkpoint's schema also wrote its refresh timestamp, so a schema repair told the scheduler the data was fresh. An overdue dataset then waited a full refresh_check_interval. A schema repair now leaves the freshness clock alone.
• A recorded snapshot schema: A snapshot's recorded schema is a foreign declaration, and a Map that declares its entries field nullable is a declaration no accelerator can hold. The restore path now conforms that declaration to the Arrow map layout.
• A source row durable write-back could not confirm: The delivery worker read a missing point-scan row as a deletion. A short visibility gap in the accelerator therefore deleted that row from the source of record. The worker now withholds a key it cannot read and retries it on a later pass. A delivery cursor advances only after the pass succeeds. Write-back also refuses a configuration it cannot uphold, which is a behavior change. See Breaking Changes.

Arrow and Storage Bug Fixes​

• Decimal128 on write paths: Three conversions could produce a plausible wrong number instead of an error. The sum of two in-range halves wrapped to a large negative decimal at scale 38. A float-to-int cast saturated to i128::MAX, and NaN became 0. The declared precision was never checked, so a value needing more digits than the column declares was appended anyway. All three were reachable from Debezium decimal ingestion, where the input is source-controlled. The runtime now validates the destination precision once, where every input form converges.

• An Iceberg DELETE an equality key cannot express: An Iceberg delete writes an equality delete file, which removes rows whose key columns equal the given values. That statement matches the user's WHERE only when the condition reads key columns alone. A condition on a float or a nested column removed rows that did not match. Spice now refuses the statement and names the offending column.

• A Parquet object overwritten mid-scan: A listing-table Parquet scan decoded two object generations as one file. The scan now pins one generation through a version id or an If-Match header. dataset_acceleration_refresh_errors carries reason=object_generation_changed|parquet_decode|other, so an expected overwrite is distinguishable from corruption.

• An Arrow relabel: relabel_array_data carries an array's values across a type change untouched, and only field names and nested nullability flags may differ. Nothing enforced that contract. It now refuses three kinds of target:

  • a target that changes what the buffers mean
  • a target that declares away nulls the array still holds
  • a target whose same-typed sibling fields are reordered

  The third kind produced silent column-value transposition on the Delta Lake column-mapping path.

• Arrow MAP columns: The Arrow map layout forbids a nullable entries field, and MapArray::try_new refuses one. Nothing enforced it at decode. A producer that declared it that way handed over a column that decoded cleanly and then failed in the first kernel that rebuilt it. The runtime now normalizes entries nullability at every Arrow decode point. A Databricks SQL Warehouse MAP column, which declares entries nullable, no longer panics the runtime. A Cayenne accelerator that has already persisted the non-conforming declaration is now repaired.

• A nested nullability difference: try_cast_to decided its fast paths with Schema::contains, which permits a nested field's nullability to differ. RecordBatch requires the two types to be identical. The shared entry point now aligns the difference instead of publishing it.

• A retired Vortex file: Retiring a Cayenne file released its Vortex segments and left its footer in DataFusion's file-metadata cache, which has no TTL. A file opened during retirement could also repopulate the path that retirement had just cleared. The retirement drain had no ceiling. A stalled put therefore held every caller of the invalidation, and the delete sink is one of them. All three faults are fixed.

• A Cayenne teardown that deleted a shared metastore: Recreation of a Cayenne dataset could delete a shared metastore and leave other datasets unavailable after restart. The guard checked only the metastore named by that dataset's settings, not catalogs inside its data directory. Open file handles hid the loss until restart. The runtime now scans that directory before catalog changes and again before deletion. It refuses recreation if it finds a Cayenne metastore or cannot safely resolve the configured paths.

Observability and Operations​

• OpenTelemetry resource attributes: The OTLP ingest path parsed resource attributes such as service.name and service.instance.id and then dropped them. Data points from two processes were therefore indistinguishable once written. Resource attributes now reach the metric data points. The same change closes four ingest races that dropped data with no error. One of those races let a write publish through a table provider that a schema evolution had already replaced.
• A panicking query: A query whose execution panicked was sometimes returned as an empty HTTP 200 success, which no client can tell apart from "no rows matched". This happened in 20 of 60 identical runs on trunk. A panicking query is now always an error.
• runtime.cpu.cores above the container's ceiling: runtime.cpu.cores was the one CPU entitlement setting taken raw rather than clamped. A pod configured with runtime.cpu.cores: 6 under resources.limits.cpu: 2 sized every derived pool for six cores and was then throttled. The runtime now warns and names both readings. It does not clamp, because an operator may size the runtime for a node the pod has not reached yet.
• HTTP latency: The HTTP server sets TCP_NODELAY, which lowers the latency of a small response body. The Flight SQL server already set it.
• MCP tools in runtime.task_history: A proxied MCP tool call was recorded under two different task values depending on the entry point, so one logical tool split across two rows. Grouping by task gave wrong per-tool counts. Both entry points now use the encoded name.
• A discarded Flight batch: The runtime reported a data_loss count that counted a message by its body length. A batch whose body is empty still carries rows, so the count was wrong. The runtime now reads the IPC header.
• A hot reload that changes functions or catalogs: A cached logical plan embeds the ScalarUDF and the TableSource it was planned against. A hot reload that redefined a SQL function or replaced a catalog left those plans in place. The same SQL then kept answering from the replaced component. The plan cache is installed unconditionally with a one-hour TTL, so no caching configuration was needed to hit this. Both handlers now discard the affected plans.
• Cloud Connect metrics cadence: A Cloud Connect instance exports metrics every 10 seconds rather than every 30, so a chart drawn from the control stream resolves at 10 seconds. The payload is a snapshot of cumulative totals, so this changes chart resolution and not what is recorded.

Other Improvements and Bug Fixes​

• Glue catalog: A discovered Glue table that Spice cannot read, such as an ORC or Avro table, was absent from the catalog with nothing said about it. The catalog connector now reports each such table and the reason.
• Databricks: The connector accepts Unity Catalog streaming tables and views. It also forwards the runtime's spark feature.
• DuckDB index materialization: The DuckDB intermediate index materialization rule reads a table's index list before it rewrites a scan into a materialized CTE. That list had been empty, so the rule never fired and an indexed column never narrowed a scan.
• Vortex list_length pushdown: DataFusion array_length(expr) and array_length(expr, 1) now convert to Vortex list_length and push into the scan. A list length is computed from offsets, and element values are not materialized.
• ScyllaDB: The ScyllaDB Data Connector is out of the default build, alongside ODBC. make install-scylladb or --features scylladb builds it. A Spicepod that names scylladb: on a build without it now says the build lacks the connector rather than offering the closest registered name. The connector also declines a physical sort that CQL cannot serve.
• Turso: The accelerator refuses a stored list whose encoding predates the version marker rather than reading it under the current encoding.
• CLI: spice run resolves spiced beside the CLI before it reaches for the managed install, and never from PATH. A Spice Cloud project listing is attributed to the organization it was requested for. spice query and spice nsql analyze keep the API key on its origin across a redirect. A Cloud Connect managed instance no longer warns about the default pods watcher on every spice run.
• MCP tools: A renamed tool forwards strict() and as_mcp_proxy() to the tool it wraps. runtime-tools declares what its mcp feature needs.
• Connector registries on shutdown: The runtime no longer clears stateless connector registries on shutdown.

Dependency Updates​

Compared with v2.2.1, this release changes the following versions:

DataFusion remains at v54.1.0 and Arrow remains at v58.3.0. Spice updates their fork revisions for the federation, Parquet scan, and cache fixes described above.

Contributors​

• @bjchambers
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @vatsalp2008

Breaking Changes​

Google models authenticate against Vertex AI. A from: google chat or embedding model no longer accepts google_api_key. Every such model now authenticates as a GCP service account.

Update each from: google model and embedding to set google_project, google_location, and exactly one credential setting.

Before:

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_api_key: ${secrets:google_api_key}

After:

models:
  - from: google:gemini-2.5-pro
    name: gemini
    params:
      google_project: my-project
      google_location: us-central1
      google_service_account_path: /etc/spice/gcp-sa.json

Set exactly one of the three credential settings.

Two behavior changes to note before you upgrade:

• Durable write-back rejects unsafe settings and operations. Before you upgrade, set mode: file, remove retention settings, and declare a single-column primary_key for each durable write-back dataset. The runtime rejects unsupported settings at load time. Submit writes inside a transaction as one BEGIN; ...; COMMIT; request. Write-back datasets reject DELETE and TRUNCATE when you issue those statements.
• The ScyllaDB Data Connector is out of the default build. Build with --features scylladb, or run make install-scylladb, to keep it. ODBC already worked this way.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.3.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.3.0 image:

docker pull spiceai/spiceai:2.3.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(search): surface an Elasticsearch delete that only partially applied (fixes #12364) by @claudespice in #12720
• fix(ci): bound every integration job, so a wedged one cannot hold the queue (refs #12718) by @grokspice in #12721
• feat(tools): add pdf-parse tool to compare liteparse and pdf-inspector by @Jeadie in #12807
• fix(turso): refuse a stored list whose encoding predates the version marker (fixes #12632) by @grokspice in #12837
• Use TypedParams for reranker parameters by @Jeadie in #13046
• fix(search): correctness, async-safety, and performance fixes across the search subsystem by @Jeadie in #13065
• Decide full-text CDC-attachment at construction, not after by @Jeadie in #13075
• fix(scylladb): decline the physical sort pushdown CQL cannot serve (fixes #10775) by @claudespice in #13107
• fix(cache): stop Pingora table invalidation from promoting every key it reads (fixes #12674) by @claudespice in #13117
• fix(cayenne): stop a failed statistics read publishing a partial count as exact (fixes #13010) by @claudespice in #13125
• fix(cayenne): refuse a widened CDC batch a partitioned acceleration cannot apply (fixes #13051) by @claudespice in #13133
• fix(catalogs): report the Glue tables Spice cannot read instead of dropping them silently (fixes #13102) by @claudespice in #13149
• fix(ci): gate every tracked Rust source tree in the merge queue's change filter (fixes #13120) by @claudespice in #13151
• fix(cache): bound max_size on the memory an entry holds, not its array bytes (fixes #12931) by @claudespice in #13154
• fix(cayenne): honour the sort-merge row floor on the memory-gated path (fixes #12958) by @claudespice in #13157
• fix(cache): serve a Pingora hit without hiding it from a concurrent reader (fixes #12987) by @claudespice in #13158
• fix(vortex): stop a file opened mid-retirement from repopulating the cleared path (fixes #12963) by @claudespice in #13161
• fix(vortex): evict a retired file's footer with its segments (fixes #12953) by @claudespice in #13162
• feat(google): switch from Google AI Studio to Vertex AI by @krinart in #13210
• Run TEI candle model-load filesystem and tokenizer work on a blocking thread by @Jeadie in #13223
• fix(correctness): validate Decimal128 precision and overflow on write paths by @lukekim in #13224
• fix(runtime): stop clearing stateless connector registries on shutdown by @Jeadie in #13225
• fix(search): use one encoding path for full-text index writes and deletes by @Jeadie in #13226
• fix(docker): install ARM64 linker toolchain by @Jeadie in #13266
• ci: build the runtime integration archives with one feature set by @bjchambers in #13270
• ci: archive every integration test target in one invocation by @bjchambers in #13273
• fix(sql): bump the datafusion pin so a bounded EXISTS refuses instead of returning wrong rows (fixes #13277) by @claudespice in #13280
• ci: have the compiler-cache action own its credentials by @bjchambers in #13282
• fix(cpu-budget): warn when a configured cores value exceeds the container's real CPU ceiling (fixes #13275) by @claudespice in #13284
• fix(databricks): publish a MAP column with the non-nullable entries field Arrow requires (fixes #7307) by @claudespice in #13288
• fix(search): bound a search result set by the requested limit, not just the index read (fixes #13274) by @claudespice in #13290
• fix(vortex): bound the segment-cache retirement drain so a stuck put cannot hold a delete open (fixes #12964) by @claudespice in #13300
• Reach the accelerator contract without going through the runtime by @bjchambers in #13304
• fix(postgres): rebuild the datasets already streaming when a join replaces their replication slot (fixes #13229) by @claudespice in #13313
• fix(cli): resolve spiced beside the CLI before the managed install, and never from PATH by @claudespice in #13317
• fix(iceberg): refuse a delete an equality key cannot express by @lukekim in #13322
• Stop the accelerator engines reaching up into the runtime by @bjchambers in #13324
• fix(benchmarks): open snapshot-update PRs against the dispatch branch, not always trunk by @krinart in #13335
• chore(deps): bump datafusion-table-providers for NUMERIC scale fidelity by @lukekim in #13349
• feat(cloud-connect): export metrics every 10s by @phillipleblanc in #13353
• Move the accelerator engines into their own crates by @bjchambers in #13354
• perf(cayenne): shard the encode by what the write is, not by what the table declares by @lukekim in #13356
• fix(cli): attribute a project listing to the org it was requested for by @lukekim in #13357
• Enable Oracle TPC-H result validation, fix the loader that trimmed leading spaces by @sgrebnov in #13360
• Readme: Change PG catalog status from Alpha to Beta by @sgrebnov in #13361
• Upgrade iceberg-rust to v0.10.1 by @krinart in #13365
• perf(cayenne): cut a rewrite's shards at equal row mass, not equal width by @lukekim in #13373
• fix(deps): bump datafusion-table-providers for AVG/division NUMERIC scale rounding by @krinart in #13387
• fix(databricks): forward runtime's spark feature to the databricks connector by @sgrebnov in #13389
• Move the Cayenne accelerator into its own crate by @bjchambers in #13391
• fix(testoperator): compare exact decimals on their mantissas, not through f64 by @bjchambers in #13407
• Configure an accelerator engine through its constructor instead of a published global by @bjchambers in #13409
• fix(opentelemetry): merge resource attributes into metric data points and close OTLP-to-sink ingest races by @peasee in #13412
• fix(arrow): refuse a relabel that changes what the array's buffers mean (fixes #13423) by @claudespice in #13435
• fix(tools): record one task_history task per proxied MCP tool (fixes #13338) by @claudespice in #13437
• fix(release): read release notes from a file and trim them to GitHub's body limit by @sgrebnov in #13457
• fix(ci): probe the cc toolchain before reaching for brew in setup-cc by @claudespice in #13466
• fix(cayenne): refuse a teardown that would delete a metastore no parameter names (fixes #13436) by @claudespice in #13471
• fix(spiced): stop warning about the default pods watcher by @phillipleblanc in #13494
• fix(arrow): align a nested nullability difference instead of advertising it (fixes #13285) by @claudespice in #13496
• fix(acceleration): make the refresh-completion signal level-triggered (fixes #13086) by @claudespice in #13505
• fix(acceleration): apply a partitioned scan's LIMIT as a fetch, not a skip by @vatsalp2008 in #13507
• fix(ci): stage the retention OOM test binary without its debug info by @claudespice in #13511
• fix(ci): let sign-off run from a worktree nested inside the checkout, and add --skip-targeted by @bjchambers in #13520
• Authenticate the stargazers dataset with a PAT and drop qa_analytics by @lukekim in #13529
• fix(ci): isolate sccache per job on shared self-hosted Macs by @lukekim in #13531
• chore(scylladb): take the ScyllaDB connector out of the default build by @lukekim in #13532
• feat(github): add review, release, milestone, user and repo tables, plus repo/owner columns by @lukekim in #13545
• fix(ci): match the allowed refresh-task warning at its current module path by @claudespice in #13547
• fix(flight): normalize MAP entries nullability at every Arrow decode point (fixes #13495) by @claudespice in #13550
• fix(anthropic): default to a model Anthropic still serves (fixes #13557) by @claudespice in #13563
• fix(postgres): rebuild an emptied CDC acceleration rather than resume its surviving position (refs #13546) by @claudespice in #13566
• chore(spicepod): accelerate GitHub datasets with Cayenne instead of DuckDB by @lukekim in #13571
• fix(arrow): refuse a relabel that declares away nulls the array still holds (fixes #13433) by @claudespice in #13585
• fix(cayenne): refuse a catalog whose data directory would hold its metastore (fixes #13105) by @claudespice in #13593
• fix(ci): exempt every App account from the assignee gate, not just Dependabot (fixes #13115) by @grokspice in #13594
• fix(mysql): assign the binlog dump session's net_write_timeout floor as an integer literal (fixes #13307) by @grokspice in #13595
• fix(ci): let the E2E macOS build share the fleet's Cargo home (fixes #13299) by @grokspice in #13596
• fix(ci): call a test binary the runner cannot load an infrastructure failure (fixes #13518) by @grokspice in #13597
• fix(spicepod): say which acceleration settings enabled: false discards (fixes #13514) by @grokspice in #13602
• feat(caching): bound a caching accelerator by size, count and entry lifetime (closes #13525) by @bjchambers in #13604
• fix(ci): catch a stale Cargo.lock before the merge queue, not after a 55-minute build (fixes #13598) by @grokspice in #13606
• fix(bedrock): say which credential AWS rejected instead of "unhandled error" (refs #12396) by @claudespice in #13616
• feat(caching): serve results stale after an acceleration refresh instead of evicting them by @krinart in #13618
• fix(write-back): never delete a source row for a key the accelerator did not return by @phillipleblanc in #13638
• fix(cayenne): fold a staged append's unpublished keys into the PK-keyset rebuild (fixes #13639) by @claudespice in #13644
• fix(tools): forward strict() and as_mcp_proxy() from a renamed tool (fixes #13443) by @claudespice in #13649
• fix(delta_lake): order a column-mapping relabel target the way the scan reads it (fixes #13434) by @claudespice in #13655
• fix(ci): raise the e2e Linux build bound above the worst legitimate run (fixes #13674) by @grokspice in #13675
• fix: comment out refresh_append_overlap in the sample spicepod by @lukekim in #13680
• fix(anthropic): refuse a log-probability request instead of narrowing sampling (fixes #13581) by @claudespice in #13682
• fix(cayenne): record a pipelined non-conflict staged append's primary keys (fixes #13642) by @claudespice in #13686
• fix(schema): stop an illegal Arrow Map entries declaration from being stored or compared (fixes #13549) by @claudespice in #13695
• build(rust): upgrade toolchain to 1.97.1 by @lukekim in #13696
• fix(ci): stop a CI git push from blocking forever on a credential prompt (fixes #13701) by @claudespice in #13702
• fix(search): remove the vector a rejected write left behind (fixes #13504) by @claudespice in #13705
• fix(acceleration): stop a refresh already running from answering a later waiter (refs #13544) by @claudespice in #13709
• fix(search): remove a chunked row's stale chunks when its text goes away (refs #13704) by @claudespice in #13716
• fix(catalogs): install the Spice function deny-list on the SQL catalog connectors (refs #13664) by @claudespice in #13731
• fix(runtime-tools): declare what the mcp feature actually needs (fixes #13648) by @grokspice in #13733
• fix(runtime): re-resolve a table after its refresh lands, before acting on the completion (fixes #13603) by @claudespice in #13735
• fix(flight): count a discarded batch by its IPC header, not its body length (fixes #13636) by @grokspice in #13736
• docs(makefile): say what SPICED_DATA_FEATURES actually is (fixes #13678) by @grokspice in #13738
• fix(anthropic): classify a streaming failure by Anthropic's error type, not its message text (fixes #13562) by @claudespice in #13748
• fix(views): apply a view's acceleration.ready_state instead of dropping it (fixes #13615) by @claudespice in #13750
• fix(ci): resolve a Python 3.11+ interpreter for the lint-rust guards (refs #13754) by @grokspice in #13755
• fix(bigquery): emit valid pushed-down SQL by @phillipleblanc in #13768
• fix(bigquery): safely federate regexp_match null checks by @krinart in #13771
• feat(hash-index): verify the bloom filter's block index with Verus by @lukekim in #13777
• fix(adbc): federate a BigQuery statement spanning datasets as one query by @phillipleblanc in #13780
• fix(adbc): cancel an abandoned query, stop the BigQuery job, free the connection by @phillipleblanc in #13782
• fix(snapshot): conform a recorded snapshot schema to the Arrow map layout (fixes #13694) by @claudespice in #13786
• fix(cayenne): materialize the in-memory CDC tier before a scanning DELETE by @lukekim in #13798
• fix(caching): say when a caching accelerator has nothing bounding it (fixes #13525) by @claudespice in #13805
• fix(bigquery): run three federated statement shapes BigQuery was refusing by @phillipleblanc in #13812
• fix(duckdb): rewrite DataFusion's btrim to DuckDB's trim (fixes #13794) by @claudespice in #13821
• fix(federation): stop pushing btrim to SQLite and MySQL, which have no btrim (fixes #13840) by @claudespice in #13823
• fix(ci): give the throughput workflow the postgres fixtures bench provisions by @krinart in #13830
• fix(acceleration,cayenne): Resolve quoted columns in keys, name which primary key columns are null by @peasee in #13845
• fix: pin listing-table Parquet reads to one object generation by @phillipleblanc in #13847
• fix(duckdb): lower-case the hex digits a federated to_hex gets back (fixes #13818) by @claudespice in #13852
• fix(bigquery): carry the merged unparser fixes through the dialect wrapper by @phillipleblanc in #13853
• fix(databricks): allow Unity Catalog streaming tables and views through the table-type check by @krinart in #13855
• fix(acceleration): report a retention policy that cannot start instead of silently building none (fixes #13804) by @claudespice in #13857
• fix(search): evict a key whose deciding row the index rejected (fixes #13848) by @claudespice in #13859
• fix(duckdb): Restore intermediate index materialization optimization by @sgrebnov in #13864
• fix(federation): refuse a user function the deny-list snapshot was built before (fixes #13726) by @claudespice in #13868
• fix(duckdb): decode a federated sha256 back to the digest's bytes (fixes #13850) by @claudespice in #13869
• fix(runtime): Add TCP_NODELAY to HTTP server by @peasee in #13874
• fix(query): surface a panicking query as an error, never an empty success (fixes #13876) by @claudespice in #13878
• fix(bigquery): keep the built-in date_trunc, forward two dialect renderings, repin the unparser by @phillipleblanc in #13882
• fix(llms): classify a provider refusal from its typed fields, not its message (refs #13747) by @claudespice in #13884
• fix(vortex): keep control-byte field names unescaped in physical schema by @lukekim in #13886
• perf(vortex): push DataFusion array_length down as Vortex list_length by @lukekim in #13888
• fix(duckdb): render a federated concat as || so a NULL argument propagates (fixes #13849) by @claudespice in #13889
• fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
• fix(duckdb): screen a federated inner_product so a non-finite result is NULL (fixes #13787) by @claudespice in #13895
• refactor(postgres): report an acceleration re-read as a refresh, not a bespoke metric by @bjchambers in #13896
• fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
• fix(duckdb): pin a connector's DuckDB session to UTC so a dataset's schema does not carry the host timezone (fixes #13899) by @claudespice in #13903
• fix(cayenne): keep a file's statistics the same whichever source serves them (refs #13829) by @claudespice in #13904
• fix(bigquery): preserve results and federation for temporal and recursive queries by @bjchambers in #13905
• Make the SQL results cache hold what it says it holds by @bjchambers in #13908
• fix(duckdb): keep a call the dialect cannot render out of the federated plan (fixes #13900) by @claudespice in #13909
• fix(functions): discard cached plans when a hot reload changes the function set (refs #13873) by @claudespice in #13911
• fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
• fix(cli): keep the API key on its origin in the SDK-built query client (fixes #12502) by @grokspice in #13923
• fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
• ci: align default and ODBC build features by @phillipleblanc in #13933
• fix(github): bound the pull request page to GitHub's per-request compute budget (refs #13762) by @grokspice in #13938
• fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
• fix(ci): preserve Cargo discovery markers during runner disk sweeps by @phillipleblanc in #13940
• fix(bigquery): push down JSON scalar text and null checks by @phillipleblanc in #13944
• fix(models): read the HuggingFace chat token as hf_token again (fixes #13932) by @claudespice in #13946
• fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
• fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
• ci: upgrade spiceio setup action to v0.9.0 by @lukekim in #13971

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.2.1...v2.3.0

Spice v2.2.1 is now available! 🛠️

Spice v2.2.1 is a patch release that improves accelerated data management, query federation, and search. Spice Cayenne now reclaims the storage that upserts and compaction leave behind, and new metrics report what each maintenance pass did. Retention policies apply to more Cayenne datasets. Federated queries push more work down to BigQuery. DuckDB and HTTP datasets keep their memory bounded. Vector search with HuggingFace embedding models returns more relevant results.

What's New in v2.2.1​

Cayenne Reclaims Storage Automatically​

A Spice Cayenne acceleration now reclaims the space that a write or a compaction supersedes. Each maintenance pass removes four kinds of state that earlier releases kept:

• tombstones for superseded upsert rows
• snapshot directories that a later commit replaced
• manifest and catalog rows for snapshots that compaction merged away
• deletion vectors that a compaction orphaned

Cayenne also releases the metastore pages that a pass frees. On a table with frequent refreshes or CDC upserts, disk and metastore usage now stay bounded. No configuration change is required.

Retention Policies Apply to More Cayenne Datasets​

Cayenne now applies retention_sql on datasets with refresh_mode: full and on CDC-accelerated datasets with refresh_mode: changes. Before this release, only the append refresh path ran the retention pass. A full refresh reloaded every source row, and the rows that a policy had removed came back.

A retention_sql predicate that calls now() also works now. Cayenne prepared the predicate once when the table opened, at a point where DataFusion cannot evaluate now(), so the pass failed and removed nothing. Cayenne now resolves now() at the start of each pass. See Data Refresh for the retention settings.

datasets:
  - from: postgres:events
    name: events
    acceleration:
      engine: cayenne
      retention_sql: DELETE FROM events WHERE created_at < now() - INTERVAL '7 days'

Cayenne also accepts refresh_append_overlap now. It warns when a dataset sets retention_sql together with mode: memory, or sets indexes, because it cannot apply either setting.

Cayenne Maintenance and Storage Metrics​

Cayenne now reports its maintenance decisions and its storage footprint as metrics on the cayenne meter. The runtime serves them at /metrics when spiced starts with --metrics <addr>. An operator can now tell a pass that found nothing from a pass that declined. The metrics also show where disk and metastore space goes. See Metrics for the metrics endpoint.

• cayenne_compaction_outcome_total and cayenne_maintenance_outcome_total count each pass by outcome, such as committed, no_op, or declined_<reason>.
• cayenne_compaction_trigger_total names the threshold that requested a pass.
• cayenne_maintenance_reclaimed_files_total, cayenne_maintenance_reclaimed_bytes_total, and cayenne_maintenance_reclaimed_rows_total count what each pass reclaimed.
• cayenne_storage_files, cayenne_storage_bytes, and cayenne_storage_rows report the live footprint by storage tier.
• cayenne_data_dir_files, cayenne_data_dir_bytes, and cayenne_data_dir_snapshot_dirs report what the data directory holds on disk.
• cayenne_metastore_db_bytes, cayenne_metastore_wal_bytes, and cayenne_metastore_table_rows report the size of the metastore.

Query Federation and SQL Improvements​

• ADBC federation: The ADBC Data Connector again keeps Spice-only functions such as json_get_str out of the SQL it sends to a source, and it honours query_federation: disabled. The v2.2.0 move of the connector into its own crate dropped both settings. A query with a Spice-only function against BigQuery then failed with an unknown-function error.
• JSON functions in BigQuery: BigQuery federation now translates json_get_int and json_get_float into BigQuery SQL. A filter or a join on a JSON value runs in BigQuery instead of in Spice.
• Negative JSON numbers: json_get_int and json_get_float now read a negative JSON number. Every earlier release returned NULL for one.
• Joins between described datasets: An INNER or FULL join between two datasets that both set a description now plans. DataFusion carries the description as schema metadata, and the join failed with a schema mismatch error. This release updates DataFusion with the fix.

Cayenne Query and Delete Fixes​

• Predicate deletes on upsert tables: A DELETE with a non-primary-key predicate on a Cayenne upsert table now removes only the rows that match. The delete matched the predicate against superseded row versions as well as live rows. A key whose old version matched therefore lost its live row too. The delete now applies the same visibility rules as a query.
• Date filters on Cayenne data files: A pushed-down CAST(date_col AS TIMESTAMP) comparison failed the scan for a Date32 column. For a Date64 column, the same comparison skipped files that held matching rows, so the query returned fewer rows. The scan cast the file's date statistic through the wrong storage type. It now converts the statistic before it prunes on it.
• Primary keys after an interrupted index rebuild: A rebuild of the primary-key index that failed or was cancelled left the index disabled for the life of the process. A later upsert could then write a second live row for a declared primary key, and every write rebuilt the whole keyset. Cayenne now closes the rebuild window on every exit path.
• Row counts after a commit: A distributed COUNT(*) that Cayenne answered from its maintained row count could miss the rows of a commit that had just become visible. Cayenne now claims a commit's row-count delta before it publishes the rows.
• Point lookups on CDC datasets: A primary-key lookup with an ORDER BY on the same key failed to plan on a CDC-accelerated Cayenne dataset after its first transactional commit. An example is WHERE id = 7 ORDER BY id. The runtime rejected the query and returned an error. The scan now forwards its constant columns across the schema cast, and the query plans again.
• Arrow Map columns in file mode: A Cayenne accelerator with mode: file now stores Arrow Map columns, such as the response_headers column of an HTTP dataset. Every write of such a column failed, and the acceleration stayed empty.

DuckDB and HTTP Datasets Keep Memory Bounded​

DuckDB: The bundled DuckDB moves from v1.5.5 to v1.4.4. On v1.5.5, a DuckDB-accelerated dataset that refreshed through an ON CONFLICT ... DO UPDATE upsert grew its memory by about 6 MiB per refresh. The row count stayed steady while the memory grew, and the write eventually failed with an out-of-memory error. The DuckDB project tracks the regression as duckdb/duckdb#25162, and it is present in every v1.5.x release. v1.4.4 holds a steady footprint. Queries that run during a file-mode DuckDB swap also now recover when the swap invalidates their statements. The configured database path stays attached across the swap.

HTTP: The HTTP Data Connector now bounds its response cache. The cache kept every fetched response for the life of the process. An API-proxy workload with request-shaped keys therefore grew by about 1 MB per distinct request.

The cache now holds at most response_cache_max_size_bytes per dataset, 64 MiB by default, and evicts the oldest entries first. It expires an entry by the origin's max-age and Age headers, and it refuses to store a response that sends no-store or no-cache. The new response_cache_fallback_ttl setting keeps a response from an origin that sends no Cache-Control header at all. Two new metrics, response_cache_size_bytes and response_cache_items_count, report occupancy per dataset.

datasets:
  - from: https://api.example.com/v1/items
    name: items
    params:
      response_cache_max_size_bytes: 16777216
      response_cache_fallback_ttl: 5m

More Relevant Vector Search with HuggingFace Embeddings​

Vector search with a HuggingFace sentence-transformer embedding model, such as all-MiniLM-L6-v2, now returns relevant results. The runtime kept the fixed padding that some models declare in tokenizer.json, so it padded every input to 128 tokens. The padding then dominated the vector of a short input. The runtime now clears that padding and lets the embedding layer own the attention mask. On the MTEB SciFact benchmark with all-MiniLM-L6-v2, nDCG@10 rises from 0.018 to 0.640 (published: 0.645). Full-text search and model2vec static models were not affected.

A search filter on a timezone-aware timestamp column now matches the exact instant when the runtime pushes it down to a DuckDB-backed search index. The filter compared whole milliseconds, so a filter inside a millisecond returned different rows than requested.

Change Data Capture and Write-Back​

• PostgreSQL write-back echo suppression: A CDC-accelerated dataset that writes back to its PostgreSQL source now applies each write once. The logical-replication stream returns every write-back transaction as a change. Spice now records the transaction ID of each delivery and drops the matching changes from the stream. Spice drops a replay after a slot reconnect the same way.
• Write-back for BIGINT primary keys: Durable write-back now delivers for a table whose primary key is a single BIGINT column. Such a table acknowledged writes but never delivered them to the source.
• MySQL rebuild after a purged binlog: A MySQL acceleration with mysql_replication_invalid_checkpoint_behavior: restart now rebuilds when the source has purged its binlog position, even when mysql_replication_initial_snapshot: disabled is set. It previously resumed from the current head and kept the rows that the source had deleted in the gap.
• Append refresh on Cayenne: An append refresh on a Cayenne dataset resumes correctly again. The runtime read the append watermark through a timestamp cast that the Cayenne scan could not evaluate. The read failed, and the dataset stopped appending. The runtime now reads the raw column and normalizes the value in memory.

Other Improvements​

• Startup logs: The log events that the runtime emits while it builds now reach the configured log output. These include the DuckDB memory budget and the query memory-limit defaults. Earlier releases dropped them at every log level.
• Spice Cloud CLI: The CLI now reconciles its stored credential state and active organization for Spice Cloud after login.
• Cayenne write concurrency: A Cayenne write that requires a single writer now runs serially even when cayenne_write_concurrency is raised.

Contributors​

• @bjchambers
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

One behavior change to note before you upgrade:

• DuckDB moves from v1.5.5 to v1.4.4 to keep the memory of DuckDB-accelerated datasets bounded under upsert refreshes (duckdb/duckdb#25162). SQL syntax and functions that DuckDB added in v1.5 are not available to DuckDB-accelerated datasets.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.2.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.2.1 image:

docker pull spiceai/spiceai:2.2.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.2.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Make a pinned single partition actually mean one writer by @lukekim in #13325
• feat(postgres): outstanding write-back xid registry for CDC echo suppression by @Jeadie in #13350
• fix(llms): clear tokenizer padding/truncation so TEI owns masking by @Jeadie in #13416
• fix(http): bound the connector's response cache and honour no-store (fixes #13460) by @bjchambers in #13462
• fix(search): render a DuckDB search filter against the table schema (fixes #13144) by @claudespice in #13489
• fix(duckdb): bump datafusion-table-providers for the attachment-race fixes by @sgrebnov in #13491
• fix(cli): reconcile Spice Cloud credential state by @phillipleblanc in #13500
• fix(mysql): rebuild a purged-position acceleration whatever the snapshot mode says (fixes #13024) by @claudespice in #13501
• fix(cayenne): decode write-back markers for a single Int64 primary key by @phillipleblanc in #13502
• fix(deps): bump DataFusion to pick up the join schema-metadata fix by @krinart in #13526
• fix(cayenne): give each cayenne_catalog test a table root of its own (fixes #13527) by @claudespice in #13535
• fix(acceleration): read the append watermark without casting the time column by @sgrebnov in #13539
• test(postgres): cover CDC echo suppression through a real delivery transaction by @phillipleblanc in #13553
• test(federation): guard the derived-table output names a pin bump landed unguarded (refs #12751) by @claudespice in #13555
• fix(runtime): carry a scan's constant columns across the schema cast by @phillipleblanc in #13565
• fix(cayenne): store Arrow Map columns in a file-mode accelerator (fixes #13524) by @bjchambers in #13567
• fix(cayenne): stop a predicate DELETE destroying live rows on an upsert table by @lukekim in #13574
• fix(logging): deliver the runtime build's events to the installed subscriber by @sgrebnov in #13587
• fix(adbc): restore the federation deny-list and query_federation wiring by @peasee in #13590
• feat(cayenne): observability for maintenance decisions and footprint growth by @bjchambers in #13619
• test(forks): guard the Spice patches carried on our upstream forks by @bjchambers in #13623
• test(cayenne): settle the tier before asserting why a subset pass declined by @lukekim in #13632
• fix(cayenne): close the PK-index checkout window on every exit path (fixes #13267) by @claudespice in #13641
• fix(cayenne): reclaim inline tombstones, and return the freed metastore pages by @lukekim in #13651
• fix(cayenne): delete merged-away manifest rows at protected-snapshot compaction by @Jeadie in #13658
• fix(cayenne): prune old-snapshot catalog rows at subset small-file compaction by @Jeadie in #13659
• fix(cayenne): delete emptied protected snapshot manifest rows in retention cleanup by @Jeadie in #13660
• fix(cayenne): schedule the orphaned deletion-vector sweep from every publication that orphans one by @phillipleblanc in #13663
• fix(cayenne): reclaim superseded snapshot dirs against live state by @peasee in #13670
• Translate the JSON extraction functions into BigQuery SQL, and read negative JSON numbers by @phillipleblanc in #13672
• test(forks): guard the four BigQuery dialect unparser fixes the datafusion pin carries by @lukekim in #13691
• fix(cayenne): claim a commit's live-row delta before it publishes the rows (fixes #13036) by @claudespice in #13720
• Downgrade DuckDB to v1.4.4 by @sgrebnov in #13742
• fix(vortex): convert a date statistic into a timestamp before pruning on it (fixes #13624) by @lukekim in #13745
• fix(cayenne): resolve now() in retention predicates once per pass by @lukekim in #13769

**Full Changelog: https://github.com/spiceai/spiceai/compare/v2.2.0...v2.2.1

Spice v2.2.0 is now available! 🚀

Spice v2.2.0 focuses on real-time data, performance, and stability. Cloud Connect links self-hosted runtimes to Spice.ai Cloud for management and observability. MySQL datasets and PostgreSQL catalogs can now stream source changes into accelerated datasets. Debezium sources can send CDC events directly without Kafka. Warm in-memory indexes improve vector and full-text search performance.

Highlights in v2.2.0 include:

• Cloud Connect — link self-hosted runtimes for BYOC (Bring Your Own Cloud) to Spice.ai Cloud for management and observability
• MySQL CDC — MySQL tables now stay in sync in real time using MySQL binlog replication, with no Kafka or Debezium infrastructure and no scheduled refreshes
• PostgreSQL Catalog CDC — accelerate an entire PostgreSQL database in real time with a few lines of configuration and no per-table setup
• Faster Search — vector and full-text search can now serve from an in-memory index by default, with no configuration change

What's New in v2.2.0​

Cloud Connect​

Spice Cloud Connect is a feature of Spice.ai Cloud.

Spice Cloud Connect links self-hosted runtimes for BYOC (Bring Your Own Cloud) to Spice.ai Cloud for management and observability. Run spice cloud link to enroll a standalone runtime.

Try it with the Cloud Connect on a Development Machine recipe.

MySQL Change Data Capture​

The MySQL Data Connector now supports refresh_mode: changes. The runtime loads an initial snapshot of the table. It then applies inserts, updates, and deletes from the MySQL binlog as they commit. The acceleration stays current without scheduled refreshes.

datasets:
  - from: mysql:orders
    name: orders
    params:
      mysql_host: localhost
      mysql_db: mydb
      mysql_user: replicator
      mysql_pass: ${secrets:mysql_pass}
    acceleration:
      engine: cayenne
      refresh_mode: changes

• Failover-safe GTID positions: When the source server has GTID enabled, the runtime tracks the replication position as a GTID set. A GTID position stays valid across a replica promotion, so replication survives a failover to a new primary without a rebuild.

PostgreSQL Catalog CDC​

The PostgreSQL Catalog Connector is now Beta with new Catalog CDC acceleration support at Alpha.

A PostgreSQL catalog can now use refresh_mode: changes. One configuration replicates every table that the include patterns match. Queries then read fresh PostgreSQL data with no per-table setup.

catalogs:
  - from: pg
    name: pg
    include:
      - 'public.*'
    params:
      pg_host: localhost
      pg_db: mydb
      pg_user: postgres
      pg_pass: ${secrets:pg_pass}
    acceleration:
      refresh_mode: changes

Known limitations while catalog CDC acceleration is Alpha: configuration may change; a table dropped and recreated in the source can serve rows captured before it was recreated (#12110); and a durable catalog acceleration can come back empty after a restart (#12729). Feedback is welcome in #11850.

Debezium CDC Without Kafka​

Any Debezium source plugin can now stream change events directly into Spice, with no Kafka bus in between. A dataset with from: cdc:… accepts change events at POST /v1/datasets/{name}/cdc, in JSON or Avro. The existing Kafka path (from: debezium:…) is unchanged.

Faster Search with Warm In-Memory Indexes​

Search adds a warm in-memory tier for vector and full-text indexes. The runtime writes each change to the warm tier and to the durable store together. Queries read the warm tier first and fall back to the durable store. The warm tier covers vector indexes, full-text indexes, .vectors datasets, views, and chunked Elasticsearch vector columns.

More search improvements:

• cosine_distance uses SIMD instructions through the simsimd library.
• Full-text search applies stemming by default.
• Full-text search pushes SQL filters down into the tantivy index.
• A delete now removes the document from full-text and vector indexes. BM25 statistics no longer count superseded documents.
• The runtime loads local rerankers from text-embeddings-inference models.
• The runtime validates vector search parameters before it runs the SQL query.

Operations & Observability​

• Query timeout: The new runtime.query.timeout setting bounds the duration of every query. An expired query fails with HTTP 504 or gRPC DEADLINE_EXCEEDED.

runtime:
  query:
    timeout: 30s

• CPU sizing for Kubernetes pods without a CPU limit: A Spice runtime pod with a CPU request and no CPU limit now sizes itself to twice the request instead of every core on the node. Set SPICE_CPU_CORES=all (or runtime.cpu.cores: all) to burst to all cores, for example when a 0.5-core request runs on a 24-core node.
• Trace IDs in logs: Every log record from a query carries the query's trace ID. Filter the logs by one ID to see everything that query did.
• Improved dashboards: The Grafana and Datadog dashboards add new and improved panels, and support multi-replica and Kubernetes deployments.
• Helm: The Helm chart supports a custom Deployment strategy and StatefulSet updateStrategy.
• Dataset status: A non-accelerated dataset now shows the Error state when its source is unavailable, instead of appearing healthy while queries fail.

Other Notable Improvements and Bug Fixes​

• Drasi (Alpha): The runtime forwards CDC changes and runtime tables to a Drasi source.
• Microsoft SQL Server: Kerberos integrated authentication now works on Unix.
• HTTP connector: OAuth2 client-credentials authentication.
• Oracle DATE: Values lost their time of day. They now map to a timestamp.
• Catalog exclude patterns: The Glue and Cayenne catalogs ignored exclude patterns. Both pattern lists now apply.
• Spice Cayenne schema changes: Partitioned accelerations could report a source schema change as applied while partitions kept the old schema. This mismatch caused lossy casts or append failures. Cayenne now rejects in-place evolution for partitioned accelerations. The configured schema-change policy then handles the change.
• Snowflake NUMBER values: The connector could remove fractional digits during schema discovery. It treated numeric metadata as absent and defaulted the scale to zero. It now preserves the source precision and scale.

See the Changelog for the full list of fixes.

SDK Updates​

Updated SDKs release alongside v2.2.0:

• spice.js v3.2.0 — adds query cancellation (listActiveQueries/cancelActiveQuery), mTLS client certificates, HTTP fallback when Flight is unavailable, and typed parameter binding via Flight SQL prepared statements. Also fixes nsql() and search() response handling and named parameters over HTTP.
• spicepy v4.0.0 — adds streaming of large results, natural-language queries (nsql), vector and hybrid search, and query cancellation. Spice.ai Cloud users need this update: the legacy hostnames are retired in favor of region-specific endpoints.
• spice-rs v4.0.0 — adds search, NSQL, async queries, query cancellation, and mTLS.

Dependency Updates​

New Contributors​

• @boittega made their first contribution in #11509
• @lesbass made their first contribution in #12253

Contributors​

• @bjchambers
• @boittega
• @ewgenius
• @Jeadie
• @krinart
• @lesbass
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @v1gnesh

Breaking Changes​

• CPU sizing for Kubernetes pods without a CPU limit: A Spice runtime pod with a CPU request and no CPU limit now sizes itself to twice the request instead of every core on the node. Set SPICE_CPU_CORES=all (or runtime.cpu.cores: all) to keep the previous behavior, or set runtime.cpu.cores to a specific core count.
• Partitioned DuckDB accelerations are removed: The DuckDB accelerator now rejects partition_by. Use the Cayenne or Arrow accelerator for partitioned datasets.
• Adaptive Cayenne tuning requires an explicit opt-in: An unset cayenne_tuning now resolves to auto. Set cayenne_tuning: adaptive to enable the closed-loop controller.
• ONNX ML inference is removed: The runtime no longer loads ONNX models, and it no longer serves the /v1/predict endpoints. Use an LLM model provider instead.
• One process-wide Vortex segment cache: Cayenne tables now share one segment cache instead of one cache per table. Set cayenne_segment_cache_mb under runtime.params to size it. When unset, the cache takes 1/64 of the memory entitlement, clamped to 256 MiB - 2 GiB.
• The Pingora cache engine is now a Spice.ai Enterprise feature: An OSS build with engine: pingora degrades to the Moka engine and logs the substitution.
• Deprecated settings: pg_replication_temporary_slot is deprecated. cayenne_segment_cache_mb at the table level is deprecated — set it under runtime.params instead. This release renames the MongoDB num_docs_to_infer_schema setting to schema_infer_max_records. The old name still works and warns.

Cookbook Updates​

• Added a new recipe for MySQL CDC (Binlog Replication): stream real-time changes from a MySQL table into Spice with refresh_mode: changes.
• Added a new recipe for Aurora MySQL CDC (Binlog Replication): the same CDC setup for an AWS-hosted Amazon Aurora MySQL cluster.
• Added a new recipe for PostgreSQL Catalog CDC: replicate every table of a PostgreSQL database with one catalog configuration.
• Added a new recipe for Cloud Connect on a Development Machine: connect a local Spice instance to Spice.ai Cloud, deploy live changes, and deliver a secret.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.2.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.2.0 image:

docker pull spiceai/spiceai:2.2.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.2.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• feat(testoperator): --prepare-only / --skip-prepare for htap source reuse by @bjchambers in #11486
• test(cluster): de-flake scheduler_failover (drop unreliable scheduler_node assertion) by @phillipleblanc in #11518
• feat(cayenne): wire orphaned-DV cleanup knob to spicepod params + doc sync by @bjchambers in #11523
• fix(cayenne): warn (not panic) on benign manifest/listing race; re-enable now-passing ignored tests by @lukekim in #11511
• fix(datafusion): accurate projected scan byte size so hash joins build the smaller side by @sgrebnov in #11503
• fix(cayenne): seed persisted num_rows for hash-join sizing by @sgrebnov in #11515
• fix(cayenne): user-visible DELETE WHERE pk IN (...) reports the real row count by @lukekim in #11514
• test(cayenne): CDC convergence/resurrection fuzz harness by @bjchambers in #11502
• fix(cayenne): correctness & memory_limit fixes from perf audit (2 P0, 3 P1) by @lukekim in #11516
• perf(cayenne): compaction IO-hygiene follow-ups — input-page evict + O_DIRECT writer + bench by @lukekim in #11498
• Update end_game.md by @bjchambers in #11535
• chore: Vendor ttf-parser, lopdf, pdf-extract by @peasee in #11521
• fix: Update tpch benchmark snapshots for federated/mysql[catalog].yaml by @app/github-actions in #11531
• Fix s3 vectors API by @krinart in #11536
• Update from spiceai/datafusion#177 by @Jeadie in #11541
• perf(cayenne): off-write_lock N>1 mem-tier checkpoint + adaptive query-admission throttle by @lukekim in #11538
• fix(flightsql): propagate bearer token to per-endpoint clients by @boittega in #11509
• fix: Placeholder table initialization lock swap by @peasee in #11540
• chore(cluster): bump ballista pin for the null-aware anti-join fix by @phillipleblanc in #11544
• fix(runtime-tools): fix memory table identifier validation rejecting valid names by @Jeadie in #11546
• test(chbench): enable 4-way mem-tier PK sharding on SF1000 memory-durability configurations by @sgrebnov in #11553
• fix(reorder): reorder inner-join islands inside EXISTS / NOT EXISTS by @sgrebnov in #11552
• feat(cayenne): enable orphaned-DV cleanup by default (per-table threshold 20) by @bjchambers in #11533
• fix(postgres): delete old key on CDC primary-key updates by @phillipleblanc in #11551
• Properly exclude vendored crates from make lint-rust-fix by @krinart in #11560
• fix: arrow IndexedMemTable serves stale results after DML/retention/sync (fixes #11262) by @claudespice in #11532
• perf(cayenne): concurrent per-shard mem-tier checkpoint encode (drain parallelism) by @lukekim in #11558
• fix(physical-plan): estimate col=literal selectivity from NDV, not the flat 20% default (Datafusion) by @sgrebnov in #11562
• fix: Read Postgres DB source error when available by @peasee in #11566
• feat(cayenne): cold object-store tier with read-optimized clustering by @lukekim in #11543
• Generic CDC replication-lag metric by @krinart in #11554
• fix: ignore transitive quick-xml advisories (RUSTSEC-2026-0194/0195) in cargo-deny by @krinart in #11571
• test(chbench): add cold-object-store-tier SF-1000 HTAP run; cap SF-1000 mem-tier at 1 GiB by @lukekim in #11573
• fix: surface silent Turso read-path conversion failures (fixes #11276) by @claudespice in #11570
• feat(testoperator): emit P99 + Max replication lag and staleness telemetry for HTAP by @sgrebnov in #11572
• feat(runtime): cost-based eager aggregation, enabled by default (datafusion spiceai-54) by @bjchambers in #11446
• Bump datafusion to include spiceai/datafusion#181 by @Jeadie in #11563
• fix(cayenne): drain in-flight maintenance before in-process reopen (mutation_property flake) by @bjchambers in #11578
• Unified JSON nesting by @krinart in #11555
• feat(testoperator): capture under-load EXPLAIN plans as HTAP artifacts by @sgrebnov in #11576
• feat(cayenne): freshness-SLO-driven adaptive mem-tier shrink (+ freshness diagnostics) by @lukekim in #11574
• fix(cayenne): fold un-checkpointed mem-tier keys into the cold keyset rebuild by @lukekim in #11592
• perf(cayenne): reuse the PK RowConverter across the sharded CDC apply by @lukekim in #11590
• Add labels to spiceai-dev by @krinart in #11591
• Rename mongo num_docs_to_infer_schema (deprecated) -> schema_infer_max_records by @krinart in #11593
• fix(postgres): don't ACK replication slot past uncommitted envelopes on empty transactions by @bjchambers in #11582
• refactor(connectors): extract abfs, adbc, cosmosdb, ducklake, gcs, git, github, glue, kafka, snowflake-native, spiceai into data-connectors crates by @Jeadie in #11407
• Revert "Generic CDC replication-lag metric" by @lukekim in #11600
• refactor(cayenne): remove orphaned-DV cleanup spicepod param, make it always-on by @bjchambers in #11575
• fix(mongodb): substitute delete when UpdateLookup finds no document by @bjchambers in #11583
• ci(chbench): SF1000 tuned+adaptive every 3h, other configs once daily by @lukekim in #11601
• feat(cayenne): add integer support to maintained AVG by @bjchambers in #11564
• Refresh spidapter Spice Cloud token from client credentials by @krinart in #11594
• fix: cluster filtered COUNT(*) on append-only tables returns unfiltered total (fixes #11599) by @claudespice in #11603
• fix(scylladb): error on lossy CQL decimal conversion instead of silent truncation/NULL (fixes #11267) by @claudespice in #11604
• ci(chbench): drop postgres-arrow and tuned (non-memory) from scheduled HTAP runs by @lukekim in #11606
• ci(chbench): lower tuned SF1000 mem-tier max age to 15s for scheduled HTAP runs by @lukekim in #11618
• fix(cayenne): fold the un-checkpointed mem-tier into the persisted-bloom PK-index rebuild (SF-100 over-count) by @lukekim in #11609
• Potential fix for 1 code quality finding by @lukekim in #11614
• fix(postgres): shared replication slot drops WAL for partitioned tables (fixes #11290) by @claudespice in #11607
• bench(chbench): add 4-slot CDC variant pod for SF-1000 memory (separate from default) by @bjchambers in #11605
• fix(cayenne): don't serve drifted memory-CDC row count as Exact to COUNT(*) by @bjchambers in #11602
• chore(deps): vendor pgwire-replication 0.3.2 by @bjchambers in #11617
• Add SQL warehouse ID to integration workflow by @Jeadie in #11561
• fix(search): cosine_distance returns NULL not NaN for zero-magnitude vectors by @claudespice in #11621
• feat(spidapter): tuned + adaptive MongoDB CDC spicepods with scenario selection by @bjchambers in #11579
• Potential fixes for 3 code quality findings by @lukekim in #11613
• Potential fixes for 2 code quality findings by @lukekim in #11611
• Potential fixes for 2 code quality findings by @lukekim in #11615
• Potential fix for 1 code quality finding by @lukekim in #11612
• feat(cayenne): split mem-tier into ingestion + immutable pieces; advance the replication slot on a cheap periodic seal (default 2s) by @lukekim in #11622
• fix(mcp): encode catalog-qualified tool names without '/' (fixes #10894) by @claudespice in #11629
• fix(refresh): apply refresh_sql override on manual refresh when source unchanged (fixes #11353) by @claudespice in #11626
• fix: report empty accelerated schema instead of misleading missing-column error (fixes #10920) by @claudespice in #11625
• fix(imap): store message date as millisecond Timestamp, not seconds in Date64 (fixes #11547) by @claudespice in #11624
• test(bucket): guard frozen partition-hash output against silent ahash drift (addresses #11277) by @claudespice in #11623
• Bump datafusion (spiceai/datafusion#182) and datafusion-table-providers (#27): fix q16 CollectLeft planning error and SQLite q6 wrong revenue by @Jeadie in #11598
• fix(cluster): prevent executor flaps and distributed query stalls by @phillipleblanc in #11565
• fix(cli): allow spice trace on dynamic MCP/custom tool_use tasks (fixes #10995) by @claudespice in #11630
• docs: add clang/lld to arm64 Linux build prerequisites (fixes #11120) by @claudespice in #11632
• perf(cayenne): mem-tier visible-batch memo + maintained MIN/MAX IVM by @lukekim in #11631
• docs: consolidate agent instructions and fix stale references by @lukekim in #11634
• build(deps): bump the github-actions-dependencies group across 2 directories with 7 updates by @app/dependabot in #11627
• perf(flight): offload query-result IPC encoding off the IO runtime by @lukekim in #11608
• feat(runtime): reservation-aware Cayenne CDC query-memory default (75%→70%) by @lukekim in #11641
• Revert "ci(chbench): lower tuned SF1000 mem-tier max age to 15s for scheduled HTAP runs" by @lukekim in #11648
• fix(cayenne): don't bump mem-tier version on seal (QPH regression from #11622) by @lukekim in #11649
• feat(cayenne): vendor versioned row_converter module (byte-compatible with arrow-row 58.3.0) by @phillipleblanc in #11654
• fix(flightsql): advertise analyzed query schemas by @phillipleblanc in #11656
• feat(cayenne): bounded append freshness — segmented stream publishing + publish-triggered executor stats broadcast by @phillipleblanc in #11620
• fix(postgres): decouple replication keepalive/feedback from apply backpressure by @bjchambers in #11653
• fix: bump DataFusion for eager decimal SUM schema by @phillipleblanc in #11657
• fix(cayenne): stop mem-tier snapshot storms under memory pressure and byte-range splitting of small snapshot scans by @sgrebnov in #11645
• fix(ducklake): add opt-in automatic_migration to attach older catalogs (fixes #10899) by @claudespice in #11635
• ci(e2e): tolerate empty-type DuckDB variant of the parquet-rename race by @claudespice in #11636
• build(deps): bump the aws-sdk group with 3 updates by @app/dependabot in #11628
• ci(chbench): FIFO-queue HTAP runs via concurrency queue: max; simplify dispatcher by @bjchambers in #11659
• fix(bucket): guard against ahash AES-path re-bucketing of partition data (fixes #11277) by @claudespice in #11665
• fix(cayenne): gate metastore reinsert_sequence schema behind a user_version (fixes #11291) by @claudespice in #11651
• fix(cayenne): durable CDC delete keeps the pk IN (...) fast path (fixes #11633) by @lukekim in #11642
• ci(chbench): make scheduled HTAP runs adaptive-only (retire tuned, convert cold tier to adaptive) by @lukekim in #11677
• fix(imap): decode text body into content instead of storing raw MIME (fixes #11549) by @claudespice in #11667
• feat(cayenne): end-to-end integrity checksums for WAL records and Vortex data files (fixes #11639) by @claudespice in #11646
• feat(spidapter): date-prefix per-run MongoDB CDC databases by @bjchambers in #11658
• test(cayenne): fuzz mem-tier CDC + seq-prefix bake; harden durable-path count exactness by @bjchambers in #11643
• deps: bump spiceai/datafusion to pick up ORDER BY alias unparser fix by @Jeadie in #11655
• fix(postgres_replication): size CDC change-batch builders to the transaction row count by @sgrebnov in #11675
• Release notes for v2.1.0 by @Jeadie in #11429
• perf(pgwire-replication): incremental zero-copy framing (FrameReader) + bounded message size by @lukekim in #11668
• fix(cdc): memory-mode changes stream no longer fatally stops on a transient deferred-commit queue (fixes #11644) by @bjchambers in #11678
• test: update snapshots from job 84685029883 by @Jeadie in #11569
• fix: Distinguish no filters vs lit(false) for RefreshSql distributed accelerations by @peasee in #11596
• Fix cargo-deny crossbeam-epoch advisory by @Jeadie in #11681
• fix: Update benchmark snapshots for trunk run by @app/github-actions in #11666
• chore(deps): upgrade Vortex to 0.76.0 by @lukekim in #11682
• refactor: extract runtime-metrics crate, move ComponentType to runtime-api-types by @Jeadie in #11524
• feat(cloud-connect): standalone instance adoption by @lukekim in #11060
• docs: keep the feature set constant across a session (incremental-build hygiene) by @bjchambers in #11679
• feat(mysql): binlog replication for refresh_mode: changes by @lukekim in #11672
• feat(cdc): cap keys per durable delete plan + absorb fall-through metrics (#11673) by @bjchambers in #11680
• ci(chbench): run scheduled SF1000-adaptive HTAP bench every 6h instead of 3h by @lukekim in #11698
• fix: MySQL connector fails to convert unsigned integer types (fixes #8364) by @claudespice in #11691
• observability(cdc): localize CDC back-pressure across the ingest pipeline by @bjchambers in #11610
• fix(cache): cache empty (zero-row) SQL result sets by @bjchambers in #11699
• Update spicepod.schema.json by @Jeadie in #11706
• Add support for MysSQL in Ch-Bench by @krinart in #11700
• observability(chbench): limit CH-BenCHmark logs to info and above by @bjchambers in #11711
• perf(cdc): cache per-dataset metric labels to avoid per-event string copies by @bjchambers in #11712
• feat(postgres): pgoutput binary-format CDC decoding + zero-copy decoder by @bjchambers in #11702
• Update integration_models.yml by @Jeadie in #11715
• fix(cayenne): zero-row append refresh fails with "No such file or directory" by @sgrebnov in #11710
• perf(cdc): cut shared-slot Postgres replication pump per-event overhead by @bjchambers in #11709
• fix(spicepod): reject unknown fields on columns[] config (fixes #10972) by @claudespice in #11719
• feat(cayenne): tier-gated O_DIRECT compaction writer (EBS/NAS network storage) by @lukekim in #11696
• fix(cayenne): harden Vortex compaction writes by @phillipleblanc in #11692
• perf(vortex): push boolean NOT filters into the Vortex scan by @lukekim in #11721
• chore(chbench): cap scheduled HTAP runs at 250K tpmC (rate 9250) by @sgrebnov in #11723
• MySQL CH-benCHmark HTAP: SF1 + SF1000 adaptive configs by @krinart in #11722
• feat(testoperator): TPC-DS SF100 federated cluster-bench spicepod + default spicehq nodegroup by @phillipleblanc in #11716
• refactor(runtime): move AccelerationSource and Acceleration into runtime-acceleration by @Jeadie in #11530
• remove(ml): delete model_components crate and ONNX/Tract ML inference by @Jeadie in #11684
• ci: dedupe testoperator query_overrides run steps by @Jeadie in #11690
• Update search snapshots by @Jeadie in #11714
• refactor: use inherent is::<T>() on dyn TableProvider/dyn ExecutionPlan for type checks by @Jeadie in #11585
• chore: remove unused BlueOak license allowance by @phillipleblanc in #11762
• perf(vortex): remove per-file and per-segment allocations on the scan path by @lukekim in #11733
• fix(ci): use GraphQL API to check isLatest in spiced_docker workflow by @Jeadie in #11735
• fix(release): use release notes heading as GitHub release title by @Jeadie in #11744
• feat(chbench): headline run summary + adaptive slot-topology variants by @bjchambers in #11734
• fix(ci): update e2e CLI tests for Windows-no-runtime and chat error wording by @Jeadie in #11717
• build(rust): upgrade toolchain to 1.96.1 by @lukekim in #11743
• fix(deps): enable reqwest "query" feature for isolated crate builds by @claudespice in #11760
• build: define clippy lints in [workspace.lints], opt in every crate by @bjchambers in #11676
• refactor(runtime): parameterize table-provider unwrapping by inner-fn set by @Jeadie in #11759
• fix(cayenne): retain source deletion index in scan memos to close ABA hazard (fixes #11303) by @claudespice in #11739
• fix(mysql): refuse binlog resume across source layout drift by @phillipleblanc in #11761
• perf: avoid unnecessary allocations on search hot paths by @phillipleblanc in #11767
• feat(spidapter): support cluster_name for private nodegroup targeting by @phillipleblanc in #11740
• feat(cayenne): finalize datalake (cold) tier v1 UX, credentials, and e2e integration test by @sgrebnov in #11731
• perf(vortex): push CAST(CASE ... END) filters into the Vortex scan by @lukekim in #11729
• docs: lead README with real-time analytics-node + CDC messaging (Spice 2.0) by @lukekim in #11774
• chore(chbench): extend scheduled HTAP runs to 15m by @lukekim in #11773
• perf(cayenne): lazy NDV — compute on file spill, off the inline hot loop by @bjchambers in #11741
• ci(rust): install 1.96.1 in workflows that pin the toolchain explicitly by @lukekim in #11769
• perf(cdc): deferred parsing — move Postgres CDC decode + build off the shared pump by @bjchambers in #11732
• docs: correct sort-kernel SIMD note (AVX2, not AVX-512) by @lukekim in #11771
• fix(cayenne): roll cold-tier files at cayenne_cold_target_file_size_mb by @sgrebnov in #11746
• docs(cayenne): move the Cayenne technical reference into the repo by @bjchambers in #11775
• perf(cayenne): unify upsert PK-key hashing on prehashed XXH3-128 digests by @bjchambers in #11736
• perf: avoid unnecessary allocations on query hot paths by @phillipleblanc in #11768
• Extended schema inference for mysql by @krinart in #11742
• chore(hash-index): add SBBF probe/insert baseline benchmark by @lukekim in #11770
• feat(cayenne): incremental datalake promotion, per-file PK blooms, physical GC by @sgrebnov in #11745
• test(cayenne): limit property-test concurrency by @phillipleblanc in #11784
• perf(cdc): lock-free per-member ack slots + coalescing for shared Postgres replication by @bjchambers in #11779
• perf(chbench): speed up MySQL and Postgres seed loading by @bjchambers in #11782
• fix(ci): download MinIO client from public-data mirror by @phillipleblanc in #11790
• ci: developer sign-off attestation gating the merge queue by @lukekim in #11776
• fix(tests): synchronize chat expect prompts correctly by @phillipleblanc in #11802
• perf(chbench): default adaptive pod to 4-slot grouping + deep prefetch by @bjchambers in #11799
• refactor(cli): replace open crate with in-tree system-open by @phillipleblanc in #11791
• refactor(runtime): drop direct rustls-pemfile dependency by @phillipleblanc in #11792
• Cache ListingTable file statistics to avoid per-query footer re-parse by @phillipleblanc in #11793
• ci: inherit sign-off across clean base merges by @phillipleblanc in #11804
• perf(cayenne): hash PK once and skip OwnedRow clone on present path by @bjchambers in #11805
• fix(postgres catalog): register partitioned parent, not child partitions by @bjchambers in #11798
• fix(postgres catalog): quote foreign-key target identifiers by @bjchambers in #11796
• refactor(cayenne): rename remaining cold_* params to cayenne_datalake_* by @sgrebnov in #11795
• build: add release-profiling Cargo profile for CPU profiling by @bjchambers in #11807
• fix(ci): make CH-benCH template restore robust to leftover replication slots by @sgrebnov in #11811
• perf(cayenne): remove eager-NDV escape hatch, speed up per-value NDV hashing by @bjchambers in #11806
• Auto-refresh the Attestation check after signoff by @bjchambers in #11815
• refactor(duckdb): remove partitioned DuckDB accelerator modes by @lukekim in #11808
• feat(testoperator): add option to skip the HTAP analytic gate by @bjchambers in #11810
• fix: prevent executor S3 region poisoning before object-store bind by @phillipleblanc in #11766
• perf(cayenne): reuse per-batch scratch allocations in KeyBasedDeletionFilterStream by @bjchambers in #11817
• Update openapi.json by @app/github-actions in #11816
• build(deps): bump the github-actions-dependencies group across 2 directories with 5 updates by @app/dependabot in #11819
• fix(task_history): capture ExplainAnalyze metrics from the executed plan by @phillipleblanc in #11794
• feat: generic spiced env-var passthrough for HTAP testoperator runs by @bjchambers in #11824
• feat(cayenne): datalake tier hardening + row-capped PK-bloom-backed promotion by @sgrebnov in #11812
• chore: release finalization for v2.2.0 by @Jeadie in #11738
• feat(cayenne): add memory mode (mode: memory) — fully in-RAM accelerator by @lukekim in #11720
• Fix secrets in .github/workflows/integration_models.yml. by @Jeadie in #11781
• feat(runtime): add runtime.query.timeout parameter by @sgrebnov in #11822
• feat(cloud-connect): align proto to canonical, fix enrollment ordering, add TLS e2e by @lukekim in #11829
• docs(criteria): mark PostgreSQL Catalog Connector as Alpha by @bjchambers in #11785
• test(cayenne): add regression tests for the mid-pass overwrite guard by @sgrebnov in #11844
• perf(cayenne): light delta encoding + higher CDC coalescing defaults by @bjchambers in #11826
• ci: run Cayenne doc PDF build only on trunk merges by @bjchambers in #11860
• fix(cayenne): drain staged Stage-B publishes before cold promotion; add cold-tier fuzz coverage by @sgrebnov in #11847
• feat(cayenne): atomic cross-partition append + delete/on-conflict atomicity by @lukekim in #11803
• feat(cayenne): min/max maintained aggregates + N>1 CDC retract by @lukekim in #11862
• perf(chbench): CSV-based seed loading for MySQL and Postgres by @bjchambers in #11843
• fix(tests): re-enable prop_concurrent_cold_sqlite cayenne cold-tier fuzz test by @sgrebnov in #11871
• fix(deps): replace yanked spin releases by @phillipleblanc in #11878
• fix(cayenne): correct stale mem_checkpoint_lock comment by @bjchambers in #11872
• Add run links to testoperator_dispatch.yml by @Jeadie in #11747
• Cayenne serializable transactions: gated writes, per-key OCC, multi-table, FlightSQL, durable write-back by @phillipleblanc in #11870
• fix(cayenne): partitioned datasets deadlock against the global encode budget and never become ready by @Jeadie in #11825
• test(runtime): re-runnable turso file cleanup + bump turso to 0.7.0 by @lukekim in #11783
• Use liteparse for PDF document parsing by @Jeadie in #11522
• ci: add remote signoff workflow by @Jeadie in #11864
• fix(cli): populate org in spice cloud apps --output json (fixes #11041) by @claudespice in #11867
• fix(embeddings): restore params broken by #10853 by @Jeadie in #11788
• Move cargo advisory checks to scheduled workflow by @Jeadie in #11879
• perf(cayenne): bound cold-promotion Z-order sort into streaming byte-capped runs by @sgrebnov in #11890
• fix(cayenne): write-back transaction atomicity — stage (not publish) and read mem-tier rows by @phillipleblanc in #11889
• fix(runtime-table-partition): restore sound modulo inequality partition pruning by @Jeadie in #11891
• Replace dotenvy with in-repo dotenv crate by @phillipleblanc in #11894
• Remove unnecessary allocations from results-cache and hot conversion paths by @phillipleblanc in #11895
• Increase ready_wait timeout for chbench sf1000 cold-tier configs by @sgrebnov in #11900
• fix(mysql): retry binlog checkpoint upsert on transient accelerator write lock by @krinart in #11876
• fix(cayenne): gate freshness mem-tier shrink on apply backlog by @lukekim in #11893
• fix(postgres catalog): honor unsupported_type_action for catalog-discovered tables by @bjchambers in #11875
• fix(postgres): make TPC-H/TPC-DS benchmark CI actually run at SF1/SF10/SF100 by @bjchambers in #11883
• feat(search): writethrough compound SearchIndex/VectorIndex with optional fallback by @Jeadie in #11892
• Revert "perf(cayenne): light delta encoding + higher CDC coalescing defaults" by @bjchambers in #11910
• perf(vss): SIMD-accelerate cosine_distance via simsimd by @Jeadie in #11748
• Add run-name to signoff.yml by @Jeadie in #11912
• fix(cayenne): serialize cold-tier mem-tier checkpoint by @sgrebnov in #11907
• fix(postgres catalog): discover materialized views and foreign tables by @bjchambers in #11874
• test(chbench): stronger content fingerprint + MySQL sidecar template caching by @krinart in #11901
• test(chbench): align money columns to canonical DECIMAL schema (CMU BenchBase) by @sgrebnov in #11921
• fix(cayenne): close per-key OCC missed-conflict holes and fused-txn IVM staleness by @lukekim in #11916
• ci(attestation): fast-track pure reverts past developer sign-off by @bjchambers in #11913
• fix(cloud-connect): accept the portal's 5-char adoption-code segments by @phillipleblanc in #11926
• docs(layering): codify crate tiers + workspace layering guard by @bjchambers in #11919
• refactor(cayenne): datalake tiering UX — param rename + tracing by @sgrebnov in #11920
• feat(params): typed component params via #[derive(TypedParams)] — embeddings pilot by @Jeadie in #11809
• Fix SQlite round type error. by @Jeadie in #11927
• fix(signoff): let remote sign-off refresh the Attestation check in CI by @bjchambers in #11929
• feat(testoperator): print rows around CH-benCH analytical-gate mismatches by @bjchambers in #11923
• fix(postgres catalog): don't abort the whole catalog on one schema's discovery failure by @bjchambers in #11873
• ci(signoff): target-lint changed crates before full local/remote gate by @lukekim in #11909
• feat(runtime)!: always-on schema inference; remove schema_inference config by @lukekim in #11880
• refactor(layering): extract ClickHouse into connector-clickhouse; add restricted_deps guard by @bjchambers in #11931
• perf: remove unnecessary allocations on hot query, search, and metrics paths by @phillipleblanc in #11918
• Release notes for v2.1.1 by @Jeadie in #11906
• refactor(layering): prepare DynamoDB for extraction from runtime by @bjchambers in #11935
• Coalesce Null primary keys in RRF. by @Jeadie in #11519
• ci(e2e): strip ANSI before duckdb_append graceful-shutdown log whitelist by @phillipleblanc in #11937
• feat(catalog): PostgreSQL catalog-level CDC acceleration (changes mode) by @bjchambers in #11897
• chbench(mysql): parallel analytical gate, decimal-comparison fix, and reseed/CI hardening by @krinart in #11946
• fix: array_any_value panic when output is hash-repartitioned (empty list elements) by @bjchambers in #11952
• feat(layering): checkpoint sidecar as per-engine crates + DI the dynamodb connector by @bjchambers in #11938
• chore(cluster): pin Ballista to upstream 54 merge tip by @phillipleblanc in #11941
• feat(vortex): upgrade fork pins to Vortex 0.79.0 by @lukekim in #11950
• chbench(htap): run mysql configs at the shared 9250 rate; 2x converge wait by @sgrebnov in #11956
• fix(testoperator): HTAP drain gate false-fails on stale probe observations — final snapshot decides convergence (fixes #11953) by @claudespice in #11966
• feat(catalog): replica-identity-aware eligibility for PostgreSQL catalog CDC by @bjchambers in #11951
• refactor(layering): extract DynamoDB into connector-dynamodb by @bjchambers in #11960
• Add MemoryVectorIndex: in-memory external-store VectorIndex with brute-force exact k-NN by @Jeadie in #11908
• ci(signoff): prefer lab SSH for remote sign-off; skip Rust when no .rs changes by @lukekim in #11977
• Reapply #11826: light delta encoding + CDC coalescing (SF1000 convergence lever) — gated on #11943 + SF1000 fingerprint run by @lukekim in #11944
• docs: align component statuses with signed release criteria by @lukekim in #11970
• fix: Spidapter flight auth passthrough by @peasee in #11976
• Unified CDC config by @krinart in #11777
• feat(cdc): Debezium plugin push ingest to Spice without Kafka by @lukekim in #11955
• refactor(connectors): unify registration on the linkme slice; restore schema coverage by @bjchambers in #11972
• GTID-based MySQL CDC by @krinart in #11813
• test(mysql): ignore mysql_binlog_replication_end_to_end_cayenne by @sgrebnov in #11986
• fix(cayenne): support Decimal128 in maintained SUM/AVG aggregates (fixes #11933) by @claudespice in #11979
• feat(params): typed params for VectorStore and FtsStore engines by @Jeadie in #11954
• chore(layering): remove orphaned dead code from connector extractions by @bjchambers in #11988
• feat(cayenne): default-on adaptive cold layout from observed filters (F4) by @lukekim in #11973
• fix: Fairly allocate partitions from scheduler assignment cycle only by @peasee in #11853
• fix(cli): avoid escaped ANSI version notification by @ewgenius in #11996
• MySQL Shared binlog connection by @krinart in #11814
• feat(catalog): fail-loud + metrics + by-kind reporting for PostgreSQL catalog CDC (#11850) by @bjchambers in #11983
• fix(cayenne): P1 audit — subset compact, SMJ 2.5× HT, mem-tier pool account by @lukekim in #11991
• Add OAuth2 client-credentials grant and configurable auth header to HTTP connector by @krinart in #11981
• fix(telemetry): validate runtime.telemetry.metric_prefix against OTel name syntax by @ewgenius in #12002
• fix(cayenne): purge CDC mem-tier on delete-all/TRUNCATE by @sgrebnov in #12009
• Update signoff.yml. by @Jeadie in #11998
• feat(dev): add scripts/signoff mine — attestation status across your open PRs by @Jeadie in #11978
• refactor(layering): extract MongoDB provider into connector-mongodb by @bjchambers in #11982
• perf: remove three unnecessary allocations on the query hot path by @phillipleblanc in #12029
• refactor(layering): evacuate cosmosdb/graphql/github/sharepoint providers into their connector crates by @bjchambers in #12024
• fix(mysql-cdc): detect source reset on GTID resume by @sgrebnov in #12023
• fix(smb): exclude final SESSION_SETUP response from preauth integrity hash (fixes #11148) by @claudespice in #12003
• fix(postgres): validate pg_replication_slot names before CDC refresh by @ewgenius in #12001
• fix(postgres): honor pg_connection_string for CDC (closes #11994) by @ewgenius in #12000
• feat: Support OTLP Histogram ingest, unix nanos time by @peasee in #11992
• feat(search): write-through warm + fallback compound index for FTS (#11886) by @Jeadie in #11971
• docs: PR-description and code-comment conventions by @bjchambers in #12032
• refactor(postgres cdc): consolidate every dataset onto the shared pump by @bjchambers in #12028
• feat(cloud-connect): Standalone instance adoption connection by @peasee in #11980
• fix(cdc): don't force the durable path on zero-row readiness heartbeats (fixes #12007) by @claudespice in #12030
• Upgrade DataFusion to 54.1.0 by @krinart in #11974
• perf(github): parallelize serial fetches (commits + workflow logs) by @lukekim in #12017
• refactor(layering): evacuate odbc/scylladb/imap/git providers into their connector crates by @bjchambers in #11993
• refactor(layering): carve runtime-component out of runtime, break the ArcRuntime cycle by @bjchambers in #12031
• feat(mssql): enable tiberius integrated-auth-gssapi for Kerberos integrated auth on Unix by @v1gnesh in #11386
• build(deps): bump the aws-sdk group across 1 directory with 5 updates by @app/dependabot in #11821
• fix(cayenne): clear a committed CDC upsert's staging WAL at finalize (#12027) by @bjchambers in #12034
• feat(catalog): deterministic instance-independent replication slot + fail-loud + restart recovery (#11850) by @bjchambers in #12026
• Auto-cap DuckDB accelerator memory to prevent startup over-commit by @lukekim in #11985
• perf(refresh): speed up dataset refresh at startup by @lukekim in #12015
• fix(cayenne): don't create stray file: directory in memory mode (fixes #11922) by @claudespice in #11940
• fix(aws): accept aws_session_token for temporary credentials (fixes #10932) by @claudespice in #12041
• fix(mysql): verify an adopted layout against the event's own column types (fixes #11764) by @claudespice in #12048
• fix(mysql): take the binlog rotate target from the ROTATE event, not its header offset (fixes #12042) by @claudespice in #12044
• fix(search): check dataset readiness before embedding the query (fixes #10956) by @claudespice in #12043
• fix(databricks): pin the authentication mode with databricks_auth_mode so an auto-loaded client secret can't switch U2M to M2M (fixes #11508) by @claudespice in #12040
• fix(imap): narrow scans with IMAP SEARCH instead of refetching the mailbox (fixes #11548) by @claudespice in #12039
• fix(cayenne): rank inference-derived sort columns below observed filter columns by @lukekim in #12049
• perf(cayenne): warm-subset compaction + mem-tier memory_limit honesty by @lukekim in #12035
• fix(cayenne): purge the mem-tier on delete-all for a table without a primary key (fixes #12072) by @claudespice in #12073
• fix(accelerator): let an index declare a finalize failure fatal so a stale index can't report a successful refresh (fixes #12038) by @claudespice in #12050
• fix(http): restore a join's embedded projection in HTTP subquery pushdown (fixes #11009) by @claudespice in #12054
• fix(e2e): report a REPL that exits mid-script instead of passing the step (fixes #12057) by @claudespice in #12059
• fix(imap): fetch the raw message only when a scan reads content (fixes #12045) by @claudespice in #12060
• chore: Update Turso crate to 0.7.1 by @claudespice in #12062
• feat(chbench): in-memory _bench_ts watermarks for the MySQL HTAP staleness probe by @sgrebnov in #12055
• test(chbench): default MySQL adaptive spicepods to one shared binlog dump by @sgrebnov in #12076
• feat(search): warm in-memory writethrough/fallback index for .vectors datasets and views by @Jeadie in #11914
• fix(search): expunge superseded documents from full-text BM25 statistics (fixes #12053) by @claudespice in #12056
• fix(write-back): gate durable write-back on a safe source delivery primitive (fixes #11915) by @claudespice in #12051
• fix(cayenne): keep protected snapshots a position-delete rewrite never folded in (fixes #11477) by @claudespice in #12052
• ci(signoff): fast-track Attestation for Dependabot and non-Rust PRs, and cut the local gate by @lukekim in #12081
• fix(search): reach a compound's inner full-text tier when a change stream attaches (fixes #12061) by @claudespice in #12063
• fix(accelerator): report PK equality pushdown Inexact so a point lookup can't return the whole table (fixes #12070) by @claudespice in #12071
• fix(github): paginate issues and pull_requests on an immutable sort key so a row touched mid-scan isn't dropped (fixes #12067) by @claudespice in #12069
• fix(otel): scope reserved metric column names to the data-point shape (fixes #12064) by @claudespice in #12065
• fix(cayenne): sample the subset compaction append fence before the listing it guards (fixes #12074) by @claudespice in #12075
• fix(search): drop a search hit whose row is not in the base table (fixes #12089) by @claudespice in #12094
• feat(datasets): mark non-accelerated datasets Error when their source is unavailable by @krinart in #12079
• fix(models): restore spice.ai/spiceai as a model source by @lukekim in #12092
• fix(cayenne): bound the cold PK-index rebuild; build shard views in one pass by @sgrebnov in #12078
• fix: Update Search integration test snapshots by @Jeadie in #12077
• feat(catalog): view warnings, docs-linked messages, and clearer startup summary for PostgreSQL catalog CDC (#11850) by @bjchambers in #12022
• fix(postgres): log the cumulative delivery wait in the sink-stall warning by @sgrebnov in #12093
• refactor(cloud-connect): single-source the sealed-secret wire crypto in a shared crate by @peasee in #12124
• build(deps): bump quinn-proto from 0.11.14 to 0.11.16 by @app/dependabot in #12037
• perf(cayenne): demand-driven scan-view cache with access-based freshness (alternative to #11948) by @bjchambers in #12005
• build: exclude libnfs from the nextest workspace run by @phillipleblanc in #12131
• build(deps): bump the aws-sdk group across 1 directory with 3 updates by @app/dependabot in #12068
• fix(search): keep CDC changes streams working when embeddings wrap a CDC source by @Jeadie in #12086
• fix(cli): report a truncated release download as a failed download (fixes #12120) by @claudespice in #12121
• fix(postgres): honor inline PEM pg_sslrootcert on the replication path by @phillipleblanc in #12128
• refactor(layering): extract RuntimeStatus into a runtime-status crate by @bjchambers in #12115
• fix(libnfs): model AUTH as an opaque type so its layout assertion holds by @phillipleblanc in #12132
• feat(duckdb): on_full_refresh: replace_file — full refresh into a new database file, atomically replaced by @lukekim in #12135
• Add v2.1.2 release notes by @sgrebnov in #12146
• feat(duckdb): 'on_full_refresh: checkpoint_file' to bound acceleration file growth by @sgrebnov in #12139
• fix(lint): remove unused DuckDBTableWriter import in duckdb accelerator tests by @sgrebnov in #12158
• fix(duckdb): drop a file-replacement test assertion that can never hold by @bjchambers in #12178
• fix(cayenne): Remove redundant EmptyExec within UnionExec used by CayenneTableProvider::scan by @Jeadie in #12126
• build(deps): bump async-openai for aggregated rate-limit logging by @phillipleblanc in #12149
• perf(mysql): fast row-image decoder for CDC change builds by @sgrebnov in #12122
• feat(catalog): pre-flight replication-slot-capacity check for PostgreSQL catalog CDC (#11850) by @bjchambers in #12114
• fix(oracle): map DATE to a timestamp so the time-of-day is not silently truncated (fixes #12096) by @claudespice in #12097
• fix(search): skip the warm vector index when nothing hydrates it (fixes #12101) by @claudespice in #12103
• refactor(layering): extract the DataAccelerator contract into data-accelerator-api by @bjchambers in #12099
• fix(cli): surface the registry error behind a failed spice add instead of blaming the archive (fixes #12116) by @claudespice in #12119
• fix(ci): gate Rust checks on the lint config the gate itself reads (fixes #12111) by @claudespice in #12112
• fix: Ensure executors report table statistics for distributed plan ordering by @peasee in #11854
• fix: EMBED_UDF_NAME without models feature by @krinart in #12136
• fix(openai): openai_responses_tools: web_search uses web_search, not web_search_preview by @Jeadie in #12142
• fix(ci): cancel workflow runs left behind by superseded merge-queue batches (fixes #12170) by @claudespice in #12177
• fix(runtime): return from a readiness wait when the runtime shuts down (fixes #12125) by @claudespice in #12165
• perf(postgres): coalesce shared-slot CDC envelopes so a slow sink stops throttling the walsender by @bjchambers in #12147
• fix(ci): make cargo_deny_advisories.yml parseable so the Rust advisory scan can run (fixes #12181) by @claudespice in #12182
• fix(cluster): run GetTaskHistory under read-only SQL validation by @lukekim in #12174
• fix(duckdb): count accelerated views in the coordinated memory budget (fixes #12123) by @claudespice in #12161
• fix(ci): run the PR hygiene check on a GitHub-hosted runner by @claudespice in #12163
• fix(cayenne): warm PK existence caches at first write; bound keyset memory during bulk load by @sgrebnov in #12133
• ci: remote sign-off dispatches GitHub Actions only, never SSH to lab hosts by @lukekim in #12205
• Standalone instances: spice connect enroll-only split + connect surface by @krinart in #12143
• fix(search): Fix ordering to correct NDCG@k by @Jeadie in #12191
• refactor(layering): carry component configuration into connectors and extract data-connector-api by @bjchambers in #12157
• fix(logging): mute the candle embedding backend's GeLU notice below -vv by @phillipleblanc in #12194
• test: redact connection context in federated explain snapshots by @bjchambers in #11800
• Enable deletes for Index, SearchIndex, and VectorIndex by @Jeadie in #11961
• feat(testoperator): trace probe latency breaches and percentiles under HTAP load by @sgrebnov in #12209
• Add v2.1.1 and 2.1.2 as supported in SECURITY.md by @sgrebnov in #12190
• fix(ci): seed the TPC-H Postgres benches from the fleet's dataset and schedule the catalog schema tests by @bjchambers in #12214
• fix(cluster): restrict ExpandSecret to spicepod-referenced keys by @lukekim in #12156
• test: redact endpoint-URL compute contexts in federated explain snapshots by @bjchambers in #12215
• fix(testoperator): a fully-censored table floors worst-P99 at the discard cap by @lukekim in #12196
• ci(e2e): tolerate any garbage type token in the duckdb_append parquet-rename race by @bjchambers in #12207
• refactor(cloud-connect): update the proto contract for evolvability and versioning by @peasee in #12153
• chore(cloud-connect): use port 443 for gateway addresses in the doc example and test fixtures by @phillipleblanc in #12252
• fix(ci): authenticate the management API integration suite with secrets that exist (fixes #12184) by @claudespice in #12185
• feat(search): add Recall@k, MRR@k, and Precision@k retrieval metrics by @Jeadie in #12193
• fix(search): enable stemming by default for full-text search by @Jeadie in #12220
• fix: Report query metrics when task history is disabled by @sgrebnov in #12227
• fix(runtime): count accelerated views when gating the Cayenne compaction pool (fixes #12164) by @claudespice in #12171
• fix(secrets): snapshot the store registry instead of holding its lock across awaits (fixes #12127) by @claudespice in #12166
• fix(search): address Elasticsearch deletes by document _id (fixes #12267) by @claudespice in #12273
• fix(postgres): deprecate pg_replication_temporary_slot, which could never work (fixes #12213) by @claudespice in #12265
• fix(cayenne): always materialize a snapshot directory the catalog may reference (fixes #12208) by @claudespice in #12262
• fix(duckdb): size the connection pool for accelerated views too (fixes #12160) by @claudespice in #12169
• fix(ci): link brew formulas that are installed but unlinked in setup-cc by @claudespice in #12286
• fix(runtime): size memory budgets from the process's own cgroup limit by @lukekim in #12263
• Harden dataset availability checks against idle connection resets by @krinart in #12180
• test(mysql): harden binlog CDC with dump-reconnect and full type-matrix coverage by @sgrebnov in #12264
• fix: send enc_pubkey_pem on /renew for standalone runtime by @phillipleblanc in #12312
• Connection-scale client modes for testoperator throughput and load tests by @lukekim in #12280
• fix(search): remove all chunk documents when a chunked Elasticsearch index deletes a row (fixes #12088) by @claudespice in #12268
• fix(duckdb): run spice_sys DuckDB sidecar writes on the blocking pool (fixes #12175) by @claudespice in #12204
• fix(cayenne): the write-concurrency raise must respect the memory brake by @lukekim in #12317
• feat(runtime): expose the memory numbers that explain an OOM as gauges by @lukekim in #12195
• fix(flightsql): mark stamped statistics inexact when a LIMIT is pushed to the remote scan (fixes #12292) by @claudespice in #12293
• fix(deps): bump wasmtime to 47.0.3 to clear RUSTSEC-2026-0222 (fixes #12295) by @claudespice in #12298
• fix(telemetry): read the cgroup CPU quota along the whole cgroup path (fixes #12299) by @claudespice in #12300
• refactor(layering): move the DataFusion helpers below runtime and derive the federation deny-list from a function registry by @bjchambers in #12210
• fix(search): reject a persisted full-text index whose schema no longer matches the configuration (fixes #12274) by @claudespice in #12275
• fix(spiced): keep dependency logging and chat progress alive when task history is off (fixes #12279) by @claudespice in #12281
• fix(runtime): count a mid-stream response failure as a failure in the HTTP metrics (fixes #12284) by @claudespice in #12291
• fix(cluster): mark a coordinator leaf scan's cached executor statistics inexact (fixes #12303) by @claudespice in #12304
• fix(runtime): report a memory-pool refusal as ResourcesExhausted and answer it with 503 (fixes #12282) by @claudespice in #12289
• fix(cayenne): enforce the sharded PK keyset byte budget; correct per-entry accounting by @lukekim in #12192
• fix(runtime): carve the Cayenne compaction memory pool only when a dataset can compact into it (fixes #12320) by @claudespice in #12326
• feat(spiced): report fatal signals before exit by @sgrebnov in #12334
• bug: Filters on _match fail during planning by @Jeadie in #12247
• bug: S3 metadata-filter conversion silently drops unconvertible AND/OR operands by @Jeadie in #12248
• fix(ci): reject a failed sign-off on the head commit (fixes #12357) by @claudespice in #12362
• ci: let signoff.yml sign off fork PRs by @Jeadie in #12025
• fix(search): resolve an append stream through a vector scan (fixes #12313) by @claudespice in #12314
• docs: Spice v2.1.3 release notes by @sgrebnov in #12378
• bug: MemoryVectorIndex leaves external-store entries on delete. by @Jeadie in #12246
• fix(cluster): keep the executor's cluster-service channel alive while idle (fixes #12301) by @claudespice in #12302
• fix(https): reject OAuth2 params on structured HTTP file datasets (fixes #12315) by @claudespice in #12321
• fix(scheduler): drive the interval-timing tests on a virtual clock (fixes #12323) by @claudespice in #12329
• fix(ci): do not report a shutdown-cancelled sidecar task as a failure (fixes #12322) by @claudespice in #12331
• fix(logging): let log colour follow the output sink (fixes #12327) by @claudespice in #12335
• fix(cayenne): report the auto-tuned config once per resolution (fixes #12330) by @claudespice in #12341
• chore: Update Turso crate to 0.7.2 by @claudespice in #12344
• fix(runtime): stop retrying a dataset configuration failure that no retry can clear (fixes #12339) by @claudespice in #12345
• build(deps): bump the github-actions-dependencies group across 3 directories with 9 updates by @app/dependabot in #12359
• feat(llms): GLM 5.2 across 3+ nodes, context_length + paged_attention params, MXFP4 for DeepSeek-V4 by @lukekim in #11990
• feat(runtime): size every CPU-derived pool from the CPU entitlement by @bjchambers in #12276
• chore(ci): bump the spiceio setup action to v0.5.9 by @lukekim in #12391
• feat(testoperator): serve /health and /v1/ready for the HTAP run by @lukekim in #12373
• feat(catalog): durable storage modes for PostgreSQL catalog CDC, and let the catalog own its accelerated tables by @bjchambers in #12222
• fix(runtime): correctly classify a memory-pool refusal as ResourcesExhausted (fixes #12380) by @sgrebnov in #12382
• fix(ci): budget the sign-off run below the pool's job wall (fixes #12340) by @claudespice in #12343
• fix(bench): install make before the seeded-database check; surface catalog error causes by @bjchambers in #12372
• Clean up SQLite sidecar files in cold bloom catalog test by @sgrebnov in #12404
• fix(ci): name an incoherent Cargo.lock before every cargo job fails on it (fixes #12375) by @grokspice in #12423
• test(search): exercise the S3 Vectors warm-tier fallback path by @Jeadie in #12197
• fix(ci): stop the spiced child E2E cleanup was leaking, scoped to this runner (fixes #12058) by @grokspice in #12431
• perf(cayenne): give refresh_mode full its own write profile by @lukekim in #12338
• fix(ci): skip the Attestation refresh when the sign-off SHA is no longer the PR head (fixes #12360) by @grokspice in #12433
• fix(monitoring): correct p99 queries in the Datadog dashboard by @sgrebnov in #12444
• fix(ci): tell an out-of-disk sign-off apart from a failing branch (fixes #12412) by @grokspice in #12426
• feat(search): warm in-memory tier for chunked Elasticsearch vector columns by @Jeadie in #12082
• Update spicepod.schema.json by @app/github-actions in #12451
• fix(search): preserve field names with capitals and '.' by @Jeadie in #12389
• fix(ci): size the cayenne property-test timeout ceiling to its measured runtime (fixes #12336) by @grokspice in #12438
• fix(models): pass the pinned revision to a HuggingFace embedding model (fixes #12430) by @grokspice in #12446
• fix(ci): make the out-of-disk sign-off verdict survive being out of disk (fixes #12427) by @grokspice in #12461
• fix(ci): stop Enforce Pulls with Spice racing itself, and let a Dependabot PR pass it (fixes #12377) by @grokspice in #12466
• docs: update v2.1.3 release notes for the Iceberg timestamptz fix (trunk sync) by @phillipleblanc in #12459
• fix(ci): install protoc before building the spice CLI in the E2E CLI workflow by @phillipleblanc in #12449
• fix(ci): reject an if: naming a context GitHub does not provide there (refs #12396) by @grokspice in #12468
• fix(monitoring): align Datadog dashboard with OTel service instance identity by @ewgenius in #12454
• chore(ci): bump the spiceio setup action to v0.5.10 by @lukekim in #12490
• test(cayenne): query result correctness vs standalone engines and Spice accelerators by @lukekim in #12098
• fix(elasticsearch): keep the response body out of errors on the row-data request paths (fixes #12409) by @grokspice in #12456
• fix(ci): key the sign-off concurrency group on the commit, not the dispatch input (fixes #12472) by @grokspice in #12474
• perf(ci): run the workspace unit-test gate in one cargo invocation by @bjchambers in #12437
• feat(monitoring): Runtime Resources observability on Grafana dashboard by @ewgenius in #12441
• fix(ci): report cache health on the build failures it explains (refs #12420) by @grokspice in #12471
• fix: resolve fixed-offset timezones when writing Vortex files by @phillipleblanc in #12463
• fix(search): stop re-parsing a dotted index key as relation.column (fixes #12462) by @grokspice in #12464
• fix(ci): build the ADBC BigQuery driver with the Go version its go.mod requires by @grokspice in #12470
• fix(llms): keep the Hugging Face cache token and harden the model E2E job by @grokspice in #12422
• fix(ci): stop a cancelled Remote Sign-off from claiming the checks failed (fixes #12424) by @grokspice in #12425
• fix(runtime): fail the write when an index cannot prepare its write window (fixes #12421) by @grokspice in #12448
• fix(runtime): stop retrying a permanent configuration failure on the catalog and dataset load paths (fixes #12417) by @grokspice in #12443
• chore(ci): bump spiceio setup action to v0.6.0 by @lukekim in #12522
• fix(ci): say what was unready when a spiced readiness wait times out (refs #12473) by @grokspice in #12484
• feat(cayenne): drive incremental vacuum from the maintenance tick by @lukekim in #12435
• feat(ci): allow additional cargo features for spiceai-dev Docker builds by @ewgenius in #12198
• fix(cli): authenticate the SQL REPL's nql line like the session's SQL (fixes #12491) by @grokspice in #12496
• fix(search): keep the row's primary key out of Elasticsearch bulk-index failures (fixes #12370) by @claudespice in #12410
• fix(cpu-budget): state a low CPU request plainly, and only below half the cores by @bjchambers in #12440
• fix(models): reject a pinned Model2Vec revision instead of 401ing on it (fixes #12445) by @grokspice in #12475
• fix(cli): keep a login credential on the origin it was minted for (fixes #12505) by @grokspice in #12508
• bug: S3 vector indexing succeeds when the embedding column is absent by @Jeadie in #12249
• bug: S3 Vectors advertises unsupported metadata predicates as exact pushdown (fixes #12243) by @Jeadie in #12250
• fix(cli): treat a blank --api-key as no key at all (fixes #12498) by @grokspice in #12501
• task: Add #[derive(TypedParams)] for secret stores. by @Jeadie in #12154
• fix(runtime): report a deferred dataset that cannot build its connector (fixes #12414) by @grokspice in #12483
• fix(mysql): pre-flight CDC replication privileges with an actionable error (fixes #11967) by @grokspice in #12485
• fix(testoperator): reject a client fleet the HTTP executors cannot bound (fixes #12348) by @claudespice in #12349
• fix(ci): commit the catalog schema snapshot baselines and stop the daily PR churn by @phillipleblanc in #12525
• refactor(runtime): describe every table-provider wrapper layer once in a layer table by @phillipleblanc in #12200
• Add testoperator dispatch support for search benchmarks and replace benchmarks_search.yml by @Jeadie in #12224
• bug: RRF discards a candidate stream after an empty first batch (fixes #12239) by @Jeadie in #12384
• fix(cli): stop the SQL REPL asking a runtime its queries never went to (fixes #12493) by @grokspice in #12494
• test(runtime): run the metrics tests that are built but never run by @sgrebnov in #12510
• fix(tests): assert the search cache status instead of wall-clock timings (fixes #12487) by @grokspice in #12488
• fix(search): resolve a chunked index's entries from the authoritative store, not the read listing (fixes #12266) by @claudespice in #12411
• fix(search): Improvements for search harness by @Jeadie in #12226
• test(cayenne): cover the selective PK join and the inline/file tier boundary by @lukekim in #12517
• fix(runtime): surface the unknown-connector suggestion, and report it once (fixes #12415) by @grokspice in #12469
• fix(ci): publish no sign-off verdict when the run was signalled (fixes #12518) by @grokspice in #12544
• refactor(layering): move the modules the accelerated table shares below runtime by @bjchambers in #12219
• chore(ci): upgrade spiceio setup action to v0.7.0 by @lukekim in #12611
• fix(runtime): keep the leading slash on an inferred local Iceberg warehouse root (fixes #12533) by @grokspice in #12540
• fix(ci): let a cancelled sign-off correct only the failure it posted (fixes #12428) by @grokspice in #12432
• fix(runtime): compare an append high-water mark inclusively on a day-granular time column (fixes #12492) by @grokspice in #12500
• fix(runtime): report a dataset connector load failure once, not once per site (fixes #12365) by @grokspice in #12526
• fix(workers): give the weighted-router test a noise margin it can survive (fixes #12537) by @grokspice in #12538
• fix(runtime): reject a Hadoop table URL too short to name a namespace (fixes #12539) by @grokspice in #12542
• fix(ci): probe a pinned Hugging Face revision at a URL the Hub has (fixes #12553) by @grokspice in #12554
• feat(testoperator): pin the load phase to a target query rate by @lukekim in #12519
• fix(cli): follow only same-origin redirects, so the API key cannot leave the origin by @grokspice in #12503
• fix(ci): use spiceio and sccache for E2E macOS aarch64 builds by @lukekim in #12590
• perf(cdc): build a drained CDC burst with one blocking-pool handoff by @sgrebnov in #12514
• feat(cloud-connect): complete the spice connect surface — install, codeless connect, remove by @peasee in #12159
• fix(scripts): distinguish merge-queue PRs in signoff mine by @Jeadie in #12383
• bug: HTTP RRF truncates each candidate leg before fusion by @Jeadie in #12386
• fix(cayenne): pair the cold manifest with the warm snapshot a scan captured by @phillipleblanc in #12577
• fix(deps): bump the Vortex pin and guard the task-cancellation regression by @phillipleblanc in #12548
• chore(deps): bump datafusion and table-providers for the DuckDB timezone and retention fixes by @phillipleblanc in #12545
• fix(cayenne): surface the real error for partitioned Cayenne writes by @Jeadie in #12535
• fix(postgres): re-snapshot in-memory Cayenne on slot resume, and match the slot's lifetime to the accelerator's by @bjchambers in #12221
• fix(cli): bound the Spice.ai login poll instead of retrying forever (fixes #12506) by @grokspice in #12523
• fix(runtime): name a UTC timestamp column's zone in a spelling DuckDB knows (fixes #12528) by @grokspice in #12534
• fix(mysql): raise net_write_timeout on the shared binlog dump session (fixes #12527) by @claudespice in #12586
• fix(google-genai): read a Gemini SSE stream as bytes, so a split character survives (fixes #12597) by @claudespice in #12601
• fix(mssql): follow an availability group's read-only routing redirect (fixes #11453) by @claudespice in #12607
• test(cayenne): measure small-file fan-out against the seeded appends, not a listed count (fixes #12602) by @claudespice in #12613
• fix(ci): keep the build cache's S3 endpoint out of the runtime under test (fixes #12624) by @claudespice in #12626
• perf(ci): verify the CLI binary instead of building it again by @bjchambers in #12486
• fix(postgres): publish a member's held CDC envelope before a bulk transaction (fixes #12311) by @claudespice in #12408
• fix(cayenne): consolidate the two partition creators, fixing the accelerator's missing directory sync (fixes #12212) by @claudespice in #12619
• fix(cayenne): run a backend-parameterized test on a stack its plan fits in (fixes #12436) by @grokspice in #12561
• fix(ci): let only the merge queue pass Attestation without a sign-off (fixes #12679) by @grokspice in #12681
• fix(ci): take the trusted sign-off helpers from the workflow's own commit (fixes #12657) by @claudespice in #12662
• fix(ci): name the E2E jobs that did not succeed (fixes #12643) by @grokspice in #12649
• fix(ci): give the openai model job a ceiling above its own setup (fixes #12644) by @grokspice in #12650
• fix(ci): recognise the Flight bind wording so a foreign ready cannot pass (fixes #12642) by @grokspice in #12648
• fix(ci): call an unreachable compiler cache infrastructure, not a failing branch (fixes #12556) by @grokspice in #12557
• fix(search): clear a replacing index instead of keeping rows the source dropped (fixes #12066) by @grokspice in #12564
• fix(ci): retry the sign-off status post, and say so when it never lands (fixes #12701) by @grokspice in #12704
• build(deps): bump aws-smithy-runtime from 1.12.0 to 1.12.1 in the aws-sdk group across 1 directory by @app/dependabot in #12358
• task: Add #[derive(TypedParams)] for LLMs. (fixes #12150) by @Jeadie in #12155
• fix(ci): let a nightly's later test suites survive an earlier one's failure (fixes #12625) by @claudespice in #12627
• fix(tests): wait for a partition to persist before snapshotting its plan (fixes #12645) by @grokspice in #12652
• fix(object-store): bound an FTP and SFTP connection attempt end to end (fixes #12647) by @grokspice in #12655
• test(iceberg): decide the Hadoop catalog test's backends by environment (fixes #12646) by @grokspice in #12665
• fix(ci): give each E2E model job its own spiced ports (fixes #12419) by @grokspice in #12685
• fix(cache): evict the Pingora cache down to max_size instead of only recording it (fixes #12688) by @grokspice in #12694
• fix(cayenne): bound protected-snapshot compaction by the compaction memory pool by @sgrebnov in #12541
• fix(ci): leave an incomplete sign-off pending, not failed (fixes #12741) by @grokspice in #12742
• fix(telemetry): resolve duration histograms below a millisecond (fixes #12693) by @grokspice in #12699
• fix(ci): bound every E2E job, so a wedged build cannot hold the queue (fixes #12717) by @grokspice in #12719
• feat(testoperator): add new search benchmark datasets by @Jeadie in #12237
• test(cayenne): name the small-file compaction tests after the path they take (fixes #12612) by @grokspice in #12740
• fix(cli): bound an inference call by silence, not by total duration (fixes #12583) by @claudespice in #12589
• fix(turso): store and read back the same set of types (fixes #12628) by @claudespice in #12633
• fix(catalogs): keep a Glue database whose tables the include patterns select (fixes #12630) by @claudespice in #12638
• fix(catalogs): apply a catalog's exclude patterns, not just its include (fixes #12636) by @claudespice in #12641
• fix(cayenne): build a carry-forward rewrite from the classified manifest rows (fixes #12708) by @claudespice in #12711
• fix(runtime): subtract an append dedup window as a multiset, not a set (fixes #12499) by @grokspice in #12513
• bug: FTS exec drops absent columns and violates its declared schema (fixes #12228) by @Jeadie in #12245
• bug: rrf() linear recency decay divides the document age by the decay window twice (fixes #12232) by @Jeadie in #12385
• chore(deps): bump datafusion rev to spiceai-54 with outer-join unparser fix by @Jeadie in #12683
• fix(ci): recognise sccache's startup-timeout wording as an unusable cache (fixes #12622) by @claudespice in #12788
• fix(spiced): open the log with the startup banner by @bjchambers in #12615
• fix(metrics): resolve S3 Vectors latency below a hundred milliseconds (fixes #12698) by @grokspice in #12702
• fix(google-genai): bound one unterminated SSE event so a stalled endpoint cannot grow spiced (fixes #12600) by @grokspice in #12690
• feat(cpu-budget): size a burstable pod from its CPU request, with an all-cores opt-out by @bjchambers in #12581
• fix(ci): refuse a sign-off the branch's Makefile cannot run (fixes #12813) by @claudespice in #12815
• test(postgres): say which readiness wait timed out and what it saw (fixes #12730) by @grokspice in #12731
• fix(mssql): bound one connection attempt, so a stalled peer cannot pin a pool slot (fixes #12606) by @grokspice in #12733
• fix(ci): ask whether the disk watcher runs before make writes to it (fixes #12734) by @grokspice in #12736
• fix: delete three source files that no crate root declares (fixes #12735) by @grokspice in #12738
• fix(telemetry): keep a startup-recorded gauge on the operator meter, so it reaches /metrics (fixes #12667) by @grokspice in #12754
• fix(cloud-connect): pin rustls where the release call presents its identity (fixes #12760) by @grokspice in #12764
• test(cloud-client): assert the redirect policy by behaviour against a live server (fixes #12509) by @grokspice in #12767
• fix(search): decline a warm vector tier the accelerator cannot refill (fixes #12102) by @grokspice in #12768
• fix(ci): build without the compiler cache when it cannot be reached (fixes #12770) by @grokspice in #12771
• fix(json): end an array element where serde did, so bare scalars read (fixes #12782) by @grokspice in #12785
• fix(cache): count an invalidation as an eviction, and export the counters before one fires (fixes #12687) by @grokspice in #12791
• fix(ci): ask whether the runner can hold the build before starting one (fixes #12798) by @grokspice in #12799
• fix(ci): size the cayenne property-test ceiling above the pool, not at it (fixes #12811) by @grokspice in #12814
• fix(ci): bound every merge-queue-required job with timeout-minutes by @grokspice in #12817
• deps: bump datafusion-table-providers to 894e279 (fixes #12585) by @bjchambers in #12663
• fix(ci): take stale build output off the runners, not just report a full one (fixes #12800) by @grokspice in #12801
• fix(cayenne): give the Turso metastore sidecars one pool over cayenne.db (fixes #12727) by @grokspice in #12804
• fix(deps): drop the unreferenced ctor dev-dependency from data_components (fixes #12664) by @grokspice in #12819
• fix(ci): decline a verdict when the sign-off run itself was signalled (fixes #12710) by @grokspice in #12724
• fix(postgres): hold a resuming shared slot's ack floor for tables with no member by @bjchambers in #12676
• refactor(layering): split the table-provider layers out of runtime by @bjchambers in #12661
• build(deps): bump the github-actions-dependencies group across 3 directories with 7 updates by @app/dependabot in #12842
• fix(spiced): make the crash handler more robust by @sgrebnov in #12834
• fix(catalogs): apply the Cayenne catalog's include and exclude patterns (fixes #12766) by @grokspice in #12833
• fix(search): decide a compound index's write fatality per half, not by OR (fixes #12826) by @grokspice in #12828
• feat(cloud): organization context for spice cloud commands by @lukekim in #12515
• perf(postgres catalog): skip metadata queries for schemas no include pattern can reach by @bjchambers in #12651
• bug: S3 vector metadata conversion silently drops filterable metadata (fixes #12240) by @Jeadie in #12387
• build(deps): bump nvidia/cuda from 13.3.0-cudnn-runtime-ubuntu24.04 to 13.3.1-cudnn-runtime-ubuntu24.04 in the docker-dependencies group by @app/dependabot in #12356
• fix(runtime): keep serving when the working directory cannot be walked (fixes #6301) by @claudespice in #12803
• fix(connectors): report an object-store timeout as a timeout, not as bad credentials (fixes #12793) by @claudespice in #12797
• test(runtime): say which components were not ready when the wait times out (refs #12396) by @grokspice in #12827
• fix(cayenne): measure tuner memory pressure as unreclaimable demand (fixes #12531) by @sgrebnov in #12623
• Include v2.1.3 and 2.1.4 in SECURITY.md by @sgrebnov in #12497
• fix(search): leave the stored rows in place when a delete cannot be applied (fixes #12822) by @grokspice in #12825
• fix(google-genai): resume the SSE scan where it stopped, not at the buffer start (fixes #12689) by @grokspice in #12713
• fix(json): report a malformed JSON array element instead of dropping the rest of the file (fixes #12755) by @claudespice in #12777
• fix(runtime): honour a declared type's precision instead of narrowing it (fixes #12756) by @claudespice in #12774
• fix(cayenne): drop the partition-value guard that key encoding made unreachable (fixes #12616) by @claudespice in #12753
• fix(runtime): pin the catalog-acceleration contract in both feature configurations (fixes #12743) by @claudespice in #12752
• fix(cli): report an answer the model stopped early, instead of printing it as whole (fixes #12596) by @grokspice in #12715
• fix(runtime): scope query jobs and active queries to the principal that submitted them by @phillipleblanc in #12841
• Compute all metrics@k for all 0 < k <= n during search quality metrics. by @Jeadie in #12521
• task: Add a truncate to huggingface and file embedding providers by @Jeadie in #12620
• fix(cache): never serve a result whose tables changed after it read them by @bjchambers in #12703
• test(cayenne): cover datalake-tier statistics, pruning, and promotion triggers by @sgrebnov in #12847
• docs(ci): warn that re-running enforce-pull-with-spice replays a stale payload (fixes #12809) by @grokspice in #12810
• fix(cayenne): bound the maintained-aggregate index by memory budget, and let a stale registry recover by @lukekim in #12573
• fix(runtime): only initialize accelerators for the datasets a reload applies by @phillipleblanc in #12872
• feat: promote BYOC and Cloud Connect changes to trunk by @phillipleblanc in #12852
• fix(flight): resolve per-request settings from the live app by @phillipleblanc in #12873
• fix(cli): keep polling when the token exchange has not seen the auth code yet by @phillipleblanc in #12871
• feat(cayenne): inline a small whole-table refresh into the metastore by @lukekim in #12367
• ci: upgrade spiceio setup action to v0.8.0 by @lukekim in #12885
• fix(runtime): measure the CDC prefetch backlog, and stop its byte counter wrapping by @lukekim in #12675
• fix(turso): keep a reloaded file-accelerated dataset queryable by @phillipleblanc in #12882
• fix(query): preserve expression ordering across partition wrappers by @Jeadie in #12854
• fix(runtime): report every start-time-only runtime.* section a reload changes by @phillipleblanc in #12874
• fix(search): handle null embeddings in vector writes and JIT search by @Jeadie in #12855
• test(embeddings): truncate over-length inputs in the MiniLM embed spicepods by @Jeadie in #12856
• fix(search): Improve search benchmarking DX and fix some vector index names by @Jeadie in #12892
• test(search): dispatch all MTEB search variants by @Jeadie in #12850
• fix(runtime): stop a superseded scheduler incarnation from claiming the heartbeat key by @phillipleblanc in #12851
• fix: Ensure OTel ingest materializes missing/null dimension columns by @peasee in #12087
• feat(cloud-connect): apply a component-only Spicepod deployment in place by @phillipleblanc in #12895
• feat(cli): typed login sessions and explicit connect organization selection by @phillipleblanc in #12912
• refactor: make each table wrapper a typed layer that answers where a walk goes by @bjchambers in #12891
• feat(runtime): put every query's trace id on its log records by @lukekim in #12610
• feat(cayenne): bound the SUM of the per-table PK keyset caches by @lukekim in #12802
• fix(cache): read and expire a Pingora entry under one hold of its shard (fixes #12832) by @grokspice in #12839
• fix: Update Search integration test snapshots by @Jeadie in #12707
• fix(runtime): invalidate localpod children's cached results on parent refresh by @krinart in #12897
• fix(cayenne): use valid snapshot IDs for deferred appends by @Jeadie in #12853
• fix(ci): New branch for each push-snap-changes (only affects integration_search.yml) by @Jeadie in #12684
• fix(search): make chunked embedding and offset columns nullable (#12778) by @Jeadie in #12783
• feat(cayenne): charge scan materialization to the query memory pool by @lukekim in #12759
• fix(cache): run the Pingora invalidation scan off the calling runtime worker (fixes #12806) by @grokspice in #12808
• fix(cache): count the removals the Pingora engine performs itself (fixes #12792) by @grokspice in #12830
• fix(cayenne): reserve append sequence for partitioned upsert appends (#12779) by @Jeadie in #12784
• test(cayenne): prove the keyset degrade releases each shard as it converts by @lukekim in #12790
• perf(postgres catalog): resolve a schema's table schemas in one query (#12106) by @bjchambers in #12886
• fix(vortex): expose segment cache metrics on scrape by @phillipleblanc in #12944
• refactor: point connector imports at the crates that actually own them by @bjchambers in #12925
• fix(postgres): rebuild the acceleration when a replication slot's history is gone by @bjchambers in #12922
• fix(cayenne)!: only an explicit cayenne_tuning: adaptive enables the closed loop by @lukekim in #12949
• fix(dev-tools): show merge-queue and conflict state as labels in scripts/signoff mine by @Jeadie in #12924
• test: cover warm-index delete paths across backends (#11964) by @Jeadie in #12635
• fix(cayenne): fractional proptest scaling + batch timestamp-partition test inserts by @Jeadie in #12796
• Helpful tool to track what users/bots have got done lately by @Jeadie in #12603
• Release notes housekeeping by @krinart in #12977
• build(deps): bump serde_with from 3.20.0 to 3.21.0 by @app/dependabot in #11969
• refactor: make runtime's re-export shims crate-visible by @bjchambers in #12969
• fix(runtime): surface OTel metric tables with duplicate columns, and count unsupported metric types as rejected by @peasee in #12923
• fix: Update Search integration test snapshots by @app/github-actions in #12979
• Make signoff.yml more searchable. by @Jeadie in #12392
• bug: Support search UDTFs in SQL function components by @Jeadie in #12919
• docs: document the stacked-PR workflow, with a tested restack helper by @bjchambers in #12951
• fix(llms): fall back to pytorch_model.bin for embeddings without safetensors by @Jeadie in #12971
• feat(monitoring): improve the Datadog dashboard for multi-replica and Kubernetes deployments by @sgrebnov in #12972
• Charge the query memory pool for concurrent Vortex split decodes by @lukekim in #12940
• docs(release-notes): write release notes in Simplified Technical English by @lukekim in #12947
• refactor: narrow the DataConnector trait off the runtime's types by @bjchambers in #12993
• fix(cayenne): compact a DDL-created partition on an interval, not only on write (fixes #12617) by @grokspice in #12757
• fix(cayenne): hand out a pooled Turso connection in autocommit (refs #12820) by @grokspice in #12821
• fix: fail the lint gate on a source file no crate root declares (fixes #12737) by @grokspice in #12744
• fix(search): stage a full refresh of the memory vector tier, so dropped rows leave it (refs #12413) by @grokspice in #12823
• fix(graphql): keep the configured schema when a page has no rows (fixes #13004) by @claudespice in #13016
• fix(runtime): register task_history before embeddings/rerankers load by @Jeadie in #12978
• test(postgres): converge the catalog on source DDL, and warn when it selects nothing by @bjchambers in #12988
• fix(cayenne): budget for catalogs in the Cayenne memory classification by @bjchambers in #13027
• fix(cayenne): hold primary keys committed while the existence index is checked out by @lukekim in #13019
• feat(search): push SQL filters down into tantivy full-text search by @Jeadie in #12812
• fix(cayenne): serve the maintained count Inexact while its delta is queued (fixes #12824) by @grokspice in #12829
• fix: Update Search integration test snapshots by @app/github-actions in #13060
• fix(testoperator benchmarks): benchmark fixes by @krinart in #13012
• fix(runtime): stop query_active_count drifting up when queries share a request (fixes #12883) by @claudespice in #13034
• chore(deps): bump model2vec-rs for fast WordPiece tokenizer by @Jeadie in #12986
• ci(cayenne-doc): render diagrams locally with mermaid-cli, and fix the diagram that has been failing the build for a month by @bjchambers in #13063
• feat(monitoring): improve results cache panels on the Datadog dashboard by @sgrebnov in #13066
• feat(cayenne)!: one process-wide Vortex segment cache instead of a cache per table by @bjchambers in #12983
• fix(connectors): keep the projected schema when a query returns no rows (fixes #13015) by @claudespice in #13078
• fix(deps): bump the datafusion pin past the merged unparser fixes (fixes #12406) by @claudespice in #13083
• fix(cache): compact a sliced result before caching it (fixes #12921) by @claudespice in #13043
• refactor(cdc): give the sidecar checkpoint stores a below-runtime interface by @bjchambers in #13045
• fix(cayenne): re-baseline the maintained row count when a promotion folds a delete tombstone (fixes #12846) by @claudespice in #12998
• fix(acceleration): stop an engine-required type rewrite reading as a stale acceleration schema (fixes #13014) by @claudespice in #13074
• fix(udfs): return NULL for a vector distance that is not defined (refs #11263) by @claudespice in #13091
• fix(cayenne): refuse a data directory that contains the metastore (fixes #13055) by @claudespice in #13101
• fix(json): report content after a JSON array instead of dropping it (fixes #12786) by @claudespice in #13097
• fix(runtime): bound the spicepod apply so an unloadable dataset cannot wedge it (fixes #12862) by @claudespice in #13095
• fix(catalogs): apply the Glue catalog's exclude patterns (fixes #12634) by @claudespice in #13103
• fix(metrics): make component metrics and tests more robust by @sgrebnov in #13094
• feat(cache)!: ship the Pingora cache engine as an Enterprise-only feature by @lukekim in #13000
• fix: Update benchmark snapshots by @app/github-actions in #13056
• fix(postgres): compare CDC positions against what the slot can stream, not what it retains by @bjchambers in #12990
• docs(cayenne): compress the changelog, and give it a rule for what earns a row by @bjchambers in #13067
• test: remove a test's container when it is done with it, including on failure by @bjchambers in #13123
• fix(flight): record one metric sample per Flight RPC (fixes #12844) by @claudespice in #13112
• feat(drasi): forward CDC changes and runtime tables to a Drasi source (Alpha) by @lukekim in #12653
• fix(bench): time the engine instead of the results cache, and validate ClickBench by @lukekim in #13150
• refactor(connector): retype the DataConnector surface to DatasetSpec by @bjchambers in #13096
• bug: SQL-tier function body does not support named arguments (=>) for search UDTFs (fixes #12898) by @Jeadie in #12920
• fix(cache): compact the sliced and wide results that compaction previously skipped by @sgrebnov in #13176
• docs(stacked-prs): correct when a restack forfeits the sign-off by @bjchambers in #13130
• fix(postgres): load a provably empty CDC acceleration through its snapshot bootstrap by @bjchambers in #13175
• fix(cayenne): keep Arrow timestamp units (Vortex supports ns) by @lukekim in #13180
• Cloud Connect: enrollment and identity foundation by @phillipleblanc in #13181
• refactor(model): give the model-provider contracts a home below llms by @bjchambers in #13201
• Fix runtime lint with zero features by @krinart in #13208
• fix(benchmarks): BigQuery + Snowflake fixes + updated snapshots by @krinart in #13136
• Revert "fix(search): decline a warm vector tier the accelerator cannot refill (fixes #12102)" (#12768) by @Jeadie in #13041
• fix(snapshots): don't bootstrap the snapshot file_create just discarded by @lukekim in #13179
• Declare the CPU entitlement to Vortex at startup by @sgrebnov in #13173
• fix(mysql): fix net_write_timeout session syntax and extend transient replication errors by @ewgenius in #13178
• fix(postgres): classify EOF and server termination errors as transient in CDC replication by @ewgenius in #13163
• feat(testoperator): support customer-supplied datasets in run search by @Jeadie in #12982
• Add PDF page splitter and FinanceBench staging job (#12858) by @Jeadie in #12984
• fix(cayenne): frame the persisted PK bloom with the probe function that filled it (fixes #13137) by @claudespice in #13153
• refactor(layering): move the connectors below the runtime by @bjchambers in #13166
• feat(cli): manage the Cloud Connect service with launchd on macOS by @phillipleblanc in #13203
• Support local rerankers from text-embeddings-inference by @Jeadie in #12929
• fix(catalog): make the PostgreSQL catalog's errors and warnings actionable by @bjchambers in #13199
• fix(cloud-connect): look again when a lock create reports the entry missing by @phillipleblanc in #13231
• Skip DynamoDB in scheduled benchmarks by @ewgenius in #13222
• Populate the file metadata cache with Vortex footers at write time by @sgrebnov in #13228
• perf(cayenne): give the PK filter a cache-line layout, and the bit count it asks for by @bjchambers in #13219
• fix(mysql): rebuild a purged-position acceleration atomically instead of emptying it (fixes #12967) by @claudespice in #13023
• fix(cloud-connect): heartbeat restart-required, attachment by app id, and this crate's integration tests in the gate by @phillipleblanc in #13234
• fix(search): join all of a dataset's vector indexes on one VectorScanTableProvider by @krinart in #13209
• Validate vector search parameters before SQL query by @lesbass in #12253
• feat(helm): support customizing Deployment strategy and StatefulSet updateStrategy by @sgrebnov in #13249
• fix(postgres): recover from a replication slot lost while streaming, and state what each slot costs by @bjchambers in #13221
• fix(cayenne): stop a partitioned acceleration reporting a schema change it never applied (fixes #12999) by @claudespice in #13057
• feat(cloud-connect): name the local spicepod a cloud-managed instance serves or ignores by @phillipleblanc in #13262
• fix(snowflake): keep NUMBER precision and scale during schema discovery by @phillipleblanc in #13272
• fix(secrets): check secret references where the components resolve them by @peasee in #13265
• docs(criteria): sign off the PostgreSQL Catalog Connector at Beta by @bjchambers in #13235
• fix(cloud-connect): resolve locked service state through the retained directory descriptor (fixes #13204) by @claudespice in #13292
• chore(deps): upgrade iceberg-rust to v0.10.0 by @krinart in #13189
• fix(cloud-connect): fix Windows build of identity state-file helpers by @sgrebnov in #13308
• fix(cayenne): normalize append de-duplication types by @ewgenius in #13200
• feat(cli): unify Spice Cloud enrollment and service workflows by @phillipleblanc in #13326
• feat(postgres): native upsert delivery for durable write-back by @Jeadie in #13323
• feat(cli): create an unattached Cloud Connect project by @phillipleblanc in #13333
• fix(cayenne): exclude protected snapshots above the delete fence from subset compaction by @krinart in #13343
• fix(cli): resolve a Cloud Connect project's data-plane region from its config by @krinart in #13370
• fix(cli): keep a granted Spice Cloud login when the identity endpoint is silent by @bjchambers in #13376
• fix(duckdb): fix concurrent-query failures after a replace_file swap with multiple DuckDB files by @Jeadie in #13383

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.0...v2.2.0

Spice v2.1.5 is now available! 🛠️

Spice v2.1.5 is a patch release focused on dependable cached data and predictable operation under load. Cached queries now reflect data changes more reliably, Cayenne workloads stay within configured memory limits, health checks remain responsive while cached results are updated, and cache dashboards provide a more complete view of activity.

What's New in v2.1.5​

Cached Queries Stay Fresh as Data Changes​

Cached results are now cleared reliably after refreshes, writes, retention changes, and updates to dependent local datasets. Expired entries are removed promptly, and entries for data that is no longer part of a Cayenne dataset are not reused by later queries.

These improvements prevent a completed data change from being followed by an older cached answer. No configuration changes are required.

More Predictable Cayenne Behavior Under Heavy Load​

Cayenne now keeps track of the memory needed to prepare query results, including when several parts of a query are prepared at once. Multiple accelerated tables also share available memory instead of each planning as though it were the only table in the deployment.

Large and concurrent workloads are therefore less likely to exhaust the available memory. When a query cannot fit within the configured limit, it fails cleanly instead of putting the entire service at risk. Operators can also limit how much work one dataset does at once when it needs a smaller memory footprint, without slowing every query.

Health Checks Remain Responsive During Cache Updates​

Refreshing or writing a dataset with many cached results no longer holds up Spice while it finds old answers that need to be cleared. Health checks and other requests can continue during this work, reducing avoidable service restarts under load.

More Trustworthy Cache Dashboards​

Cache dashboards now show total space, space in use, stored results, requests, and successful reuse whenever the dashboard is refreshed, including for new or lightly used datasets. Counts for expired results, automatic size cleanup, and cleanup after data changes are also reported consistently, so a zero value represents no activity rather than missing information.

Contributors​

• @bjchambers
• @krinart
• @lukekim
• @phillipleblanc

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.5, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.5 image:

docker pull spiceai/spiceai:2.1.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.5

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix: arrow IndexedMemTable serves stale results after DML/retention/sync (fixes #11262) by @claudespice in #11532
• fix(cache): evict the Pingora cache down to max_size instead of only recording it (fixes #12688) by @grokspice in #12694
• fix(cache): count an invalidation as an eviction, and export the counters before one fires (fixes #12687) by @grokspice in #12791
• fix(cache): never serve a result whose tables changed after it read them by @bjchambers in #12703
• Remove unnecessary allocations from results-cache and hot conversion paths by @phillipleblanc in #11895
• fix(cache): read and expire a Pingora entry under one hold of its shard (fixes #12832) by @grokspice in #12839
• fix(runtime): invalidate localpod children's cached results on parent refresh by @krinart in #12897
• fix(cache): run the Pingora invalidation scan off the calling runtime worker (fixes #12806) by @grokspice in #12808
• fix(cache): count the removals the Pingora engine performs itself (fixes #12792) by @grokspice in #12830
• feat(cayenne): charge scan materialization to the query memory pool by @lukekim in #12759
• feat(cayenne): bound the SUM of the per-table PK keyset caches by @lukekim in #12802
• Charge the query memory pool for concurrent Vortex split decodes by @lukekim in #12940
• fix(vortex): invalidate retired segment cache entries by @phillipleblanc in #12943
• fix(vortex): expose segment cache metrics on scrape by @phillipleblanc in #12944

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.4...v2.1.5

Spice v2.1.4 is now available! 🛠️

Spice v2.1.4 is a patch release that improves DuckDB acceleration and runtime stability: datasets with UTC timestamp columns, such as Iceberg tables, now load successfully when accelerated with DuckDB, retention policies run reliably, and the runtime is more resilient under heavy query load.

What's New in v2.1.4​

DuckDB Acceleration Works with Iceberg Timestamps​

A dataset with a UTC timestamp column — for example, any Iceberg timestamptz column — could previously fail to load when accelerated with DuckDB, leaving the dataset unhealthy and unqueryable. These datasets now load and become ready normally, with no configuration changes needed.

Reliable Retention Policies on DuckDB​

Retention policies now apply cleanly on DuckDB-accelerated datasets, including policies that combine a time window with an additional condition. Expired rows are evicted on every retention interval, keeping accelerated data fresh and storage bounded.

Improved Runtime Stability​

The runtime is now more robust when queries are cancelled — whether by a client disconnecting, a timeout, or a new refresh superseding an in-flight read. A rare crash in this path has been eliminated.

Contributors​

• @phillipleblanc
• @peasee

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.4, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.4 image:

docker pull spiceai/spiceai:2.1.4

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.4

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• chore(deps): bump datafusion and table-providers for the DuckDB timezone and retention fixes (2.1 backport) by @phillipleblanc in #12546
• fix(deps): bump the Vortex pin to pick up the task-cancellation fix by @phillipleblanc in #12547

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.3...v2.1.4

Spice v2.1.3 is now available! 🛠️

Spice v2.1.3 is a patch release focused on resource efficiency: explicit CPU sizing with the new runtime.cpu.cores setting, more query memory for Cayenne deployments, and improved memory and crash diagnostics. It also fixes Cayenne acceleration of Iceberg datasets with timestamptz columns and restores WHERE filters on federated LEFT/RIGHT JOIN queries.

What's New in v2.1.3​

CPU Sizing with runtime.cpu.cores​

The new runtime.cpu.cores setting controls how many cores the runtime targets. Thread pools, query partitioning, and accelerator concurrency are all derived from it.

runtime:
  cpu:
    cores: 4 # `auto` (default) detects. Accepts 4, 3.5, 3500m

Also available as --cpu-cores and SPICE_CPU_CORES (precedence: flag > environment > Spicepod).

This is most useful on large, shared nodes. A pod that sets resources.requests.cpu without a CPU limit exposes no cgroup quota, so the runtime sizes itself for every core on the node rather than its allocated share. Setting the entitlement aligns parallelism and memory footprint with the CPU the pod actually receives.

The effective value, its source, and the derived sizing are logged at startup and exported as the spiced_cpu_budget_cores gauge.

More Query Memory for Cayenne Deployments​

The Cayenne compaction memory pool is now reserved only for accelerations that can compact into it: file mode with a small-write refresh profile. Other deployments, including refresh_mode: full, keep the full memory limit available to queries — up to 6.4 GiB on a 32 GiB limit, with no configuration change.

Memory budgets are now derived from the process's own cgroup limit rather than total host memory.

Diagnostics​

Three new gauges report memory in use: query_memory_pool_used_bytes, cayenne_compaction_memory_pool_used_bytes, and process_resident_memory_bytes.

Memory pool refusals now return ResourcesExhausted and HTTP 503, distinguishing them from query errors.

Fatal native signals (SIGSEGV, SIGBUS, SIGILL, SIGFPE) report the signal, faulting address, and thread before exit, so a crash can be diagnosed from logs.

Fixed a bug where setting runtime.task_history.enabled: false also disabled every query metric — query_duration_ms, query_execution_duration_ms, query_executions, query_failures, query_returned_rows, and query_returned_bytes. These are now reported regardless of the task history setting.

Cayenne Acceleration of Iceberg timestamptz Columns​

Accelerating an Iceberg dataset with a timestamptz column using the Cayenne engine previously failed during the refresh write with an error resolving the time zone +00:00. Iceberg maps every timestamptz column to the fixed-offset Arrow time zone +00:00, which the file writer could not resolve when building column statistics. Fixed-offset time zones (±HH:MM, ±HHMM, and ±HH) are now resolved wherever time zones are handled, so these datasets accelerate correctly.

Federated Outer Join Filters​

A federated query combining a LEFT JOIN with a WHERE filter on the left table previously returned all rows instead of the filtered rows: when the query was pushed down to the data source or accelerator, the filter was folded into the JOIN ON clause, where it no longer filters the left side (RIGHT JOIN was affected symmetrically). Filters now stay on the side of the join they came from, so these queries return the correct rows.

Contributors​

• @lukekim
• @sgrebnov
• @phillipleblanc
• @bjchambers
• @Jeadie

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.3, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.3 image:

docker pull spiceai/spiceai:2.1.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.3

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• feat(runtime): size every CPU-derived pool from the CPU entitlement by @bjchambers in #12276
• fix(runtime): carve the Cayenne compaction memory pool only when a dataset can compact into it by @sgrebnov in #12326
• fix(runtime): size memory budgets from the process's own cgroup limit by @lukekim in #12263
• fix(telemetry): read the cgroup CPU quota along the whole cgroup path by @sgrebnov in #12300
• feat(runtime): expose the memory numbers that explain an OOM as gauges by @lukekim in #12195
• fix(runtime): report a memory-pool refusal as ResourcesExhausted and answer it with 503 by @sgrebnov in #12289
• fix(cayenne): the write-concurrency raise must respect the memory brake by @lukekim in #12317
• feat(spiced): report fatal signals before exit by @sgrebnov in #12334
• fix: report query metrics when task history is disabled by @sgrebnov in #12227
• chore(deps): repoint vortex at the 2.1 fixed-offset timezone fix by @phillipleblanc in #12455
• chore(deps): bump datafusion rev for outer-join unparser fix by @Jeadie in #12460

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.2...v2.1.3

Spice v2.1.2 is now available! 🛠️

Spice v2.1.2 is a patch release focused on improving the DuckDB data accelerator. It upgrades the DuckDB engine to v1.5.5 and introduces the on_full_refresh parameter, giving file-mode DuckDB accelerations compact, predictable disk usage across repeated full refreshes.

What's New in v2.1.2​

Bounded DuckDB Acceleration File Growth with on_full_refresh​

File-mode DuckDB accelerations using refresh_mode: full now reclaim disk space on every refresh, keeping the database file compact and disk usage predictable for long-running deployments. Each full refresh bulk-loads a fresh copy of the data, and the new on_full_refresh modes ensure the space held by prior copies is returned rather than accumulating in the file.

The new on_full_refresh acceleration parameter controls how disk space is reclaimed after each full refresh:

acceleration:
  engine: duckdb
  mode: file
  refresh_mode: full
  params:
    duckdb_file: /data/shared.duckdb
    on_full_refresh: replace_file # default: reuse_file

• reuse_file (default): Existing behavior — refresh into the existing database file.
• replace_file: Each full refresh streams data into a fresh staging database file, carries over every other object sharing the file (other datasets' tables, views, indexes, and Spice metadata), checkpoints it, and atomically replaces the configured file. Queries are never interrupted — in-flight queries drain against the old file while new queries see the new file — and space is fully reclaimed on every refresh.
• checkpoint_file: After each refresh, run a CHECKPOINT in place, escalating to FORCE CHECKPOINT when concurrent transactions block the plain attempt (waiting for in-flight transactions; never aborting them).

DuckDB 1.5.5​

The DuckDB engine is upgraded from v1.5.3 to v1.5.5, bringing the latest upstream stability fixes.

Contributors​

• @lukekim
• @sgrebnov
• @bjchambers
• @krinart

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.2 image:

docker pull spiceai/spiceai:2.1.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.2

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• feat(duckdb): on_full_refresh: replace_file — full refresh into a new database file, atomically replaced by @lukekim in #12135
• feat(duckdb): 'on_full_refresh: checkpoint_file' to bound acceleration file growth by @sgrebnov in #12139

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.1...v2.1.2

Spice v2.1.1 is now available! 🛠️

Spice v2.1.1 is a patch release focused on reliability and performance. It resolves a possible deadlock that affected datasets with partitioned Cayenne accelerators, caches empty SQL result sets so repeat queries are served from cache, speeds up repeated queries on multi-file datasets, and restores Bedrock embedding provider parameters.

What's New in v2.1.1​

Cayenne Partitioned Dataset Deadlock Fix​

Cayenne datasets configured with partition_by could deadlock during their initial refresh and never become ready. Non-partitioned tables and small partitioned tables were unaffected. The root cause was a deadlock between the partition routing and the global Vortex encode budget introduced in v2.1.0.

Cayenne Zero-Row Append Refresh Stability​

An idle append refresh, one where no source rows are newer than the current max(time_column), wrote no Vortex files, so the expected snapshot directory was never created. The subsequent fsync on that directory failed with ENOENT, marking the dataset unhealthy. The fix skips the snapshot sequence record and protected-snapshot publish when the write carried no rows.

Caching of Empty Result Sets​

The SQL results cache now stores empty (zero-row) result sets. Queries that legitimately return no rows (e.g. WHERE 1=0, LIMIT 0) are now served from the results cache on subsequent requests instead of being re-executed against the source, reducing planning and query latency for these patterns.

Faster Repeated Queries on Multi-File Datasets​

Object store datasets using parquet now cache Parquet footer statistics across queries. This reduces the frequency of Parquet footer parsing during planning, subsequently heavily reducing planning latency for certain query patterns (e.g. COUNT(*)).

Embedding Parameter Regression Fixes​

v2.1.0 introduced explicit definitions across embedding component parameters (i.e. .embeddings[].params). This introduced regressions for AWS Bedrock parameters truncation and truncation_mode that caused panics.

Contributors​

• @bjchambers
• @Jeadie
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.1 image:

docker pull spiceai/spiceai:2.1.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(cache): cache empty (zero-row) SQL result sets by @bjchambers in #11699
• fix(cayenne): zero-row append refresh fails with "No such file or directory" by @sgrebnov in #11710
• feat(file): cache ListingTable file statistics to avoid per-query footer re-parse by @phillipleblanc in #11793
• fix(embeddings): restore params broken by #10853 by @Jeadie in #11788
• fix(cayenne): partitioned datasets deadlock against the global encode budget and never become ready by @Jeadie in #11825

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.0...v2.1.1

Spice v2.1.0 is now available! 🎉

Spice v2.1.0 is the next minor release of Spice, headlined by high-throughput Cayenne CDC, scaling and resilience improvements to PostgreSQL logical replication, expanded distributed query with Iceberg catalog scans and broadcast joins, and the upgrade to DataFusion v54 (including v53), Arrow v58.3, and Vortex v0.74. The release also adds experimental adaptive self-tuning for the Cayenne accelerator, distributed GLM inference, and a range of security, search, and connector improvements.

Highlights in v2.1.0 include:

• High-Throughput Cayenne CDC — in-memory CDC tier, dedicated compaction runtime, and write-path optimizations that cut replication lag on high-volume CDC workloads
• PostgreSQL Replication at Scale — multiple changes-mode datasets share a single replication slot, unchanged-TOAST recovery, and resilient reconnects across rolling deploys
• Distributed Query — distributed Ballista scans of Iceberg catalog tables, broadcast joins for small dimension tables, and shared scheduler job state with failover
• DataFusion v54 — upgrade to DataFusion v54 (folding in v53), Arrow v58.3, and Vortex v0.74, bringing faster joins, scans, and planning
• Adaptive Self-Tuning (Experimental) — opt-in closed-loop tuning and maintained aggregates that adapt Cayenne to hardware, schema, and live workload

What's New in v2.1.0​

High-Throughput Cayenne CDC​

A major focus of v2.1 is Spice Cayenne write-path throughput for change-data-capture (HTAP) workloads:

• In-Memory CDC Tier: A new in-memory CDC tier and follow-ups cut replication lag on hot upsert tables, with bounded mem-tier checkpointing and O(1) per-scan deletion views, plus a two-phase off-fence checkpoint on the ingest path.
• Dedicated Compaction Runtime: A dedicated compaction runtime with CDC pipelining and protected snapshots isolates compaction from query and ingest paths, with parallelized deletion-vector writes, per-batch directory-barrier coalescing, and size-aware parallel encode for protected-snapshot compaction.
• Incremental Protected Snapshot Compaction: Incremental compaction of protected snapshots (used in Cayenne's merge-on-read deletion index) reduces disk usage and improves query performance.
• Smaller WAL & Metadata-Only Publish: cayenne_insert_record table IDs are stored as 16-byte raw-UUID BLOBs, cutting CDC WAL volume ~34%; upsert commits publish metadata-only, dropping per-key insert records; transient staged CDC deltas are light-encoded.
• Delta-Write Encoding Levels: A new cayenne_delta_encoding setting (default auto) selects delta-write encoding, and cayenne_compression_strategy: zstd is now fully wired.
• In-Memory CDC Sharding: PK-hash intra-apply sharding parallelizes in-memory CDC apply.
• Scan Safety Under Write: In-flight scans are ref-counted so snapshot GC can't delete Vortex files mid-read; in-RAM scan parallelism, query admission control, and sound scan output ordering improve read behavior under sustained CDC.

Delta-write encoding effort and Vortex compression are tunable per accelerator. cayenne_delta_encoding: auto (the default) size-gates fresh CDC/append writes — small deltas use a light scheme and are re-encoded during compaction — or pin an explicit level 0..10 (7 is the full default cascade); cayenne_compression_strategy selects the Vortex compression:

acceleration:
  engine: cayenne
  refresh_mode: changes
  params:
    cayenne_delta_encoding: auto # 'auto' (default), or pin a level 0..10 (7 = full cascade)
    cayenne_compression_strategy: zstd # 'btrblocks' (default) or 'zstd'

Change Data Capture & HTAP​

PostgreSQL logical replication (CDC, refresh_mode: changes, introduced in v2.0) gets significant scaling and resilience work in v2.1:

• Shared Replication Slot: Multiple refresh_mode: changes PostgreSQL datasets on the same connection can name the same pg_replication_slot to share a single replication slot, walsender decoder, and publication, with decoded changes multiplexed by (schema, table) to each dataset. This collapses the slot count from one-per-dataset to one — staying well under Postgres's default max_replication_slots = 10.

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # shared slot name
    acceleration:
      refresh_mode: changes
  - from: postgres:public.customers
    name: customers
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # same name -> one slot, walsender & publication
    acceleration:
      refresh_mode: changes

• Unchanged-TOAST Recovery: Under REPLICA IDENTITY FULL, when an UPDATE leaves a large TOASTed column unchanged, pgoutput sends an "unchanged" marker; Spice now fills that value from the old tuple — its old value is its current value — so updates no longer error or drop columns. Without an old tuple, the error persists with a hint to enable REPLICA IDENTITY FULL.
• Transient Walsender Contention: Slot-contention errors during rolling deploys — SQLSTATE 55006 ("replication slot is active for PID") and 53300 ("requested standby connections exceeds max_wal_senders") — are now classified as transient and retried with backoff instead of fatally terminating the stream. Replication connections are also released at shutdown start (not process exit), freeing walsender seats for replacement instances.
• Strict CDC Param Validation: PostgreSQL CDC parameters are strictly validated rather than silently defaulted.
• Debezium Schema Evolution: Fixes for Debezium schema-evolution support, including tombstone-message handling and sign-extension of minimal-width base64 decimals.

Distributed Query​

Distributed Query gains:

• Distributed Iceberg Catalog Scans: Ballista distributes scans of Iceberg catalog tables across executors.
• Broadcast Joins: Small dimension tables are broadcast to executors for distributed joins.
• Shared Scheduler Job State with Failover: Ballista job state is shared so the scheduler can fail over without losing in-flight work.

Performance & Query Engine​

Apache DataFusion is upgraded to v54, folding in v53, alongside Arrow v58.3 and Vortex v0.74 (with a pin bump adding intra-file decode split and a per-execution kernel cache). Two DataFusion releases land in this upgrade:

• DataFusion v54 (release notes): adds LATERAL joins, SQL lambda functions (x -> expr with array_transform/array_filter/array_any_match), spilling nested-loop joins, and a faster arrow-avro reader. Performance work includes morsel-driven Parquet scans (up to ~2x faster for skewed scans), 20-50x faster sort-merge semi/anti/mark joins, redundant-sort-key pruning, NDV-based cardinality estimation, and inner_product/cosine_distance functions.
• DataFusion v53 (release notes): adds LIMIT-aware Parquet row-group pruning, broader filter pushdown through joins and UNION, nested-field pushdown (get_field into the scan), faster query planning (some plans dropping from ~4-5ms to ~100us), and 42 faster built-in functions.

Federation deny-list enforcement and catalog DDL are restored after the DataFusion upgrades, and a cost-based left-deep join reordering rule is added for Cayenne acceleration.

AI & LLM​

• Native GLM Support with Distributed Inference: Native GLM model support with surfaced reasoning_content, including tensor-parallel GLM inference. Load a GLM model with model_type: glm4 (glm4moe and glm4moelite are also supported):

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4

For large models, GLM inference can be distributed across nodes (tensor parallelism) via the mistral.rs pure-TCP ring all-reduce backend — no NCCL/system dependency. This is a Spice.ai Enterprise feature requiring the distributed build. Run the same model on each node, changing only node_rank:

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4
      distributed_backend: ring
      nodes: 10.0.4.21,10.0.4.22 # ordered host/IP per rank; the ring backend currently requires exactly 2
      node_rank: 0 # rank of THIS node in [0, world_size); rank 0 serves the API. Set node_rank: 1 on 10.0.4.22

• NSQL Context Endpoint: A new GET /v1/nsql/context endpoint returns the SQL dialect, dataset schemas (with optional sample rows), and registered functions that Spice injects into natural-language-to-SQL (POST /v1/nsql) requests — useful for inspecting or caching exactly what the model sees:

# Inspect the context injected into /v1/nsql requests (examples_limit default 3, max 100)
curl "http://localhost:8090/v1/nsql/context?include_examples=true&examples_limit=3"

Returns the dialect, per-dataset schema (keys, indexes, searchable columns), the registered function inventory, and sample rows (abbreviated):

{
  "context": "# Spice.ai NSQL Context",
  "instructions": [
    "Write SQL for the Spice runtime, which uses Apache DataFusion with the SQL parser configured for the PostgreSQL dialect.",
    "Use table and column descriptions, primary keys, foreign keys, unique constraints, and indexes when choosing joins and filters."
  ],
  "sql": {
    "engine": "Apache DataFusion",
    "version": "54.0.0",
    "dialect": "PostgreSQL",
    "parser": "DataFusion SQL parser configured with PostgreSQL dialect"
  },
  "datasets": [
    {
      "name": "sales.orders",
      "table": "orders",
      "description": "Customer orders",
      "columns": [
        { "name": "order_id", "data_type": "Int64", "nullable": false, "primary_key": true, "indexed": true },
        { "name": "customer_id", "data_type": "Utf8", "nullable": false, "vector_search": true, "full_text_search": true }
      ],
      "primary_key": ["order_id"],
      "foreign_keys": [
        { "columns": ["customer_id"], "foreign_table": "spice.sales.customers", "foreign_columns": ["id"] }
      ]
    }
  ],
  "functions": {
    "summary": "Spice SQL runs on Apache DataFusion ... Run SELECT * FROM list_udfs() to inspect the full registered function inventory",
    "search": [
      { "name": "vector_search", "syntax": "vector_search(dataset, 'query text'[, column])" },
      { "name": "text_search", "syntax": "text_search(dataset, 'query text'[, column])" }
    ]
  },
  "samples": [
    { "title": "Example rows for `sales.orders`", "content": "| order_id | customer_id |\n| --- | --- |\n| 42 | CUST-1 |" }
  ]
}

Search & Vectors​

• S3 Vectors Pagination: QueryVectors paginates for top-K up to 10,000.
• Elasticsearch kNN Candidate Pool: The default kNN candidate pool is raised from 10 to 1000 for better recall.

SQL & Query Engine​

• FlightSQL Substrait Plans: CommandStatementSubstraitPlan support.
• Large Result Streaming: Flight streaming is optimized for large result sets.
• Write Authorization: The SQL tool allows writes for ReadWrite API keys.
• Schema Evolution Policies: on_schema_change supports widening-only evolution and a drop_and_recreate policy.

Security & Connectors​

• Kafka mTLS: Mutual TLS configuration is surfaced in the Kafka data connector.
• Secret Resolution at Startup: Secret references are checked and reported at startup.
• DuckDB HNSW: Upgrade to DuckDB v1.5.3 with the statically linked VSS (HNSW) vector extension.

Adaptive Self-Tuning (Experimental)​

The Spice Cayenne accelerator gains experimental opt-in self-tuning. cayenne_tuning: auto derives configuration from the detected hardware and inferred schema, while adaptive additionally runs a per-table closed-feedback controller that adapts flush caps, the in-memory CDC tier, compaction cadence, and write concurrency toward operator SLOs (replication lag, freshness, query latency, queries-per-hour). Cayenne can also maintain aggregates incrementally — with predicate-aware delta serving and incremental retraction — and fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics. These features are experimental and disabled by default.

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      refresh_mode: changes
      params:
        cayenne_tuning: adaptive # 'auto' (static, env- + schema-derived) or 'adaptive' (closed-loop)

Observability​

• Per-Dataset Query Attribution: The query_executions metric gains a datasets dimension.
• HTAP Diagnostics: Improved HTAP replication diagnostics on non-convergence.
• Cayenne Write Observability: Write-phase observability for the in-memory CDC tier.

Notable Bug Fixes​

• Cayenne Utf8View: The Utf8View read schema avoids a hash-join offset overflow.
• Cayenne metastore: cayenne_metastore: turso is honored for partitioned tables and the dataset checkpoint.
• Dual-write detection: Dual-write accelerated tables are detected behind the metadata-enrichment wrapper.
• digest_many collisions: Values are length-prefixed so column boundaries can't collide.
• Turso WAL checkpoint: WAL checkpoints route through the native Turso connection.
• TLS status probe: The status check probes the metrics endpoint over HTTPS when TLS is enabled.
• Search snippet offsets: Character chunk offsets persist so search snippets aren't shifted or garbled.
• Async query chunk offsets: /v1/queries chunk row_offset uses the cumulative offset rather than chunk_index * chunk_size.

Dependency Updates​

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @melks
• @mvanhorn
• @Oxygen56
• @peasee
• @phillipleblanc
• @sgrebnov
• @v1gnesh

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.1.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.0 image:

docker pull spiceai/spiceai:2.1.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Make DuckDB schema cast logic more robust by @sgrebnov in #10991
• perf(cayenne): reduce allocation overheads in hot paths by @lukekim in #10950
• improve error message for 'params.spiceai_region' by @Jeadie in #10954
• fix(acceleration): rename WriteMode variants to fix #10960 by @phillipleblanc in #10974
• add Scylla, bigquery and turso to throughput benchmarking by @Jeadie in #11006
• deps(ballista): pull in shuffle-on-object-store correctness fixes (datafusion-ballista PRs #42 + #43) by @phillipleblanc in #10919
• fix(kafka): seek to sidecar offsets via post_rebalance callback on restart by @ewgenius in #11007
• Propagate source comments into schema metadata by @lukekim in #10944
• feat(chbench-driver): better alignment to BenchBase by @sgrebnov in #11012
• fix: handle EXISTS/NOT EXISTS subqueries in federation analyzer by @sgrebnov in #10996
• Enable filter pushdown in spicepod defined UDTFs by @Jeadie in #11004
• Refactor spice dataset configuration command by @Jeadie in #10999
• fix: ensure HashJoinExec partition counts match when join input is statically empty by @phillipleblanc in #11025
• feat(chbench): Improve OLTP throughput and reduce PostgreSQL CDC overhead by @sgrebnov in #11018
• feat(cluster): add distributed query observability metrics by @phillipleblanc in #10990
• fix: delegate truncate in PolyTableProvider to inner write provider by @claudespice in #11036
• Remove default runtime features - enable explicitly in spiced by @phillipleblanc in #11037
• fix: preserve field and schema metadata in Vortex physical schema calculation by @claudespice in #11013
• Expose metadata descriptions via PostgreSQL UDFs by @lukekim in #11032
• fix: route Turso WAL checkpoint through native turso connection (fixes #10657) by @claudespice in #11048
• fix: add missing truncate delegation and guards for wrapper TableProviders by @claudespice in #11014
• Update DataConnector statuses by @krinart in #11052
• remove redundant readonly checks by @Jeadie in #10975
• Fix Unity Catalog connector compatibility with OSS Unity Catalog by @ewgenius in #11026
• refactor(cdc): reduce CDC sub-batch splits for interleaved upsert/delete workloads by @sgrebnov in #11051
• feat(cayenne): allow inline writes with pending deletions (deletes/upserts) by @sgrebnov in #11031
• fix(sql-tool): defer read-only gate to caller's API key role by @phillipleblanc in #11040
• fix: map Gemini Recitation finish reason to ContentFilter by @claudespice in #11046
• feat(cayenne): fast-path CDC deletes by extracting PK values from filters by @sgrebnov in #11049
• fix(cluster): gate scheduler readiness on executor partition loads by @phillipleblanc in #10992
• fix(snowflake): enforce function deny-list in federation pushdown by @claudespice in #11057
• perf(cdc): Last-write-wins dedup in group_into_sub_batches to reduce sub-batch splits by @sgrebnov in #11059
• [Bug] Timing between reconnect and AllocateInitialPartitions leaves connection without flight_sql_client by @Jeadie in #10805
• Define 'trait QueryEngine' to refactor runtime crate by @Jeadie in #11028
• fix(snowflake): apply Spice function deny-list in extracted connector crate by @claudespice in #11071
• perf(cayenne): keep CDC upsert PK keysets resident to avoid per-batch full-table rebuilds by @lukekim in #11074
• fix(postgres-replication): emit recovery log + reduce reconnect-warn volume by @claudespice in #11084
• fix metadata on search indexing by @Jeadie in #11080
• perf(cayenne): scale CDC inline flush caps with memory + storage class by @lukekim in #11087
• feat(cayenne): merge-on-read position deletes for PK upsert tables + memory-pool accounting by @lukekim in #11085
• Support tuple-IN composite PK extraction in Cayenne delete fast-path by @sgrebnov in #11093
• feat(cluster): report per-executor table statistics so distributed JoinSelection can size joins by @phillipleblanc in #11089
• feat(cluster): NDV-aware executor stats so CDC q18 join swap fires by @phillipleblanc in #11098
• Improve HTAP replication diagnostics on non-convergence by @sgrebnov in #11100
• Normalize DataType::Null to Int32 in acceleration schema for duckdb by @krinart in #11062
• Fix debezium schema evolution support by @ewgenius in #11095
• feat(cayenne): incremental write-path executor statistics for distributed join sizing by @phillipleblanc in #11104
• fix(cache): periodic moka maintenance to drain invalidation predicates (#11077) by @phillipleblanc in #11106
• fix: validate Snowflake account identifiers and auth config by @Jeadie in #11024
• fix: trace external mcp server tool calls by @ewgenius in #11058
• Remove unwrap from test code; drop clippy::unwrap_used suppressions by @phillipleblanc in #11108
• Upgrade to DuckDB 1.5.3 + statically link the VSS (HNSW) extension by @sgrebnov in #11107
• Fix fetched_at for HTTP connector by @Jeadie in #11116
• fix(search): propagate LIMIT to base TableScan in VectorScanTableProvider (fixes #8368) by @claudespice in #11124
• feat(runtime): add spicebench feature to register the Cayenne catalog connector by @phillipleblanc in #11122
• fix(cayenne): tombstone inline-checkpointed rows on upsert to prevent duplicate PKs by @sgrebnov in #11129
• Remove possibility of a deadlock in RuntimeStatus by @krinart in #11114
• Fix Windows build: vendored-vss duckdb-rs + adapt to table-providers mongodb API by @phillipleblanc in #11140
• localpod: synchronize child refreshes when parent uses in-memory (arrow) accelerator by @phillipleblanc in #11139
• Add datasets dimension to the query_executions metric by @phillipleblanc in #11138
• fix(spiceai): keep correlated subqueries out of JOIN ON for Spice Cloud federation by @phillipleblanc in #11143
• fix(duckdb): normalize timestamp columns to microsecond precision (fixes #10627) by @claudespice in #11145
• feat(cayenne): sharded parallel Vortex encode with key/time clustering by @lukekim in #11144
• fix(cluster): prevent DoPut write pipeline self-deadlock under ingest backpressure by @phillipleblanc in #11160
• feat(chbench): configurable HTAP concurrency, DuckDB query overrides, and OLTP rate control by @sgrebnov in #11162
• fix(http): preserve non-JSON response rows instead of crashing nested decomposition (fixes #11155) by @claudespice in #11161
• Use declared schema in DynamoDB/MongoDB/Debezium by @krinart in #11066
• fix(cluster): prevent partitioned datasets from staying Refreshing by @phillipleblanc in #11157
• fix(runtime): don't list postgres as a valid accelerator engine when postgres-accel is disabled by @sgrebnov in #11169
• fix(spark): recover stale or broken Spark Connect sessions on failure by @lukekim in #11171
• fix(secrets): don't abort secret lookup precedence walk on a failing store by @phillipleblanc in #11175
• feat(cayenne): bound aggregate write concurrency, conservative defaults, and write/read observability by @lukekim in #11170
• feat(unity_catalog): support Unity Catalog credential vending for Delta Lake tables by @phillipleblanc in #11180
• fix(secrets): keep failed secret stores registered so lookups report the init root cause by @phillipleblanc in #11181
• fix(debezium): sign-extend minimal-width base64 decimals instead of zero-padding by @claudespice in #11184
• fix(deps): update hickory-resolver to 0.26 (evicts hickory-proto 0.25.x) by @phillipleblanc in #11183
• refactor(secrets): derive secret store metadata from a single registry table by @phillipleblanc in #11188
• perf(cayenne): cut CDC replication lag on hot upsert tables by @lukekim in #11191
• feat(cayenne): async inline-fallback (per-tombstone published flag) + 64c/256GB tuning by @lukekim in #11194
• Add HTTP connector mTLS support by @lukekim in #11127
• feat(snowflake): push AT TIME ZONE as CONVERT_TIMEZONE and pin session to UTC by @lukekim in #11190
• feat(cdc): make cdc_max_coalesce_age_ms a real apply-loop linger by @sgrebnov in #11196
• feat(cayenne): delta-write encoding levels (cayenne_delta_encoding, default auto) + make compression_strategy=zstd real by @lukekim in #11199
• Add NSQL context endpoint by @lukekim in #11075
• fix(federation): respect the Spice function deny-list across all SQL connectors; dialect-aware DuckDB pushdown by @claudespice in #11186
• fix: surface unknown/applied cayenne_* runtime.params at startup (fixes #10970) by @claudespice in #11133
• perf(cayenne): plain-fsync ordering tier on the staged-commit hot path by @lukekim in #11198
• fix(kafka): decode JSON payloads to Arrow directly — fixes lossy Decimal128 + removes double-parse (#11192) by @claudespice in #11207
• feat(cayenne): self-tuning accelerator — hardware + schema + closed-loop adaptive (auto/adaptive modes) by @lukekim in #11213
• perf(cayenne): CDC throughput — SF-100 @10K txn/s toward <5s lag + 5K QPH by @lukekim in #11206
• fix(cluster): distribute accelerated tables wrapped by metadata/index providers by @phillipleblanc in #11226
• Improve Cayenne adaptive tuning and schema safety by @lukekim in #11237
• feat: Add cayenne_file_pruning param by @peasee in #11239
• Debezium connector - handle tombstone messages in kafka topic, with schema evolution enabled by @ewgenius in #11197
• feat(cayenne): broadcast small-dimension joins to executors by @phillipleblanc in #11245
• fix: scope request context across the managed query runtime by @phillipleblanc in #11253
• fix: Strip inference columns from table schema on query by @peasee in #11251
• fix(flightsql): don't drop un-pushed FilterExec predicates in distributed pushdown rules by @claudespice in #11256
• feat(postgres): share one replication slot across multiple changes-mode datasets by @phillipleblanc in #11255
• Upgrade to DataFusion v53.1, Arrow v58.3, Vortex v0.74, and dependencies by @lukekim in #11118
• feat(connectors): support file_format: vortex everywhere parquet is supported by @lukekim in #11282
• perf(cayenne): metadata-only publish — drop per-key insert records on upsert commit by @lukekim in #11260
• fix(kafka): harden fetch_latest_message for multi-partition topics by @ewgenius in #11285
• perf(cayenne): bound mem-tier checkpoint churn + O(1) per-scan deletion view by @lukekim in #11249
• fix(udfs): length-prefix digest_many values so column boundaries can't collide (fixes #11272) by @claudespice in #11288
• fix: restore federation deny-list enforcement regressed by the DataFusion 53 upgrade by @claudespice in #11294
• fix(postgres): recover unchanged-TOAST columns from the old tuple; classify walsender contention as transient by @phillipleblanc in #11293
• feat: deepen extended schema inference and wire it into cayenne compaction sharding/sorting by @lukekim in #11284
• perf(cayenne): in-memory CDC tier follow-ups + write-phase observability by @lukekim in #11278
• Support per-dataset CDC tunable overrides by @sgrebnov in #11295
• feat(cayenne): harden adaptive auto-tuner (controller hygiene, mem-tier actuator, delete/burst signals, single opt-in) by @lukekim in #11302
• fix(cayenne): scan inlined-view capture starvation under sustained CDC (analytical QPH) by @lukekim in #11299
• fix(vortex): don't row-evaluate hash-join dynamic filters in the scan by @sgrebnov in #11307
• fix(deps): bump rust-postgres crates (RUSTSEC-2026-0178/0179) by @lukekim in #11313
• fix: strict validation of Postgres CDC params instead of silent defaults (fixes #11274) by @claudespice in #11304
• fix(duckdb): always quote on-refresh sort columns so reserved-word names don't break refresh by @claudespice in #11305
• feat(flightsql): fall back to original connection when endpoint location is unreachable by @melks in #11287
• perf(cayenne): light-encode transient staged CDC deltas by @lukekim in #11311
• feat(acceleration): widening-only schema evolution via on_schema_change by @lukekim in #11261
• deps(vortex): bump pin to spiceai-53 HEAD — intra-file decode split + per-execution kernel cache by @lukekim in #11314
• feat(github): enhance GitHub component validation and error handling by @lukekim in #11259
• feat(cayenne): goal-driven adaptive tuning toward operator SLOs (lag, freshness, query latency, QPH) by @lukekim in #11310
• fix(cayenne): broadcast-join rewrite must bail on ambiguous columns, NULL-equal joins, and residual filters by @claudespice in #11252
• Add Cayenne maintained aggregates by @lukekim in #11235
• perf(cayenne): single-hash composite deletion filter via KeyDeletionIndex::get_batch by @phillipleblanc in #11325
• fix(Vortex): decline only the InList membership conjunct of hash-join dynamic filters by @sgrebnov in #11335
• fix: scope SQL UDF arg inlining to args-table columns (fixes #11273) by @claudespice in #11337
• fix(refresh): restore S3 ETag/Version refresh-skip behind provider wrappers by @phillipleblanc in #11339
• fix(cayenne): ref-count in-flight scans so GC can't delete Vortex files mid-read by @phillipleblanc in #11321
• fix(runtime): retry object-store dataset load when source files are not yet available by @phillipleblanc in #11342
• feat(s3): default to path-style for dotted bucket names on standard AWS by @phillipleblanc in #11347
• fix(runtime): resolve accelerated table through metadata-enrichment wrapper by @phillipleblanc in #11345
• fix: detect dual-write accelerated tables behind the metadata-enrichment wrapper by @claudespice in #11351
• feat(cayenne): incremental seq-prefix bake — shrink the merge-on-read deletion index by @lukekim in #11326
• fix(adbc): prevent Spice-specific UDFs from being pushed down to ADBC sources by @krinart in #11297
• fix: Query Redshift schema details from svv_redshift tables by @peasee in #11362
• perf(cayenne): tune Turso connection PRAGMAs + jitter metastore retries by @lukekim in #11359
• Upgrade to DataFusion 54 by @sgrebnov in #11360
• feat(runtime): dedicated CDC-apply tokio runtime + per-runtime tokio metrics by @lukekim in #11370
• fix(cayenne): spill oversized hash joins via sort-merge to avoid OOM by @lukekim in #11371
• fix(cayenne): honor cayenne_metastore: turso for partitioned tables and the dataset checkpoint by @phillipleblanc in #11365
• perf(cayenne): in-RAM scan parallelism, query admission control, skip no-op deletion encode, sound scan output_ordering by @lukekim in #11332
• fix(catalog): restore DDL after DataFusion 54 broke transparent catalog-provider downcasts by @phillipleblanc in #11375
• feat(flightsql): infer schema via SELECT * LIMIT 1 when GetTables is unimplemented by @melks in #11286
• fix(cayenne): Utf8View read schema avoids hash-join offset overflow by @lukekim in #11379
• feat(cluster): support distributed (Ballista) scans of Iceberg tables by @phillipleblanc in #11378
• feat(optimizer): cost-based left-deep join reordering for Cayenne acceleration by @sgrebnov in #11377
• cli - fix service-account auth in spice cloud * commands by @ewgenius in #11316
• fix: Support external Redshift table schema inference and Hive external type parsing by @peasee in #11399
• feat(llms): native GLM support — opt-in flash-attn + surface reasoning_content by @lukekim in #11400
• feat(s3_vectors): paginate QueryVectors for topK up to 10,000 by @bjchambers in #11405
• fix: surface .env parse errors with line numbers instead of silently skipping by @Oxygen56 in #11306
• fix(status): probe metrics endpoint over https when TLS is enabled by @phillipleblanc in #11393
• fix(mcp): record task_history spans for tool calls proxied through /v1/mcp by @phillipleblanc in #11397
• Properly handle date_trunc in BigQueryDialect by @krinart in #11416
• fix(snowflake): honor column scale in Int64 timestamp arm and cast TIME by @claudespice in #11418
• fix: /v1/queries chunk row_offset uses cumulative offset, not chunk_index * chunk_size (fixes #11271) by @claudespice in #11398
• feat(cayenne): incremental retraction for maintained aggregates + anchor bench by @lukekim in #11389
• feat(cluster): support distributed (Ballista) scans of Iceberg catalog tables by @phillipleblanc in #11419
• Simplify chat/responses models by @Jeadie in #10997
• feat(llms): distributed tensor-parallel GLM inference via mistral.rs ring backend by @lukekim in #11406
• Optimize Flight streaming for large result sets by @lukekim in #11420
• fix(deps): evict rustls 0.21 / rustls-webpki 0.101.7 (GHSA-82j2-j2ch-gfr8) by @phillipleblanc in #11428
• feat(cayenne): in-memory CDC intra-apply sharding (PK-hash shards) by @lukekim in #11421
• fix(cayenne): shard CDC upserts with pending deletions so the N>1 slot-ack advances by @lukekim in #11445
• fix(cluster): keep built-in avg over Spark avg (distributed aggregate state schema mismatch) by @phillipleblanc in #11434
• feat(views): support params.file_format for embedding chunking by @Jeadie in #11424
• fix: offload blocking sync calls off the primary async runtime by @phillipleblanc in #11435
• fix(udfs): rebind dot_product alias to Spice's inner_product on DataFusion 54 by @lukekim in #11443
• feat(secrets): add full-fidelity reference iteration and a resolution-status API by @phillipleblanc in #11195
• fix: Deny unsupported array functions for Postgres pushdown by @peasee in #11450
• fix(cli): spice query honors --http-endpoint instead of failing on a Flight connect by @phillipleblanc in #11452
• fix(cayenne): coordinate query-pool + in-memory CDC tier budgets to prevent adaptive OOM by @lukekim in #11449
• Surface mTLS config in Kafka data connector by @v1gnesh in #11372
• feat(secrets): check and report secret references at startup by @phillipleblanc in #11457
• Default cayenne_force_view_types to false by @sgrebnov in #11459
• fix: resolve table-reference qualification in results-cache invalidation (fixes #11266) by @claudespice in #11460
• feat(cayenne): metadata aggregate pushdown — fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics by @bjchambers in #11414
• fix(search): restore numeric trunc and fix SortPreservingMergeExec planning error (DF54) by @Jeadie in #11415
• fix(cluster): route Ballista shuffle/temp to the data PVC by @phillipleblanc in #11454
• fix(search): default Elasticsearch kNN candidate pool to 1000 instead of 10 (fixes #11264) by @claudespice in #11467
• feat(acceleration): add on_schema_change drop_and_recreate policy by @lukekim in #11462
• feat(cayenne): predicate-aware maintained aggregates serve filtered analytical queries from the CDC delta by @lukekim in #11458
• feat(cluster): shared Ballista job state with scheduler failover by @phillipleblanc in #11436
• feat(cayenne): extend HLL NDV sketching to string and date columns by @bjchambers in #11468
• fix(search): persist character chunk offsets so search snippets aren't shifted/garbled (fixes #11269) by @claudespice in #11479
• feat(cayenne): storage-aware adaptive CDC tuning — calibration probe, IMDS, I/O-cliff fast path, infeasible-SLO feedback by @lukekim in #11463
• fix(cayenne): LIMIT N under-delivers on key-deletion tables by @lukekim in #11490
• fix(cayenne): live/tier-accurate join build-side stats (merge-on-read deletes + never-shrink NDV) by @lukekim in #11496
• perf(cayenne): kernel-space I/O hygiene — compaction fadvise + staged-commit barrier reduction by @lukekim in #11495
• feat(cayenne): feed maintained-aggregate IVM from the staged-disk CDC path by @lukekim in #11491
• feat(cayenne): global adaptive-tuning SLOs with per-dataset overrides; QPH global-only by @lukekim in #11497
• Update search snapshots by @sgrebnov in #11473
• fix: Support reading column types longer than 128 chars in Redshift by @peasee in #11500
• fix(cluster): distributed (Ballista) query-execution config + scheduled SF10 bench by @phillipleblanc in #11478
• fix(http): retry transient response-body read failures; de-flake backoff test by @claudespice in #11482
• Re-land orphaned deletion-vector cleanup during retention deletes by @lukekim in #11501
• fix(cayenne): re-upsert over a pending delete tombstone records an insert-record (overwrite resurrection) by @bjchambers in #11469
• fix: Flight DoPut silently dropped client batches on early sink completion by @claudespice in #11507
• Upgrade OpenTelemetry to 0.32 and reqwest to 0.13 by @phillipleblanc in #11506
• fix(queries): run async /v1/queries jobs under the submitting request context by @phillipleblanc in #11505
• fix(cayenne): restore append-only current-snapshot compaction by @Jeadie in #11439
• perf(cayenne): orphaned deletion-vector cleanup off the write path, behind a knob by @bjchambers in #11517
• fix(datafusion): accurate projected scan byte size so hash joins build the smaller side by @sgrebnov in #11503
• fix(cayenne): user-visible DELETE WHERE pk IN (...) reports the real row count by @lukekim in #11514
• fix(cayenne): seed persisted num_rows for hash-join sizing by @sgrebnov in #11515
• fix(cayenne): correctness & memory_limit fixes from perf audit (2 P0, 3 P1) by @lukekim in #11516
• feat(cayenne): wire orphaned-DV cleanup knob to spicepod params + doc sync by @bjchambers in #11523
• Fix s3 vectors API by @krinart in #11536
• fix: Placeholder table initialization lock swap by @peasee in #11540
• chore(cluster): bump ballista pin for the null-aware anti-join fix by @phillipleblanc in #11544
• fix(runtime-tools): fix memory table identifier validation rejecting valid names by @Jeadie in #11546
• Bump datafusion to include spiceai/datafusion#181 by @Jeadie in #11563
• Update deny.toml by @krinart in #11571
• Bump datafusion (spiceai/datafusion#182) and datafusion-table-providers (#27): fix q16 CollectLeft planning error and SQLite q6 wrong revenue by @Jeadie in #11598

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0...v2.1.0

Spice v2.0.1 is now available! 🛠️

Spice v2.0.1 is a patch release focused on reliability and performance. It speeds up Apache Iceberg reads and fixes bugs across AWS S3 and object-store datasets, data acceleration, distributed query, and authenticated access.

What's New in v2.0.1​

Faster Iceberg Reads with Parallel File Scanning​

The Apache Iceberg reader now scans data files in parallel (#11331), improving read throughput and latency for Iceberg tables that span many files.

AWS S3 & Object-Store Reliability​

Three fixes improve S3 and object-store dataset behavior:

• Refresh-skip restored (#11339): ETag/Version-based refresh-skip works reliably again, so unchanged S3 objects are no longer re-downloaded on every refresh.
• Retry when source files are not yet available (#11342): an object-store dataset whose source files are not present at startup now retries and becomes ready once the data appears, instead of failing permanently.
• Path-style addressing for dotted bucket names (#11347): on standard AWS, buckets whose names contain dots now default to path-style addressing, avoiding TLS wildcard certificate errors under virtual-hosted-style HTTPS.

Data Acceleration & Distributed Query Fixes​

Two fixes ensure accelerated datasets behave correctly in more configurations:

• Acceleration endpoints (#11345): /v1/datasets/{name}/acceleration/refresh (and the related update-refresh-sql, partition-filters, and snapshots endpoints) now work for all accelerated datasets, fixing cases where some incorrectly reported Table is not accelerated.
• Distributed clusters (#11226): the distributed query coordinator now serves accelerated data from executors for all accelerated datasets, instead of falling back to reading from the source for some.

Authenticated Query Fixes​

With authentication enabled, queries now consistently run as the requesting user (#11253), so per-user behavior such as results caching is correctly scoped to each user.

Contributors​

• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.1 image:

docker pull spiceai/spiceai:2.0.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(ci): key testoperator & validator artifacts by checked-out commit by @sgrebnov in #11281
• chore(deps): fix cargo-deny advisory failures on release/2.0 by @phillipleblanc in #11333
• chore(deps): bump iceberg-rust to parallel file scanning fork (release/2.0) by @phillipleblanc in #11331
• fix(refresh): restore S3 ETag/Version refresh-skip behind provider wrappers by @phillipleblanc in #11339
• fix(runtime): retry object-store dataset load when source files are not yet available by @phillipleblanc in #11342
• feat(s3): default to path-style for dotted bucket names on standard AWS by @phillipleblanc in #11347
• fix(runtime): resolve accelerated table through metadata-enrichment wrapper by @phillipleblanc in #11345
• fix(cluster): distribute accelerated tables wrapped by metadata/index providers by @phillipleblanc in #11226
• fix: scope request context across the managed query runtime by @phillipleblanc in #11253

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0...v2.0.1