Ben Chambers

Spice v2.2.1 is now available! 🛠️

Spice v2.2.1 is a patch release that improves accelerated data management, query federation, and search. Spice Cayenne now reclaims the storage that upserts and compaction leave behind, and new metrics report what each maintenance pass did. Retention policies apply to more Cayenne datasets. Federated queries push more work down to BigQuery. DuckDB and HTTP datasets keep their memory bounded. Vector search with HuggingFace embedding models returns more relevant results.

What's New in v2.2.1​

Cayenne Reclaims Storage Automatically​

A Spice Cayenne acceleration now reclaims the space that a write or a compaction supersedes. Each maintenance pass removes four kinds of state that earlier releases kept:

• tombstones for superseded upsert rows
• snapshot directories that a later commit replaced
• manifest and catalog rows for snapshots that compaction merged away
• deletion vectors that a compaction orphaned

Cayenne also releases the metastore pages that a pass frees. On a table with frequent refreshes or CDC upserts, disk and metastore usage now stay bounded. No configuration change is required.

Retention Policies Apply to More Cayenne Datasets​

Cayenne now applies retention_sql on datasets with refresh_mode: full and on CDC-accelerated datasets with refresh_mode: changes. Before this release, only the append refresh path ran the retention pass. A full refresh reloaded every source row, and the rows that a policy had removed came back.

A retention_sql predicate that calls now() also works now. Cayenne prepared the predicate once when the table opened, at a point where DataFusion cannot evaluate now(), so the pass failed and removed nothing. Cayenne now resolves now() at the start of each pass. See Data Refresh for the retention settings.

datasets:
  - from: postgres:events
    name: events
    acceleration:
      engine: cayenne
      retention_sql: DELETE FROM events WHERE created_at < now() - INTERVAL '7 days'

Cayenne also accepts refresh_append_overlap now. It warns when a dataset sets retention_sql together with mode: memory, or sets indexes, because it cannot apply either setting.

Cayenne Maintenance and Storage Metrics​

Cayenne now reports its maintenance decisions and its storage footprint as metrics on the cayenne meter. The runtime serves them at /metrics when spiced starts with --metrics <addr>. An operator can now tell a pass that found nothing from a pass that declined. The metrics also show where disk and metastore space goes. See Metrics for the metrics endpoint.

• cayenne_compaction_outcome_total and cayenne_maintenance_outcome_total count each pass by outcome, such as committed, no_op, or declined_<reason>.
• cayenne_compaction_trigger_total names the threshold that requested a pass.
• cayenne_maintenance_reclaimed_files_total, cayenne_maintenance_reclaimed_bytes_total, and cayenne_maintenance_reclaimed_rows_total count what each pass reclaimed.
• cayenne_storage_files, cayenne_storage_bytes, and cayenne_storage_rows report the live footprint by storage tier.
• cayenne_data_dir_files, cayenne_data_dir_bytes, and cayenne_data_dir_snapshot_dirs report what the data directory holds on disk.
• cayenne_metastore_db_bytes, cayenne_metastore_wal_bytes, and cayenne_metastore_table_rows report the size of the metastore.

Query Federation and SQL Improvements​

• ADBC federation: The ADBC Data Connector again keeps Spice-only functions such as json_get_str out of the SQL it sends to a source, and it honours query_federation: disabled. The v2.2.0 move of the connector into its own crate dropped both settings. A query with a Spice-only function against BigQuery then failed with an unknown-function error.
• JSON functions in BigQuery: BigQuery federation now translates json_get_int and json_get_float into BigQuery SQL. A filter or a join on a JSON value runs in BigQuery instead of in Spice.
• Negative JSON numbers: json_get_int and json_get_float now read a negative JSON number. Every earlier release returned NULL for one.
• Joins between described datasets: An INNER or FULL join between two datasets that both set a description now plans. DataFusion carries the description as schema metadata, and the join failed with a schema mismatch error. This release updates DataFusion with the fix.

Cayenne Query and Delete Fixes​

• Predicate deletes on upsert tables: A DELETE with a non-primary-key predicate on a Cayenne upsert table now removes only the rows that match. The delete matched the predicate against superseded row versions as well as live rows. A key whose old version matched therefore lost its live row too. The delete now applies the same visibility rules as a query.
• Date filters on Cayenne data files: A pushed-down CAST(date_col AS TIMESTAMP) comparison failed the scan for a Date32 column. For a Date64 column, the same comparison skipped files that held matching rows, so the query returned fewer rows. The scan cast the file's date statistic through the wrong storage type. It now converts the statistic before it prunes on it.
• Primary keys after an interrupted index rebuild: A rebuild of the primary-key index that failed or was cancelled left the index disabled for the life of the process. A later upsert could then write a second live row for a declared primary key, and every write rebuilt the whole keyset. Cayenne now closes the rebuild window on every exit path.
• Row counts after a commit: A distributed COUNT(*) that Cayenne answered from its maintained row count could miss the rows of a commit that had just become visible. Cayenne now claims a commit's row-count delta before it publishes the rows.
• Point lookups on CDC datasets: A primary-key lookup with an ORDER BY on the same key failed to plan on a CDC-accelerated Cayenne dataset after its first transactional commit. An example is WHERE id = 7 ORDER BY id. The runtime rejected the query and returned an error. The scan now forwards its constant columns across the schema cast, and the query plans again.
• Arrow Map columns in file mode: A Cayenne accelerator with mode: file now stores Arrow Map columns, such as the response_headers column of an HTTP dataset. Every write of such a column failed, and the acceleration stayed empty.

DuckDB and HTTP Datasets Keep Memory Bounded​

DuckDB: The bundled DuckDB moves from v1.5.5 to v1.4.4. On v1.5.5, a DuckDB-accelerated dataset that refreshed through an ON CONFLICT ... DO UPDATE upsert grew its memory by about 6 MiB per refresh. The row count stayed steady while the memory grew, and the write eventually failed with an out-of-memory error. The DuckDB project tracks the regression as duckdb/duckdb#25162, and it is present in every v1.5.x release. v1.4.4 holds a steady footprint. Queries that run during a file-mode DuckDB swap also now recover when the swap invalidates their statements. The configured database path stays attached across the swap.

HTTP: The HTTP Data Connector now bounds its response cache. The cache kept every fetched response for the life of the process. An API-proxy workload with request-shaped keys therefore grew by about 1 MB per distinct request.

The cache now holds at most response_cache_max_size_bytes per dataset, 64 MiB by default, and evicts the oldest entries first. It expires an entry by the origin's max-age and Age headers, and it refuses to store a response that sends no-store or no-cache. The new response_cache_fallback_ttl setting keeps a response from an origin that sends no Cache-Control header at all. Two new metrics, response_cache_size_bytes and response_cache_items_count, report occupancy per dataset.

datasets:
  - from: https://api.example.com/v1/items
    name: items
    params:
      response_cache_max_size_bytes: 16777216
      response_cache_fallback_ttl: 5m

More Relevant Vector Search with HuggingFace Embeddings​

Vector search with a HuggingFace sentence-transformer embedding model, such as all-MiniLM-L6-v2, now returns relevant results. The runtime kept the fixed padding that some models declare in tokenizer.json, so it padded every input to 128 tokens. The padding then dominated the vector of a short input. The runtime now clears that padding and lets the embedding layer own the attention mask. On the MTEB SciFact benchmark with all-MiniLM-L6-v2, nDCG@10 rises from 0.018 to 0.640 (published: 0.645). Full-text search and model2vec static models were not affected.

A search filter on a timezone-aware timestamp column now matches the exact instant when the runtime pushes it down to a DuckDB-backed search index. The filter compared whole milliseconds, so a filter inside a millisecond returned different rows than requested.

Change Data Capture and Write-Back​

• PostgreSQL write-back echo suppression: A CDC-accelerated dataset that writes back to its PostgreSQL source now applies each write once. The logical-replication stream returns every write-back transaction as a change. Spice now records the transaction ID of each delivery and drops the matching changes from the stream. Spice drops a replay after a slot reconnect the same way.
• Write-back for BIGINT primary keys: Durable write-back now delivers for a table whose primary key is a single BIGINT column. Such a table acknowledged writes but never delivered them to the source.
• MySQL rebuild after a purged binlog: A MySQL acceleration with mysql_replication_invalid_checkpoint_behavior: restart now rebuilds when the source has purged its binlog position, even when mysql_replication_initial_snapshot: disabled is set. It previously resumed from the current head and kept the rows that the source had deleted in the gap.
• Append refresh on Cayenne: An append refresh on a Cayenne dataset resumes correctly again. The runtime read the append watermark through a timestamp cast that the Cayenne scan could not evaluate. The read failed, and the dataset stopped appending. The runtime now reads the raw column and normalizes the value in memory.

Other Improvements​

• Startup logs: The log events that the runtime emits while it builds now reach the configured log output. These include the DuckDB memory budget and the query memory-limit defaults. Earlier releases dropped them at every log level.
• Spice Cloud CLI: The CLI now reconciles its stored credential state and active organization for Spice Cloud after login.
• Cayenne write concurrency: A Cayenne write that requires a single writer now runs serially even when cayenne_write_concurrency is raised.

Contributors​

• @bjchambers
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

One behavior change to note before you upgrade:

• DuckDB moves from v1.5.5 to v1.4.4 to keep the memory of DuckDB-accelerated datasets bounded under upsert refreshes (duckdb/duckdb#25162). SQL syntax and functions that DuckDB added in v1.5 are not available to DuckDB-accelerated datasets.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.2.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.2.1 image:

docker pull spiceai/spiceai:2.2.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.2.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Make a pinned single partition actually mean one writer by @lukekim in #13325
• feat(postgres): outstanding write-back xid registry for CDC echo suppression by @Jeadie in #13350
• fix(llms): clear tokenizer padding/truncation so TEI owns masking by @Jeadie in #13416
• fix(http): bound the connector's response cache and honour no-store (fixes #13460) by @bjchambers in #13462
• fix(search): render a DuckDB search filter against the table schema (fixes #13144) by @claudespice in #13489
• fix(duckdb): bump datafusion-table-providers for the attachment-race fixes by @sgrebnov in #13491
• fix(cli): reconcile Spice Cloud credential state by @phillipleblanc in #13500
• fix(mysql): rebuild a purged-position acceleration whatever the snapshot mode says (fixes #13024) by @claudespice in #13501
• fix(cayenne): decode write-back markers for a single Int64 primary key by @phillipleblanc in #13502
• fix(deps): bump DataFusion to pick up the join schema-metadata fix by @krinart in #13526
• fix(cayenne): give each cayenne_catalog test a table root of its own (fixes #13527) by @claudespice in #13535
• fix(acceleration): read the append watermark without casting the time column by @sgrebnov in #13539
• test(postgres): cover CDC echo suppression through a real delivery transaction by @phillipleblanc in #13553
• test(federation): guard the derived-table output names a pin bump landed unguarded (refs #12751) by @claudespice in #13555
• fix(runtime): carry a scan's constant columns across the schema cast by @phillipleblanc in #13565
• fix(cayenne): store Arrow Map columns in a file-mode accelerator (fixes #13524) by @bjchambers in #13567
• fix(cayenne): stop a predicate DELETE destroying live rows on an upsert table by @lukekim in #13574
• fix(logging): deliver the runtime build's events to the installed subscriber by @sgrebnov in #13587
• fix(adbc): restore the federation deny-list and query_federation wiring by @peasee in #13590
• feat(cayenne): observability for maintenance decisions and footprint growth by @bjchambers in #13619
• test(forks): guard the Spice patches carried on our upstream forks by @bjchambers in #13623
• test(cayenne): settle the tier before asserting why a subset pass declined by @lukekim in #13632
• fix(cayenne): close the PK-index checkout window on every exit path (fixes #13267) by @claudespice in #13641
• fix(cayenne): reclaim inline tombstones, and return the freed metastore pages by @lukekim in #13651
• fix(cayenne): delete merged-away manifest rows at protected-snapshot compaction by @Jeadie in #13658
• fix(cayenne): prune old-snapshot catalog rows at subset small-file compaction by @Jeadie in #13659
• fix(cayenne): delete emptied protected snapshot manifest rows in retention cleanup by @Jeadie in #13660
• fix(cayenne): schedule the orphaned deletion-vector sweep from every publication that orphans one by @phillipleblanc in #13663
• fix(cayenne): reclaim superseded snapshot dirs against live state by @peasee in #13670
• Translate the JSON extraction functions into BigQuery SQL, and read negative JSON numbers by @phillipleblanc in #13672
• test(forks): guard the four BigQuery dialect unparser fixes the datafusion pin carries by @lukekim in #13691
• fix(cayenne): claim a commit's live-row delta before it publishes the rows (fixes #13036) by @claudespice in #13720
• Downgrade DuckDB to v1.4.4 by @sgrebnov in #13742
• fix(vortex): convert a date statistic into a timestamp before pruning on it (fixes #13624) by @lukekim in #13745
• fix(cayenne): resolve now() in retention predicates once per pass by @lukekim in #13769

**Full Changelog: https://github.com/spiceai/spiceai/compare/v2.2.0...v2.2.1