6 posts tagged with "snapshots"

Spice v2.4.0-rc.2 is now available! 🔥

Spice v2.4.0-rc.2 is the second release candidate of v2.4.0 for early testing and feedback. It introduces a new OpenAI-compatible POST /v1/decisions API backed by an OpenAI decision model, TypeSafe Jev, or any other LLM in the Spicepod that can answer decisions. Five new SQL functions bring the same AI classification, scoring, and decisions into SQL.

This release also adds Iceberg reads without a catalog and improves acceleration snapshots. Cayenne gains runtime-wide adaptive tuning and applies acceleration configuration changes on restart.

Highlights in this release:

• AI Decisions API and decisions in SQL: An OpenAI-compatible /v1/decisions API, plus five SQL functions for filtering, classifying, and scoring data with models.
• Iceberg without a catalog: Read a table directly from its root directory or pin a metadata file to read a fixed version.
• Shared runtime state: Startup validation and a common location for cluster state, rate control, cache warmup, and acceleration snapshots.
• SQL results cache: Cache entries expire by TTL and remain available across data refreshes.
• Cayenne acceleration: Improvements to primary-key loading, secondary indexes, and configuration updates.
• OpenTelemetry log export: Send runtime logs to any OTLP endpoint as structured log records, and choose which failed queries are logged.

These notes cover changes since v2.4.0-rc.1. Review Breaking Changes before upgrading; upgrades from earlier versions also require the rc.1 migration guidance.

What's New in v2.4.0-rc.2​

AI Decisions API and decisions in SQL​

Spice now supports POST /v1/decisions, which is compatible with the OpenAI Decisions API. A decision request asks typed questions about an input. Each answer comes back with its probabilities and a confidence.

Any model in the Spicepod can answer a decision request:

OpenAI decision models use the native OpenAI Decisions API. gpt-6-luna also supports chat and /v1/nsql. TypeSafe Jev returns calibrated probabilities and answers decisions only. Other LLMs answer through chat completions and return their own uncalibrated estimates.

models:
  - from: openai:gpt-6-luna
    name: luna
    params:
      openai_api_key: ${secrets:OPENAI_API_KEY}
  - from: typesafe:jev
    name: jev
    params:
      typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Five new SQL functions make the same decisions available in queries:

SELECT
    id,
    ai_classify(body, ['billing', 'technical', 'account'], model => 'support_model' ) AS team,
    ai_probability(body, 'The customer is requesting a refund', model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI functions for more examples.

Iceberg Without a Catalog​

Read an Iceberg table directly from object storage or the local filesystem, without configuring a catalog. Set from to the table root using file://, s3://, s3a://, gs://, or gcs://:

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1

Spice resolves the metadata file on each query and refresh. It supports metadata/version-hint.text as well as a fixed version (a path relative to the table root or a full URL):

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1
      iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Fixed tables are read-only. The earlier metadata_path parameter remains supported with a deprecation warning; see Breaking Changes.

Shared Runtime State​

runtime.state provides a shared, object-store location for cluster state, rate control, SQL results-cache warmup, and acceleration snapshots. Each feature uses its own subdirectory within the provided location.

Acceleration snapshots now default to the snapshots/ subdirectory when runtime.state is configured.

runtime:
  state:
    location: s3://my-bucket/spice/

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled

A dataset can override the snapshot location, its settings, and the bootstrap behavior on its own acceleration:

datasets:
  - from: postgres:events
    name: events
    acceleration:
      snapshots: enabled
      snapshots_location: s3://archive-bucket/snapshots/
      snapshots_params:
        s3_region: eu-west-1
      snapshots_bootstrap_on_failure_behavior: fallback

Other changes to shared state:

• Snapshot queues: An SQS queue that covers the whole runtime.state prefix no longer causes an ERROR log for each scheduler heartbeat. The snapshot reader now deletes messages for the other state folders without a log.

SQL Results Cache​

The SQL results cache now expires entries by TTL, independent of acceleration refreshes, CDC batches, and retention passes no longer invalidate cached results.

Review item_ttl and stale_while_revalidate_ttl when upgrading: a successful refresh no longer makes cached results reflect the refreshed data immediately. See Breaking Changes.

Cayenne Acceleration​

Adaptive tuning is configured once for the runtime. Enable it with runtime.adaptive_tuning and set targets under runtime.params:

runtime:
  adaptive_tuning: enabled
  params:
    target_freshness: 30s

Supported targets are target_replication_lag, target_freshness, target_query_latency, target_convergence_window, and target_qph. Adaptive tuning defaults to disabled. See Breaking Changes and the runtime YAML documentation.

Other Cayenne improvements include:

• Primary-key loading: Keyed tables build their primary-key filter during the initial load and bulk-load without per-row routing or a whole-table fold.
• Position deletes: cayenne_deletion_mode: auto can select position deletes for small primary-key tables on local storage on Linux systems with at least eight physical cores.
• Secondary indexes: Index runs are persisted, indexes remain complete across acceleration snapshots, and CDC maintains indexes for each operation type. Hash joins read each file once when narrowing a scan through an index.
• Partition handling: Cayenne rejects stored data with an incompatible partition_by, protects keyed appends to partitioned tables during publication, and prepares every partition after a refresh.
• Storage and query correctness: Cayenne reclaims snapshot directories superseded by a reload, corrects pruning for NaN values, and fixes deletion-index bounds and reload fencing.
• Snapshot storage growth: A writer with snapshots_trigger: refresh_complete kept one more copy of the table for each refresh. Each snapshot held file deletions, so Cayenne skipped each cleanup pass. Cayenne now runs the cleanup when a snapshot releases its hold. In a 10-minute test with a refresh every 60 seconds, each dataset kept 1 snapshot directory instead of 10.
• Configuration changes on restart: A change to write settings only, such as cayenne_sort_columns or cayenne_compression_strategy, applies to the existing acceleration. A change to primary_key or partition_by rebuilds the acceleration from the source only under specific conditions. See Cayenne Configuration Updates.

OpenTelemetry Log Export​

Spice can now send its logs to an OpenTelemetry Collector, or any OTLP endpoint, as structured OTLP log records. Logs use the same otel_exporter that already sends metrics. A new runtime.query.log_errors setting chooses which failed queries are logged.

runtime:
  telemetry:
    otel_exporter:
      endpoint: otel-collector:4317
      signals: [metrics, logs] # new; default [metrics]
  query:
    log_errors: execution # new; none (default) | execution | all

• Structured records: Each log line becomes an OTLP log record with its severity, the message as the body, typed fields as attributes, and the logging module as the instrumentation scope. Records share the resource attributes of exported metrics.
• Trace correlation: A record logged during a query carries the query's trace ID, the same ID returned in spice-trace-id and stored in runtime.task_history.
• Delivery: Records are batched on a dedicated export runtime, separate from query and HTTP traffic, and flushed at shutdown so the final error and Goodbye! reach the endpoint. For OTLP/HTTP endpoints, logs are sent to <base>/v1/logs.
• Cluster executors: Executors export their logs with the scheduler's telemetry settings, including headers resolved from the scheduler's secrets.

runtime.query.log_errors controls which failed queries are logged:

Each record reads Query failed: <error>, with error_code and datasets fields and the query's trace ID. The SQL text is never logged.

Unknown-column errors are now reported with the correct error_code instead of InternalError, in logs, runtime.task_history, and the query_failures metric.

See OpenTelemetry Logs Export, Exporting logs, and runtime.query.log_errors.

Fixes and Improvements​

AI APIs and Models​

• Responses API: /v1/responses executes Spice tools and honors tool_recursion_limit when streaming. It returns Spice tool calls as mcp_call items in output, allowing clients that manage their own conversation history to replay them.
• Model controls: /v1/nsql accepts reasoning_effort, and rerank() uses evaluation models with a strategy selected for each query.
• Anthropic: Streaming handles responses that begin with a thinking block, empty text blocks are dropped, and anthropic_workspace_id is available as a model parameter.
• OpenAI: Models with an unknown service tier can load and serve requests.
• Luna: gpt-6-luna and its snapshots support chat, /v1/nsql, and native decisions.

Acceleration​

• Caching point lookups: Lookups on a refresh_mode: caching dataset failed with HTTP 400. This occurred when Cayenne accelerated the dataset, the primary key covered the request columns, and the table held two protected snapshots. The point-lookup plan did not combine the partitions of the Cayenne scan. Lookups now return the cached row or fetch it from the source.

Connectors​

• MySQL timestamps: Timestamp literals in queries that the runtime sends to MySQL now keep their fractional seconds.

Dependency Updates​

Breaking Changes​

Shared State Directory Layout and Startup Checks​

State written at the runtime.state root by rc.1 is no longer read from that location. Move existing state objects into the appropriate subdirectories before upgrading to retain them:

Azure locations (abfs:// and abfss://) need one more step. In rc.1, the runtime ignored the path in an Azure URL and wrote state at the container root. For example, with abfs://container/spice/, rc.1 wrote cluster.json at the container root. This release writes spice/cluster/cluster.json. Move Azure state from the container root into the configured path and then into the feature subdirectory.

An explicit runtime.scheduler.state_location retains its earlier layout. Without migration, Spice starts without the state previously stored at the shared root.

The configured store must accept writes and conditional writes. Spice does not become ready until startup validation succeeds and refuses stores with a newer state format. Check storage permissions and compatibility before upgrading.

SQL Results Cache Freshness​

A refresh no longer invalidates SQL results-cache entries. Queries can return data from before the latest refresh until item_ttl expires, plus stale_while_revalidate_ttl when configured. Reduce item_ttl if results need to follow source refreshes more closely.

Cayenne Tuning Configuration​

Replace the Cayenne tuning parameters with runtime settings that apply across datasets and catalogs:

runtime.adaptive_tuning accepts only enabled or disabled. Other values, including true, fail the Spicepod load.

Pinned Iceberg Tables​

Replace the deprecated metadata_path parameter with iceberg_metadata_path. The two settings resolve paths differently. metadata_path uses its value as given. iceberg_metadata_path resolves a value without a URL scheme relative to the table root in from. Convert an absolute local path to a full file:// URL or to a path relative to the table root.

Datasets pinned to a .json metadata file are read-only and fail to load with access: read_write. Remove that setting from pinned datasets before upgrading.

Cayenne Configuration Updates​

A restart now applies changed settings to an existing Cayenne acceleration. The result depends on the setting and the dataset:

• Write settings only: A change to only cayenne_sort_columns, cayenne_cluster_by, cayenne_compression_strategy, cayenne_delta_encoding, or cayenne_shard_key_columns applies to the existing acceleration. Stored rows keep their layout until a refresh or compaction rewrites them.
• Identity settings: A change to primary_key, partition_by, cayenne_pk_conflict_detection from none to auto, or the data location rebuilds the acceleration from the source. The runtime rebuilds only a dataset with refresh_mode: full, mode: file or file_update, and write_mode: write_through. The source must not be sink:, and must be readable with the schema of the dataset.
• All other datasets: The acceleration keeps all of its stored settings. This includes write settings that changed in the same restart. The runtime logs a warning that names the change and the fix.

While the runtime cannot read the source, a dataset that qualifies for a rebuild keeps its existing acceleration. A partition_by change or a conflict-detection change cannot use the stored rows, so it fails the dataset instead. Revert the setting, or restart when the source is readable.

A restart alone does not migrate a dataset that does not meet these conditions. Read the startup warning for the dataset before you delete stored data. Allow time for a rebuild when the conditions apply.

Upgrading​

Once release artifacts are available, select v2.4.0-rc.2 explicitly with the CLI, Docker tag, or Helm chart version.

CLI​

spice upgrade v2.4.0-rc.2

Docker​

docker pull spiceai/spiceai:2.4.0-rc.2

See Docker Hub for available tags.

Helm​

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.2

Homebrew and AWS Marketplace​

Homebrew and AWS Marketplace follow their published versions. To select this release candidate, use an explicit version through the CLI or Docker.

Spice is also available in the AWS Marketplace.

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @spiceemma

Changelog​

See the full changelog from v2.4.0-rc.1 to v2.4.0-rc.2.

• feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
• fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588
• fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart in #14851
• fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice in #14792
• fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-duckdb[file]-2gib.yaml by @github-actions in #14850
• Update openapi.json by @github-actions in #14858
• test: strengthen assertions and enforce sign-off coverage by @lukekim in #14740
• Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart in #14888
• fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice in #14884
• fix(anthropic): drop empty text blocks from assistant messages by @Jeadie in #14834
• fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim in #14882
• test(sqlite): guard the Date32 literal cast through the SQLite acceleration's own scan (fixes #14491) by @grokspice in #14703
• fix(cayenne): read each file once when a hash join narrows an indexed table's scan by @lukekim in #14934
• fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc in #14946
• fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc in #14939
• fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim in #14944
• fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim in #14955
• fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc in #14942
• fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc in #14960
• feat(spicepod): publish pull request reviews by @lukekim in #14964
• fix(ci): bound the REPL expect-script waits on silence, not total response time (fixes #13711) by @grokspice in #14963
• fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc in #14957
• fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc in #14956
• fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim in #14798
• fix(ci): provision DuckLake for the ducklake[catalog] schema test, which has never passed by @grokspice in #14952
• fix(llms): stream Anthropic responses that open with a thinking block (fixes #14909) by @krinart in #14920
• fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov in #14968
• feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim in #14852
• fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart in #14972
• fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart in #14927
• feat(cayenne): always persist secondary index runs by @sgrebnov in #14979
• fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers in #14919
• feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim in #14838
• fix(rrf): keep named arguments on nested search calls by @phillipleblanc in #14975
• fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart in #14932
• fix(vortex): keep temporal casts Vortex can't evaluate out of the scan by @sgrebnov in #14985
• fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers in #14928
• test(huggingface): unblock the merge queue: read imdb's parquet conversion through @~parquet and fail fast on a vanished Hub pin by @grokspice in #14998
• test(cayenne): link the integration tests into one test binary by @bjchambers in #14976
• test(vortex): compile the integration tests into a binary the sign-off gate runs by @bjchambers in #14987
• fix(http): reject a request_path that URL parsing would rewrite by @bjchambers in #14965
• fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc in #14953
• Add anthropic_workspace_id param for Anthropic models by @Jeadie in #14829
• fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim in #14783
• test: run federated source tests on every supported LTS release line by @lukekim in #14936
• fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius in #14791
• fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim in #14806
• fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim in #15009
• Improve types methods in runtime-rate-control by @Jeadie in #14573
• fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart in #14930
• Update openapi.json by @github-actions in #14908
• test(cayenne): run the heap-measuring tests in one isolated test binary by @bjchambers in #15010
• test(vortex): run vortex-datafusion's integration tests in the sign-off gate by @bjchambers in #15001
• fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc in #15019
• Update openapi.json by @github-actions in #15018
• fix(ci): keep the OpenAPI regeneration PR current instead of skipping while one is open by @phillipleblanc in #15023
• fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim in #14993
• fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers in #14891
• chore: Update Turso crate to 0.8.2 by @spiceemma in #15022
• feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee in #14844
• ci: apply whole-binary nextest overrides to the module test sets they select by @bjchambers in #15011
• fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc in #15028
• test(data_components): compile the integration tests into one test binary by @bjchambers in #14991
• fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc in #15053
• fix(runtime): give each runtime.state feature its own folder by @phillipleblanc in #15036
• test(data_components): make the HTTP provider snapshots independent of the machine by @bjchambers in #15000
• fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc in #15054
• test(runtime): drive the cdc_cayenne tests through Cayenne's change sink by @bjchambers in #15003
• ci(object_store_occ): run the WAL E2E suite on self-hosted runners against disk, MinIO, RustFS and spiceio by @lukekim in #15031
• test(llms): compile the credential-free tests into the integration binary by @bjchambers in #14990
• fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov in #15068
• ci(setup-minio): name the missing secret instead of failing in mc's argument parser by @grokspice in #15073
• ci: run Substrait Mode B on the Linux runners that can reach the trunk spiced builds by @grokspice in #15074
• feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie in #14782
• build(deps): bump the github-actions-dependencies group across 5 directories with 7 updates by @dependabot in #15027
• fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov in #15069
• feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov in #15081
• Fixes for MCP server by @Jeadie in #15041
• fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice in #15056
• fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice in #15037
• perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers in #15072
• fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart in #14931
• fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart in #14937
• fix(decisions): Fixes for /v1/decisions by @Jeadie in #15040
• perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers in #15076
• fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers in #15077
• test(runtime,spiced): compile integration tests into declared binaries, run the ones no CI job ran, and remove the uncalled supersede_initial_load by @bjchambers in #15005
• fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim in #15083
• ci(e2e): don't run rust-cache on the macOS build, whose save deletes cargo from sibling runners by @grokspice in #15100
• feat(testoperator): record per-query search metrics by @Jeadie in #15090
• fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc in #15096
• feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie in #14340
• ci: run merge-queue Mode A TPC-H and LLM integration tests on spiceai-macos-large by @lukekim in #15116
• docs(cayenne): the per-file fold merges no source files, but may roll by @bjchambers in #15101
• build(deps): bump async-openai to the fork's merge of upstream main by @krinart in #14980
• feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov in #15095
• fix(runtime): fail recovery of an unsaved running query by @phillipleblanc in #15092
• perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc in #15099
• test(cayenne): guard per-file statistics sharing for listings outside a scan view by @bjchambers in #15103
• feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers in #15033
• fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers in #14140
• Change codeql runner from spiceai-macos to spiceai-macos-large by @lukekim in #15120
• fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim in #15106
• fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim in #15108
• fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers in #15111
• fix: Update Search integration test snapshots by @github-actions in #15050
• test(iceberg): read the Glue fixture unpinned and check the picked metadata file by @sgrebnov in #15115
• test(cayenne): differential and plan coverage for secondary indexes on partitioned accelerations by @bjchambers in #15113
• test: answer results-cache misses from the acceleration while the source is down by @bjchambers in #15112
• ci(lint): fail on test files no target compiles, and document the integration test layout by @bjchambers in #15109
• feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov in #15117
• fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov in #15110
• fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart in #14977
• feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim in #15042
• fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim in #15105
• fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc in #15128
• fix(test): wait for pkd_ctl to load before reading its bootstrap row in the write-back pk_conflict test by @grokspice in #15102
• fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart in #15126
• test(mysql): skip the SQL results cache in the binlog replication end-to-end tests by @grokspice in #15141
• feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc in #15138
• feat(acceleration): apply changed acceleration settings on restart by @lukekim in #15094
• fix(models): enable Luna chat and NSQL support by @krinart in #15166
• fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc in #15177
• fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov in #15179
• feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim in #15176

Announcing the release of Spice v1.11.0-stable! ⚡

In Spice v1.11.0, Spice Cayenne reaches Beta status with acceleration snapshots, Key-based deletion vectors, and Amazon S3 Express One Zone support. DataFusion has been upgraded to v51 along with Arrow v57.2, and iceberg-rust v0.8.0. v1.11 adds several DynamoDB & DynamoDB Streams improvements such as JSON nesting, and adds significant improvements to Distributed Query with active-active schedulers and mTLS for enterprise-grade high-availability and secure cluster communication.

This release also adds new SMB, NFS, and ScyllaDB Data Connectors (Alpha), Prepared Statements with full SDK support (gospice, spice-rs, spice-dotnet, spice-java, spice.js, and spicepy), Google LLM Support for expanded AI inference capabilities, and significant improvements to caching, observability, and Hash Indexing for Arrow Acceleration.

What's New in v1.11.0​

Spice Cayenne Accelerator Reaches Beta​

Spice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous performance and stability improvements.

Key Enhancements:

• Key-based Deletion Vectors: Improved deletion vector support using key-based lookups for more efficient data management and faster delete operations. Key-based deletion vectors are more memory-efficient than positional vectors for sparse deletions.
• S3 Express One Zone Support: Store Cayenne data files in S3 Express One Zone for single-digit millisecond latency, ideal for latency-sensitive query workloads that require persistence.

Improved Reliability:

• Resolved FuturesUnordered reentrant drop crashes
• Fixed memory growth issues related to Vortex metrics allocation
• Metadata catalog now properly respects cayenne_file_path location
• Added warnings for unparseable configuration values

For more details, refer to the Cayenne Documentation.

DataFusion v51 Upgrade​

Apache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.

Performance Improvements:

• Faster CASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patterns
• Better Defaults for Remote Parquet Reads: DataFusion now fetches the last 512KB of Parquet files by default, typically avoiding 2 I/O requests per file
• Faster Parquet Metadata Parsing: Leverages Arrow 57's new thrift metadata parser for up to 4x faster metadata parsing

New SQL Features:

• SQL Pipe Operators: Support for |> syntax for inline transforms
• DESCRIBE <query>: Returns the schema of any query without executing it
• Named Arguments in SQL Functions: PostgreSQL-style param => value syntax for scalar, aggregate, and window functions
• Decimal32/Decimal64 Support: New Arrow types supported including aggregations like SUM, AVG, and MIN/MAX

Example pipe operator:

SELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;

Improved Observability:

• Improved EXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdowns

Arrow 57.2 Upgrade​

Apache Arrow has been upgraded to v57.2, bringing major performance improvements and new capabilities.

Key Features:

• 4x Faster Parquet Metadata Parsing: A rewritten thrift metadata parser delivers up to 4x faster metadata parsing, especially beneficial for low-latency use cases and files with large amounts of metadata
• Parquet Variant Support: Experimental support for reading and writing the new Parquet Variant type for semi-structured data, including shredded variant values
• Parquet Geometry Support: Read and write support for Parquet Geometry types (GEOMETRY and GEOGRAPHY) with GeospatialStatistics
• New arrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution support

DynamoDB Connector Enhancements​

• Added JSON nesting for DynamoDB Streams
• Improved batch deletion handling

Distributed Query Improvements​

High Availability Clusters: Spice now supports running multiple active schedulers in an active/active configuration for production deployments. This eliminates the scheduler as a single point of failure and enables graceful handling of node failures.

• Multiple schedulers run simultaneously, each capable of accepting queries
• Schedulers coordinate via a shared S3-compatible object store
• Executors discover all schedulers automatically
• A load balancer distributes client queries across schedulers

Example HA configuration:

runtime:
  scheduler:
    state_location: s3://my-bucket/spice-cluster
    params:
      region: us-east-1

mTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.

Credential Propagation: S3, ABFS, and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.

Improved Resilience:

• Exponential backoff for scheduler disconnection recovery
• Increased gRPC message size limit from 16MB to 100MB for large query plans
• HTTP health endpoint for cluster executors
• Automatic executor role inference when --scheduler-address is provided

For more details, refer to the Distributed Query Documentation.

iceberg-rust v0.8.0 Upgrade​

Spice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.

Key Features:

• V3 Metadata Support: Full support for Iceberg V3 table metadata format
• INSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tables
• Improved Delete File Handling: Better support for position and equality delete files, including shared delete file loading and caching
• SQL Catalog Updates: Implement update_table and register_table for SQL catalog
• S3 Tables Catalog: Implement update_table for S3 Tables catalog
• Enhanced Arrow Integration: Convert Arrow schema to Iceberg schema with auto-assigned field IDs, _file column support, and Date32 type support

Acceleration Snapshots​

Acceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.

Key Features:

• Flexible Triggers: Configure when snapshots are created based on time intervals or stream batch counts
• Automatic Compaction: Reduce storage overhead by compacting older snapshots (DuckDB only)
• Bootstrap Integration: Snapshots can reset cache expiry on load for seamless recovery (DuckDB with Caching refresh mode)
• Smart Creation Policies: Only create snapshots when data has actually changed

Example configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled
      snapshots_trigger: time_interval
      snapshots_trigger_threshold: 1h
      snapshots_creation_policy: on_changed

Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically.

CLI Commands:

# List all snapshots for a dataset
spice acceleration snapshots taxi_trips

# Get details of a specific snapshot
spice acceleration snapshot taxi_trips 3

# Set the current snapshot for rollback (requires runtime restart)
spice acceleration set-snapshot taxi_trips 2

HTTP API Endpoints:

For more details, refer to the Acceleration Snapshots Documentation.

Caching Acceleration Mode Improvements​

The Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.

Performance Optimizations:

• Non-blocking Cache Writes: Cache misses no longer block query responses. Data is written to the cache asynchronously after the query returns, reducing query latency for cache miss scenarios.
• Batch Cache Writes: Multiple cache entries are now written in batches rather than individually, significantly improving write throughput for high-volume cache operations.

Reliability Fixes:

• Correct SWR Refresh Behavior: The stale-while-revalidate (SWR) pattern now correctly refreshes only the specific entries that were accessed instead of refreshing all stale rows in the dataset. This prevents unnecessary source queries and reduces load on upstream data sources.
• Deduplicated Refresh Requests: Fixed an issue where JSON array responses could trigger multiple redundant refresh operations. Refresh requests are now properly deduplicated.
• Fixed Cache Hit Detection: Resolved an issue where queries that didn't include fetched_at in their projection would always result in cache misses, even when cached data was available.
• Unfiltered Query Optimization: SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.

For more details, refer to the Caching Acceleration Mode Documentation.

Prepared Statements​

Improved Query Performance and Security: Spice now supports prepared statements, enabling parameterized queries that improve both performance through query plan caching and security by preventing SQL injection attacks.

Key Features:

• Query Plan Caching: Prepared statements cache query plans, reducing planning overhead for repeated queries
• SQL Injection Prevention: Parameters are safely bound, preventing SQL injection vulnerabilities
• Arrow Flight SQL Support: Full prepared statement support via Arrow Flight SQL protocol

SDK Support:

Example (Go):

import "github.com/spiceai/gospice/v8"

client, _ := spice.NewClient()
defer client.Close()

// Parameterized query with typed parameters
results, _ := client.SqlWithParams(ctx,
    "SELECT * FROM products WHERE price > $1 AND category = $2",
    spice.Float64Param(10.0),
    spice.StringParam("electronics"),
)

Example (Java):

import ai.spice.SpiceClient;
import ai.spice.Param;
import org.apache.arrow.adbc.core.ArrowReader;

try (SpiceClient client = new SpiceClient()) {
    // With automatic type inference
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM products WHERE price > $1 AND category = $2",
        10.0, "electronics");

    // With explicit typed parameters
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM products WHERE price > $1 AND category = $2",
        Param.float64(10.0),
        Param.string("electronics"));
}

For more details, refer to the Parameterized Queries Documentation.

Spice Java SDK v0.5.0​

Parameterized Query Support for Java: The Spice Java SDK v0.5.0 introduces parameterized queries using ADBC (Arrow Database Connectivity), providing a safer and more efficient way to execute queries with dynamic parameters.

Key Features:

• SQL Injection Prevention: Parameters are safely bound, preventing SQL injection vulnerabilities
• Automatic Type Inference: Java types are automatically mapped to Arrow types (e.g., double → Float64, String → Utf8)
• Explicit Type Control: Use the new Param class with typed factory methods (Param.int64(), Param.string(), Param.decimal128(), etc.) for precise control over Arrow types
• Updated Dependencies: Apache Arrow Flight SQL upgraded to 18.3.0, plus new ADBC driver support

Example:

import ai.spice.SpiceClient;
import ai.spice.Param;

try (SpiceClient client = new SpiceClient()) {
    // With automatic type inference
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM taxi_trips WHERE trip_distance > $1 LIMIT 10",
        5.0);

    // With explicit typed parameters for precise control
    ArrowReader reader = client.queryWithParams(
        "SELECT * FROM orders WHERE order_id = $1 AND amount >= $2",
        Param.int64(12345),
        Param.decimal128(new BigDecimal("99.99"), 10, 2));
}

Maven:

<dependency>
  <groupId>ai.spice</groupId>
  <artifactId>spiceai</artifactId>
  <version>0.5.0</version>
</dependency>

For more details, refer to the Spice Java SDK Repository.

Google LLM Support​

Expanded AI Provider Support: Spice now supports Google embedding and chat models via the Google AI provider, expanding the available LLM options for AI inference workloads alongside existing providers like OpenAI, Anthropic, and AWS Bedrock.

Key Features:

• Google Chat Models: Access Google's Gemini models for chat completions
• Google Embeddings: Generate embeddings using Google's text embedding models
• Unified API: Use the same OpenAI-compatible API endpoints for all LLM providers

Example spicepod.yaml configuration:

models:
  - from: google:gemini-2.0-flash
    name: gemini
    params:
      google_api_key: ${secrets:GOOGLE_API_KEY}

embeddings:
  - from: google:text-embedding-004
    name: google_embeddings
    params:
      google_api_key: ${secrets:GOOGLE_API_KEY}

For more details, refer to the Google LLM Documentation (see docs PR #1286).

URL Tables​

Query data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.

Supported Patterns:

• Single files: SELECT * FROM 's3://bucket/data.parquet'
• Directories/prefixes: SELECT * FROM 's3://bucket/data/'
• Glob patterns: SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'

Key Features:

• Automatic file format detection (Parquet, CSV, JSON, etc.)
• Hive-style partition inference with filter pushdown
• Schema inference from files
• Works with both SQL and DataFrame APIs

Example with hive partitioning:

-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'

Enable via spicepod.yml:

runtime:
  params:
    url_tables: enabled

Cluster Mode Async Query APIs (experimental)​

New asynchronous query APIs for long-running queries in cluster mode:

• /v1/queries endpoint: Submit queries and retrieve results asynchronously

OpenTelemetry Improvements​

Unified Telemetry Endpoint: OTel metrics ingestion has been consolidated to the Flight port (50051), simplifying deployment by removing the separate OTel port (50052). The push-based metrics exporter continues to support integration with OpenTelemetry collectors.

Note: This is a breaking change. Update your configurations if you were using the dedicated OTel port 50052. Internal cluster communication now uses port 50052 exclusively.

Observability Improvements​

Enhanced Dashboards: Updated Grafana and Datadog example dashboards with:

• Snapshot monitoring widgets
• Improved accelerated datasets section
• Renamed ingestion lag charts for clarity

Additional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.

For more details, refer to the Monitoring Documentation.

Hash Indexing for Arrow Acceleration (experimental)​

Arrow-based accelerations now support hash indexing for faster point lookups on equality predicates. Hash indexes provide O(1) average-case lookup performance for columns with high cardinality.

Features:

• Primary key hash index support
• Secondary index support for non-primary key columns
• Composite key support with proper null value handling

Example configuration:

datasets:
  - from: postgres:users
    name: users
    acceleration:
      enabled: true
      engine: arrow
      primary_key: user_id
      indexes:
        '(tenant_id, user_id)': unique  # Composite hash index

For more details, refer to the Hash Index Documentation.

SMB and NFS Data Connectors​

Network-Attached Storage Connectors: New data connectors for SMB (Server Message Block) and NFS (Network File System) protocols enable direct federated queries against network-attached storage without requiring data movement to cloud object stores.

Key Features:

• SMB Protocol Support: Connect to Windows file shares and Samba servers with authentication support
• NFS Protocol Support: Connect to Unix/Linux NFS exports for direct data access
• Federated Queries: Query Parquet, CSV, JSON, and other file formats directly from network storage with full SQL support
• Acceleration Support: Accelerate data from SMB/NFS sources using DuckDB, Spice Cayenne, or other accelerators

Example spicepod.yaml configuration:

datasets:
  # SMB share
  - from: smb://fileserver/share/data.parquet
    name: smb_data
    params:
      smb_username: ${secrets:SMB_USER}
      smb_password: ${secrets:SMB_PASS}

  # NFS export
  - from: nfs://nfsserver/export/data.parquet
    name: nfs_data

For more details, refer to the Data Connectors Documentation.

ScyllaDB Data Connector​

A new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.

Example configuration:

datasets:
  - from: scylladb:my_keyspace.my_table
    name: scylla_data
    acceleration:
      enabled: true
      engine: duckdb

For more details, refer to the ScyllaDB Data Connector Documentation.

Flight SQL TLS Connection Fixes​

TLS Connection Support: Fixed TLS connection issues when using grpc+tls:// scheme with Flight SQL endpoints. Added support for custom CA certificate files via the new flightsql_tls_ca_certificate_file parameter.

Developer Experience Improvements​

• Turso v0.3.2 Upgrade: Upgraded Turso accelerator for improved performance and reliability
• Rust 1.91 Upgrade: Updated to Rust 1.91 for latest language features and performance improvements
• Spice Cloud CLI: Added spice cloud CLI commands for cloud deployment management
• Improved Spicepod Schema: Improved JSON schema generation for better IDE support and validation
• Acceleration Snapshots: Added configurable snapshots_create_interval for periodic acceleration snapshots independent of refresh cycles
• Tiered Caching with Localpod: The Localpod connector now supports caching refresh mode, enabling multi-layer acceleration where a persistent cache feeds a fast in-memory cache
• GitHub Data Connector: Added workflows and workflow runs support for GitHub repositories
• NDJSON/LDJSON Support: Added support for Newline Delimited JSON and Line Delimited JSON file formats

Additional Improvements & Bug Fixes​

• Model Listing: New functionality to list available models across multiple AI providers
• DuckDB Partitioned Tables: Primary key constraints now supported in partitioned DuckDB table mode
• Post-refresh Sorting: New on_refresh_sort_columns parameter for DuckDB enables data ordering after writes
• Improved Install Scripts: Removed jq dependency and improved cross-platform compatibility
• Better Error Messages: Improved error messaging for bucket UDF arguments and deprecated OpenAI parameters
• Reliability: Fixed DynamoDB IAM role authentication with new dynamodb_auth: iam_role parameter
• Reliability: Fixed cluster executors to use scheduler's temp_directory parameter for shuffle files
• Reliability: Initialize secrets before object stores in cluster executor mode
• Reliability: Added page-level retry with backoff for transient GitHub GraphQL errors
• Performance: Improved statistics for rewritten DistributeFileScanOptimizer plans
• Developer Experience: Added max_message_size configuration for Flight service

Contributors​

• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @mach-kernel
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

OTel Ingestion Port Change​

OTel ingestion has been moved to the Flight port (50051), removing the separate OTel port 50052. Port 50052 is now used exclusively for internal cluster communication. Update your configurations if you were using the dedicated OTel port.

Distributed Query Cluster Mode Requires mTLS​

Distributed query cluster mode now requires mTLS for secure communication between cluster nodes. This is a security enhancement to prevent unauthorized nodes from joining the cluster and accessing secrets.

Migration Steps:

1. Generate certificates using spice cluster tls init and spice cluster tls add
2. Update scheduler and executor startup commands with --node-mtls-* arguments
3. For development/testing, use --allow-insecure-connections to opt out of mTLS

Renamed CLI Arguments:

Removed CLI Arguments:

• --cluster-api-key: Replaced by mTLS authentication

Cookbook Updates​

New ScyllaDB Data Connector Recipe: New recipe demonstrating how to use the ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.

New SMB Data Connector Recipe: New recipe demonstrating how to use the SMB Data Connector. See SMB Data Connector Recipe for details.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.0 image:

docker pull spiceai/spiceai:1.11.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

Dependencies​

• DataFusion: Upgraded to v51 (release notes)
• Arrow: Upgraded to v57.2 (release notes)
• iceberg-rust: Upgraded to v0.8.0 (release notes)

What's Changed​

Changelog​

• OTel exporter for push metrics by @lukekim in #8442
• fix: Update benchmark snapshots by @app/github-actions in #8448
• Add TPCH append tests to scheduled dispatch workflow by @sgrebnov in #8451
• Add snapshot creation logging by @krinart in #8469
• Fix PeriodicReader panic by @krinart in #8471
• Benchmarks: increase readiness timeout for turso acceleration (TPC-H) by @sgrebnov in #8470
• fix: Pin CUDA build actions to commits by @peasee in #8477
• Add Criterion benchmarking to chunking crate. by @Jeadie in #8431
• DuckDB agg pushdown: gate behind accelerator parameter by @mach-kernel in #8474
• Rename aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485
• Handle throttling exception for DynamoDB streams by @phillipleblanc in #8492
• docs: Add release notes by @peasee in #8478
• Update spicepod.schema.json by @app/github-actions in #8496
• Move 'test_projection_pushdown' to runtime-datafusion by @Jeadie in #8490
• Fix OTEL metrics HTTP exporter client setup by @phillipleblanc in #8489
• Update endgame to include new caching accelerator cookbook by @phillipleblanc in #8487
• DynamoDB tests and fixes by @lukekim in #8491
• Align make lint-rust-fix with make lint-rust by @Jeadie in #8499
• fix: Remove unused Cayenne parameters by @peasee in #8500
• Force task history captured_plan outputs to be captured even if they would be filtered out otherwise by @phillipleblanc in #8501
• release: post-release updates by @peasee in #8503
• CI: Fix E2E models dispatch by @mach-kernel in #8505
• Use an isolated Tokio runtime for refresh tasks that is separate from the main query API by @phillipleblanc in #8504
• Update openapi.json by @app/github-actions in #8512
• Update dependencies by @phillipleblanc in #8513
• fix: Avoid double hashing cache key by @peasee in #8511
• fix: Eagerly drop cached records for results larger than max by @peasee in #8516
• feat: Support vortex zstd compressor by @peasee in #8515
• warning if column is defined in spicepod but is non-existant by @Jeadie in #8498
• Return summarized spicepods from /v1/spicepods by @phillipleblanc in #8404
• Fix for idle DynamoDB Stream by @krinart in #8506
• Use Datafusion::Plan over Datafusion::Internal for user-facing search errors. by @Jeadie in #8484
• DDB Streams Integration Test + Memory Acceleration + Improved Warning by @krinart in #8520
• Upgrade to gospice v8 by @lukekim in #8524
• Vortex file format for object store by @Jeadie in #8525
• docs: Add missing cookbooks to endgame, focus area section by @peasee in #8527
• ListingTableConnector: Drop partition columns that reoccur in file schema by @mach-kernel in #8519
• fix(cluster): initialize secrets before object stores in executor by @sgrebnov in #8532
• Use separate Tokio runtime for SWR refreshes by @phillipleblanc in #8530
• fix: SQL Results Cache SWR triggers incorrect cache miss metric by @phillipleblanc in #8529
• feat: testoperator: OTLP streaming metrics / connect to existing instance / infinite mode / query timeout by @phillipleblanc in #8537
• Update openapi.json by @app/github-actions in #8518
• Add better attributes for search testing by @Jeadie in #8531
• fix: Improve Cayenne errors, ID selection for table/partition creation by @peasee in #8523
• Percent-encode Kubernetes secret name path segment by @phillipleblanc in #8522
• Initial parameterisation of Search integration tests by @Jeadie in #8066
• fix: Add warning when multiple partitions are defined for the same table by @peasee in #8540
• Add DuckDB file-mode support for search test parameterization. by @Jeadie in #8541
• fix: Add recursion depth limits to prevent DoS via deeply nested data (DynamoDB + S3 Vectors) by @phillipleblanc in #8544
• Remove the clippy::too_many_lines lint by @phillipleblanc in #8549
• feat: Add spice cluster tls commands by @phillipleblanc in #8550
• Move OTel ingestion to Flight port, remove separate OTel port 50052 by @phillipleblanc in #8551
• Move verbose tool init messages to trace by @phillipleblanc in #8552
• Add SANs to spice cluster tls add certificates by @phillipleblanc in #8554
• Add cpu, gpu, and memory to telemetry by @lukekim in #8483
• feat: Add workflows and workflow runs to GitHub Data Connector by @peasee in #8548
• Add S3Vectors option for paramterised search tests by @Jeadie in #8555
• Distributed query: TLS + API key by @mach-kernel in #8468
• Refactor RRF SQL to use LogicalPlanBuilder by @Jeadie in #7968
• Require mTLS for distributed query cluster mode by @phillipleblanc in #8580
• Fix stats for rewritten DistributeFileScanOptimizer plans by @mach-kernel in #8581
• Bump actions/cache from 4.3.0 to 5.0.1 by @app/dependabot in #8573
• Show user-friendly error on empty DDB table by @krinart in #8586
• Add (Deprecated) labels to deprecated spice sql params by @krinart in #8588
• Fix kafka warning when security.protocol is set to PLAINTEXT by @krinart in #8587
• Upgrade dependencies by @phillipleblanc in #8593
• Release notes for v1.10.1 by @Jeadie in #8568
• Run search benchmarks twice a week. by @Jeadie in #8592
• Add cayenne data accelerator by @Jeadie in #8553
• SQL allowlist for tools: sql, list_datasets, search, table_schema by @Jeadie in #8449
• use rstest for llms integration tests by @Jeadie in #8566
• Post v1.10.1 housekeeping by @Jeadie in #8600
• Add checklist for SDK publication in end_game.md by @Jeadie in #8602
• Fix IMAGE_TAG assignment for Docker compatibility by @Jeadie in #8599
• Modify cluster arguments to spiced for UX review by @phillipleblanc in #8603
• Update QA analytics with new release data by @Jeadie in #8601
• Remove 'tract-core' dependency. by @Jeadie in #8605
• Google embedding and chat models. by @Jeadie in #8423
• Fix test_github_workflows integration test by @sgrebnov in #8607
• Update openapi.json by @app/github-actions in #8604
• ci: Upload artifacts to MinIO eagerly after each build step by @phillipleblanc in #8615
• fix: SQLite accelerator decimal/date handling by @phillipleblanc in #8606
• Configure mTLS for executor-to-executor gRPC connections by @sgrebnov in #8617
• feat: Enable localpod with caching mode accelerator for tiered caching by @phillipleblanc in #8621
• Add Cayenne S3 Express One Zone support for data files by @lukekim in #8502
• Add snapshot interval for acceleration snapshots by @phillipleblanc in #8627
• Add dataset_load_parallelism parameter to spicepod.yml by @peasee in #8630
• Json Nesting for DynamoDB by @krinart in #8623
• Restore deprecated open-telemetry flag in spiced by @phillipleblanc in #8629
• Implement batching for Kafka/Debezium + null Decimal handling by @krinart in #8622
• fix: Status field in /v1/datasets & /v1/models by @lukekim in #8633
• Add Spice test operator improvements by @sgrebnov in #8625
• Add v1.10.2 release notes by @sgrebnov in #8640
• Align object-store vortex with runtime feature flagging by @Jeadie in #8620
• Extended LLM & search tests on cron by @Jeadie in #8624
• Test-operator: emit main metrics as part of load tests by @sgrebnov in #8639
• Build a local docker image from an existing Spice binary by @phillipleblanc in #8619
• fix: Use runtime-rate-control for GitHub Data Connector by @peasee in #8638
• Upgrade dependencies by @phillipleblanc in #8655
• Bump headers-accept from 0.1.4 to 0.3.0 by @app/dependabot in #8644
• Update openapi.json by @app/github-actions in #8637
• Update SECURITY.md - Include v1.10.2 by @sgrebnov in #8661
• Serialize acceleration snapshots with refresh writes by @phillipleblanc in #8652
• Update AI Installation test to use minilm_l6_v2 by @sgrebnov in #8659
• Fixes for search integration test CI by @Jeadie in #8656
• fix: Use a GitHub rate controller per auth context by @peasee in #8662
• fix: Update Search integration test snapshots by @app/github-actions in #8654
• Make E2E Test Release Installation (AI, Local HF model) test more robust by @sgrebnov in #8666
• Fix issue with location predicate for custom S3 endpoints + regression integration test by @phillipleblanc in #8668
• fix: Validate schema match before projection pushdown in UnionProjectionPushdownOptimizer by @phillipleblanc in #8669
• Proper batch commit for kafka/debezium by @krinart in #8671
• Improve spicepod json schema generation by @ewgenius in #8547
• Start the anonymous telemetry exporter asynchronously by @phillipleblanc in #8679
• fix: Move enforce-pulls to hosted runner by @phillipleblanc in #8686
• Update QA analytics with 1.10.2 release data by @sgrebnov in #8667
• fix: Azure does not support suffix range requests by @phillipleblanc in #8685
• Remove spicepod-validator cargo build from build-dev target by @Jeadie in #8684
• fix: Update test snapshots by @app/github-actions in #8680
• fix: Update Search integration test snapshots by @app/github-actions in #8681
• SMB and NFS Data Connectors by @lukekim in #8674
• Upgrade to openai-async v0.32 by @lukekim in #8635
• v1.10.3 release notes by @phillipleblanc in #8693
• Upgrade dependencies by @phillipleblanc in #8704
• fix: Support NDJSON and LDJSON by @lukekim in #8649
• move OpenAI overrides to non-prefixed by @Jeadie in #8678
• Update Google LLM param: google_dimensions -> dimensions by @Jeadie in #8677
• Make cluster mTLS optional with insecure flag by @phillipleblanc in #8703
• Revert "fix: Move enforce-pulls to hosted runner (#8686)" by @phillipleblanc in #8709
• Initial 'testoperator run text-to-sql' by @Jeadie in #8618
• Add support for abfss by @krinart in #8706
• Add testoperator TPCH dispatch for ABFS with hierarchical namespace disabled + versioning enabled by @phillipleblanc in #8711
• Update openapi.json by @app/github-actions in #8692
• cluster: validate --role argument by @phillipleblanc in #8717
• Upgrade to Turso v0.3.2 by @lukekim in #8716
• Rename --insecure to --allow-insecure-connections to be consistent with existing naming by @lukekim in #8720
• Remove 'testoperator run http-consistency/http-overhead' by @Jeadie in #8708
• refactor: Remove cluster feature flag by @phillipleblanc in #8718
• Docs: Distributed query ADR by @mach-kernel in #8608
• Use model.datasets to allowlist on tools by @Jeadie in #8714
• cluster: quality of life improvements to starting cluster mode locally by @phillipleblanc in #8719
• Docs: Ballista extension ADR by @mach-kernel in #8616
• Improve deprecation messages when going from prefixed -> non-prefixed. by @Jeadie in #8724
• Remove tools from auto-defaults by @Jeadie in #8725
• Make distinct providers for vector spilling, vector partitioning. by @Jeadie in #8546
• cluster: default scheduler address port by @phillipleblanc in #8728
• Add Makefile targets for testoperator by @Jeadie in #8729
• text-to-sql dispatch in testoperator by @Jeadie in #8705
• DR-006: High Availability Distributed Query with Stateless Schedulers by @lukekim in #8721
• DR-007: mTLS for Distributed Query Cluster Communication by @lukekim in #8722
• SMB and NFS improvements by @lukekim in #8710
• fix: Cluster executors use scheduler's temp_directory for shuffle files by @phillipleblanc in #8733
• use 'max_message_size' in flight service too by @Jeadie in #8730
• Add page-level retry for transient GraphQL errors with backoff and increase GitHub rate limit buffer up to 100 by @ewgenius in #8726
• Make testoperator Dockerfile; CI to build docker image to ghcr.io. by @Jeadie in #8732
• cluster: UnionProjectionPushdownOptimizer: Add projection pushdown diagnostics for union children by @phillipleblanc in #8734
• Fix column projection order mismatch with location metadata columns by @phillipleblanc in #8738
• Fixes for testoperator. by @Jeadie in #8737
• Improve Cayenne Deletion Vectors with KeyBased support by @lukekim in #8713
• Fix testoperator_dispatch.yaml by @Jeadie in #8740
• Add spice cloud CLI commands by @lukekim in #8528
• Add FTP, NFS, & SMB TPCH SF1 spicepods by @lukekim in #8739
• Prepared Statements by @lukekim in #7588
• Schedule dispatch of testoperator run text-to-sql. by @Jeadie in #8745
• Fix minio for ai benchmark CI by @Jeadie in #8743
• Upgrade to Rust 1.91 by @phillipleblanc in #8749
• fix: Update benchmark snapshots by @app/github-actions in #8763
• Benchmarks: make row count validation skip logic configurable by scale factor, query set, and overrides by @sgrebnov in #8756
• Make benchmark tests more robust by @sgrebnov in #8766
• Add parameter to force using iam_role for DynamoDB by @krinart in #8767
• fix: Update Search integration test snapshots by @app/github-actions in #8735
• v1.10.4 release notes by @phillipleblanc in #8790
• Trace metrics export errors by @sgrebnov in #8791
• v1.10.4 SECURITY.md update by @phillipleblanc in #8800
• Add timezone database to Docker image to fix Cayenne acceleration panic by @sgrebnov in #8799
• Upgrade dependencies by @phillipleblanc in #8801
• Fix table_allowlist for table sampling and NSQL by @Jeadie in #8789
• Cayenne primary key on-conflict handling by @lukekim in #8788
• fix: Update benchmark snapshots by @app/github-actions in #8773
• fix: correctly identify deprecated openai_* parameters by @phillipleblanc in #8809
• fix: Update benchmark snapshots by @app/github-actions in #8812
• Use workspace version for cayenne crate by @phillipleblanc in #8811
• Don't CAST strings which breaks push down optimizer by @lukekim in #8810
• fix: Update benchmark snapshots by @app/github-actions in #8815
• Update async-openai to latest revision 4dcd633aad6f - brings fix for openai compatible model providers by @ewgenius in #8816
• Add auth/iam_role_source to DynamoDB connector by @krinart in #8808
• DynamoDB fixes: JSON nesting for Streams, proper batch deletions by @krinart in #8821
• Rough roadmap for 2026-2027 by @lukekim in #8805
• Release notes for v1.11.0-rc1 by @ewgenius in #8786
• Make S3V integration tests prepare_for_aws_tests more robust by @sgrebnov in #8820
• Bump rsa from 0.9.9 to 0.9.10 in the cargo group across 1 directory by @app/dependabot in #8819
• Add timezone database to Release and CUDA Docker images to fix Cayene panic by @sgrebnov in #8832
• fix: UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831
• Update grafana/datadog example dashboards by @krinart in #8833
• Add Dev bird bench as text-to-sql queryset in CI. by @Jeadie in #8753
• Update testoperator scheduler to use release/1.11 branch by @ewgenius in #8829
• Spice Cayenne fixes and test spicepods for Beta & RC by @lukekim in #8787
• testoperator dispatch all bird-bench database variants by @Jeadie in #8835
• feat: Improve column statistics handling with safe access and defaults by @phillipleblanc in #8836
• cluster: mTLS verification by @phillipleblanc in #8837
• fix: 8770: Unsupported ScalarFunctionExpr in ORDER BY by @lukekim in #8838
• Workflow tweaks by @lukekim in #8845
• Cayenne: metadata catalog should respect cayenne_file_path location by @sgrebnov in #8844
• Expand Cayenne feature coverage by @lukekim in #8848
• docs: HA distributed query decisions by @phillipleblanc in #8817
• fix(optimizer): Fix correctness issues in UnionProjectionPushdownOptimizer by @phillipleblanc in #8851
• Pin reqwest to 0.12.24 to fix HuggingFace embedding model download by @ewgenius in #8853
• Fix builds and pin to Ubuntu 22.04 by @lukekim in #8856
• Revert "Fix builds and pin to Ubuntu 22.04" by @lukekim in #8861
• Ensure setup Rust is run by @lukekim in #8862
• fix: Ubuntu 24.04+ renamed libaio1 to libaio1t64 by @lukekim in #8865
• Upgrade to Pulls with Spice v2 by @lukekim in #8866
• Add limit and configuration name to 'testoperator run text-to-sql' by @Jeadie in #8839
• PR check and test optimization by @lukekim in #8868
• Upgrade S3 Vectors SDK and improve test robustness by @lukekim in #8867
• [Testoperator] Query level and improved aggregate level for NSQL by @Jeadie in #8840
• Add docker build for private branches for ghcr.io/spiceai/spiceai-dev by @phillipleblanc in #8873
• Expand the data acceleration round-trip test coverage by @lukekim in #8855
• fix: Provide a better error for improper bucket UDF arguments by @peasee in #8849
• ScyllaDB Data Connector by @lukekim in #8827
• Use tokio-rusqlite for Spice Cayenne SQLite by @lukekim in #8857
• Cayenne: fix FuturesUnordered reentrant drop crash by @sgrebnov in #8863
• Bump github/codeql-action from 4.31.9 to 4.31.10 by @app/dependabot in #8884
• Bump golang.org/x/sys from 0.39.0 to 0.40.0 by @app/dependabot in #8881
• Bump github.com/spiceai/gospice/v8 from 8.0.0 to 8.0.1 by @app/dependabot in #8883
• Bump roaring from 0.11.2 to 0.11.3 by @app/dependabot in #8885
• Bump golang.org/x/mod from 0.31.0 to 0.32.0 by @app/dependabot in #8882
• Bump aws-sdk-s3 from 1.115.0 to 1.119.0 by @app/dependabot in #8887
• Bump libc from 0.2.177 to 0.2.180 by @app/dependabot in #8886
• Bump tokio-util from 0.7.17 to 0.7.18 by @app/dependabot in #8889
• Bump governor from 0.10.2 to 0.10.4 by @app/dependabot in #8888
• fix: flaky test test_concurrent_partition_creation by @phillipleblanc in #8898
• Update Cayenne snapshots for TPC-DS by @lukekim in #8890
• Add more buckets to histogram metrics by @krinart in #8850
• feat: Add HTTP health endpoint for cluster executors by @phillipleblanc in #8899
• feat: Implement model listing functionality for multiple providers by @lukekim in #8901
• feat: Initial HA schedulers distributed query implementation by @phillipleblanc in #8852
• fix: infer executor role from --scheduler-address when --role is omitted by @phillipleblanc in #8903
• Improve install scripts and remove jq dependency by @lukekim in #8847
• Benchmarks: sort PartitionedUnionExec children for deterministic snapshot comparison by @sgrebnov in #8877
• Cayenne: share VortexFileCache across partitions via CayenneContext by @sgrebnov in #8880
• Update ballista to add exponential backoff for scheduler disconnection by @phillipleblanc in #8905
• Configurably add BirdBench evidence to testoperator text-to-SQL. by @Jeadie in #8904
• Helm: Allow command override via values.yaml by @sgrebnov in #8906
• Fix distributed query gRPC message size limit (16MB -> 100MB) by @phillipleblanc in #8900
• OS specific setup actions by @lukekim in #8909
• Cayenne should warn if unable to parse configuration value by @sgrebnov in #8907
• Add snapshots widgets to example dashboard by @krinart in #8910
• Add quality criteria for the features by @krinart in #8897
• Improve Accelerated Datasets section for Grafana/Datadog dashboards by @krinart in #8915
• Use HTTP traceparent in NSQL to support concurrency in 'testoperator run text-to-SQL' by @Jeadie in #8912
• Remove setup for cc from integration_models.yml by @Jeadie in #8917
• Propagate Azure and GCS credentials to executors in cluster mode by @phillipleblanc in #8918
• Cayenne: fix memory growth due to vortex metrics allocation by @sgrebnov in #8908
• fix(caching): Deduplicate refresh requests for JSON array responses by @sgrebnov in #8921
• fix(caching): Return cached data directly for unfiltered queries (SELECT *) by @sgrebnov in #8919
• Correct MinIO path syntax for spiced download by @Jeadie in #8916
• Acceleration snapshots compaction + Improved Snapshots UX by @krinart in #8858
• Change base image from bookworm-slim to trixie-slim by @Jeadie in #8923
• Add testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895
• Fix spicepod dependencies in testoperator by @Jeadie in #8875
• Update copilot instructions for data correctness by @lukekim in #8922
• Add BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926
• fix: add missing feature-gate for AWS Secrets Manager error variant by @phillipleblanc in #8928
• refactor: make ConnectorParams fields public for external connectors by @phillipleblanc in #8929
• fix(caching): SWR refreshes only accessed entry instead of all stale rows by @sgrebnov in #8931
• Cayenne: include cayenne_metadata_dir to known params by @sgrebnov in #8933
• Rename Ingestion Lag chart in example dashboards by @krinart in #8932
• fix(caching): Fix HTTP caching always MISS when projection excludes fetched_at by @sgrebnov in #8930
• Reset expiry after snapshot bootstraping for Caching by @krinart in #8925
• Set use_ssl=false for sccache by @lukekim in #8945
• Hash indexing for Arrow Acceleration by @lukekim in #8924
• [Cayenne] Acceleration snapshots support by @lukekim in #7973
• perf(caching): Non-blocking cache writes on cache miss by @sgrebnov in #8948
• Update NSQL models by @lukekim in #8951
• Hash Index Key verification by @lukekim in #8949
• Add snapshots_creation_policy param by @krinart in #8954
• Remove candle & cudarc from non-models build by @lukekim in #8955
• Acceleration Snapshots API and CLI by @lukekim in #8934
• Ignore test for data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962
• fix: Update benchmark snapshots by @app/github-actions in #8965
• Hash Index secondary index support by @lukekim in #8958
• fix: Support primary key constraints in partitioned DuckDB tables mode by @sgrebnov in #8966
• perf(caching): Batch cache writes by @sgrebnov in #8959
• CI perf optimizations by @lukekim in #8968
• Fix Makefile linting by @Jeadie in #8970
• Fixes in testoperator run text-to-sql. by @Jeadie in #8927
• implement Chat::as_sql for xAI anthropic by @Jeadie in #8957
• Fix duckdb_file_path in search integration test by @Jeadie in #8972
• fix: Update benchmark snapshots by @app/github-actions in #8971
• Maintenance updates to Anthropic API by @Jeadie in #8956
• Add CacheBackend Trait, implement pingora-lru, and add throughput tests by @lukekim in #8080
• fix: Update benchmark snapshots by @app/github-actions in #8974
• fix: Update benchmark snapshots by @app/github-actions in #8975
• Make accelerator shutdown more robust by @lukekim in #8969
• feat(duckdb): Add on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964
• Proper handling for initial snapshot by @krinart in #8911
• fix: Remove --no-default-features from cargo-hack command in features workflow by @phillipleblanc in #8977
• build(deps): bump actions/cache from 5.0.1 to 5.0.2 by @app/dependabot in #8983
• build(deps): bump actions/checkout from 4 to 6 by @app/dependabot in #8982
• build(deps): bump actions/setup-go from 6.1.0 to 6.2.0 by @app/dependabot in #8984
• build(deps): bump github.com/olekukonko/tablewriter from 1.1.2 to 1.1.3 by @app/dependabot in #8979
• build(deps): bump github.com/klauspost/compress from 1.18.2 to 1.18.3 by @app/dependabot in #8980
• Add /v1/queries and Arrow Flight async APIs by @lukekim in #8946
• build(deps): bump Vampire/setup-wsl from 5 to 6 by @app/dependabot in #8981
• fix: Update Search integration test snapshots by @app/github-actions in #8973
• build(deps): bump insta from 1.46.0 to 1.46.1 by @app/dependabot in #8988
• build(deps): bump schemars from 1.1.0 to 1.2.0 by @app/dependabot in #8985
• fix: Update benchmark snapshots by @app/github-actions in #8978
• fix: Data correctness edge cases by @lukekim in #8953
• Correct MinIO path syntax for spiced download (Part 2) by @Jeadie in #8995
• Make .spice/data in search integration tests by @Jeadie in #8992
• fix: Hash index composite keys null values by @lukekim in #9001
• Update Cayenne status to Beta by @lukekim in #9002
• fix: Disable TPC-DS result validation (not yet supported) by @sgrebnov in #9004
• feat: Upgrade to DataFusion v51 and dependencies by @lukekim in #8864
• Improvements for snapshots_creation_policy by @krinart in #9003
• fix(ci): restore cached spicepod-validator binary instead of lookup-only by @phillipleblanc in #9007
• Update version by @krinart in #9010
• Update lock file - https://github.com/spiceai/spiceai/commit/53babbf07ca8c1c7b2e1da42ce58c465d9bc9276/
• fix: Enable Cayenne acceleration snapshots by @lukekim in #9020
• Add TPC-DS integration tests with S3 source and PostgreSQL acceleration by @phillipleblanc in #9006
• fix(tests): fix flaky/slow/failing unit tests by @phillipleblanc in #9009
• fix: Update benchmark snapshots for DF51 upgrade by @app/github-actions in #9008
• fix: add feature gate to rrf TEST_EMBEDDING_MODEL by @phillipleblanc in #9017
• fix: features check by @phillipleblanc in #9014
• URL table support by @lukekim in #9018
• ScyllaDB key filter by @lukekim in #8997
• fix: Schema mismatch when using column projection with HTTP caching by @phillipleblanc in #9021
• Add more tests for HTTP caching with columns selection by @sgrebnov in #9025
• HTTP cache snapshots: default to time_interval and fix snapshots_creation_policy: on_change by @sgrebnov in #9026
• Fix duplicate snapshot creation on startup by @sgrebnov in #9029
• Remove waiting for runtime to be ready before creating snapshot by @krinart in #9033
• Fix snapshot on_change policy to skip when no writes occurred by @sgrebnov in #9028
• Release notes for release release/1.11.0-rc.2 by @krinart in #9016
• ci: use arduino/setup-protoc for official protobuf compiler by @phillipleblanc in #9036
• ci: install unzip on aarch64 runner for arduino/setup-protoc by @phillipleblanc in #9038
• fix: don't fail release if upload to minio fails by @phillipleblanc in #9039
• Improve validation and logging for hash indexes by @lukekim in #9047
• Pin to ubuntu-22.04 by @lukekim in #9068
• Fix broken telemetry for testoperator by @krinart in #9054
• Fix release builds by @lukekim in #9069
• Spice 1.11.0-rc3 release notes by @krinart in #9070
• Update spicepod.schema.json by @app/github-actions in #9071
• Add missing protoc step to setup-cc action by @krinart in #9041
• Fix TLS connection for grpc+tls:// Flight SQL endpoints and add custom CA certificate support by @phillipleblanc in #9073
• Update 1.11.0-rc.3 release notes by @krinart in #9082
• Fix formula_1 and codebase_community in bird-bench by @Jeadie in #9000
• Cayenne S3 Express One Zone improvements by @lukekim in #9015
• Add zlib1g-dev to CI by @lukekim in #9052
• Upgrade Vortex with CASE-WHEN by @lukekim in #9051
• fix: Cayenne CatalogError handling for constraint violations by @lukekim in #9050
• Fix Docker build failing to copy shared libraries due to ldd output parsing by @phillipleblanc in #9058
• feat: Change /v1/sql and FlightSQL to use local execution in cluster mode by @phillipleblanc in #9055
• Remove unmaintained dependencies by @lukekim in #9045
• Enable cayenne + changes stream by @Jeadie in #9053
• feat(cli): add spice query command for async queries REPL by @phillipleblanc in #9057
• Remove unncessary allocations by @lukekim in #9059
• Add dataset_acceleration_size_bytes metric by @krinart in #9062
• Fix tracing of sql_query beneath tool_use::sample_data. by @Jeadie in #9043
• Basic script to run distributed spice by @Jeadie in #9049
• Add integration tests for Acceleration Snapshots by @krinart in #9067
• Upgrade CUDA toolkit to 12.6.0 by @sgrebnov in #9079
• Install required protoc dependency for CUDA build by @sgrebnov in #9080
• feat(cluster): add executor control stream heartbeat by @phillipleblanc in #9072
• feat: Fix async queries API and integrate Ballista shuffle improvements by @lukekim in #9075
• Remove models variant (now default) & Windows builds (use WSL) by @lukekim in #9063
• Cayenne: share upload semaphore across partitions to bound memory growth and optimize I/O by @sgrebnov in #9078
• Snowflake data connector - add snowflake_private_key parameter by @ewgenius in #9085
• Rewrite Go CLI in Rust by @phillipleblanc in #9061
• GCS Data Connector (Alpha) by @lukekim in #9084
• Skip 'latest' Docker tag for pre-release versions by @sgrebnov in #9077
• Add HTTP endpoints for acceleration snapshots API by @phillipleblanc in #9065
• Cayenne: Allow append mode with both primary_key and time_column by @sgrebnov in #9090
• Add Cluster Observability (Metrics+Dashboard) by @phillipleblanc in #9066
• proto for 'CayenneAccelerationExec' by @Jeadie in #9094
• Add 'anthropic-beta' header for structured outputs by @Jeadie in #9093
• Cayenne: refactor write path to use insert_into() as single entry point (part 1) by @sgrebnov in #9088
• Fix testoperator dispatch by @sgrebnov in #9097
• Fix setup-spiced GH action (_models suffix does not exist anymore) by @sgrebnov in #9102
• build(deps): bump github/codeql-action from 4.31.10 to 4.31.11 by @app/dependabot in #9108
• Remove DistributeFileScanOptimizer and UnionProjectionPushdownOptimizer & set target_partitions dynamically based on cluster capacity by @phillipleblanc in #9100
• build(deps): bump zip from 2.4.2 to 6.0.0 by @app/dependabot in #9111
• fix: Preserve query parameter order in HTTP connector to match filter values by @sgrebnov in #9114
• Add PollNow interrupt for Ballista executors to reduce task scheduling latency by @phillipleblanc in #9098
• Revert "GCS Data Connector (Alpha) " by @lukekim in #9084
• Fix stack overflow for CDC batching by @krinart in #9115
• fix: update Ballista fork to include executor timeout fix by @phillipleblanc in #9124
• Properly propagate SIGINT/SIGTERM from CLI to runtime by @krinart in #9127
• fix: Use the same vortex dependency as ballista by @peasee in #9123
• release: Bump version to 1.11.0 for stable - https://github.com/spiceai/spiceai/commit/14d09f8e262008df69ded898ed3bebee08471508/
• Cayenne snapshots with shared metadata by @lukekim in #9118
• Improve error handling for URL tables with Azure URLs by @phillipleblanc in #9129
• Add missing Windows build step for spice CLI in build_and_release workflow by @phillipleblanc in #9143
• Fix install-dev to use debug build path for spice binary by @phillipleblanc in #9142
• fix: CLI builds by @peasee in #9145
• Always create initial snapshots (unless bootstrapped) + when no snapshots exist by @krinart in #9119
• fix(cayenne): Fix upsert with pending deletions causing duplicate PKs by @sgrebnov in #9152
• fix(flightrepl): Add chrono-tz feature to flightrepl for timezone formatting by @sgrebnov in #9153
• fix(delta_lake): Preserve container name in ABFSS URLs for Azure Delta Lake tables by @sgrebnov in #9155
• fix: Make CLI system and asset type detection more robust by @peasee in #9148
• fix: Set query set properly on benchmarks telemetry metrics attributes by @peasee in #9162
• fix: Download _models variant - https://github.com/spiceai/spiceai/commit/27f3058d0007595b02c198755c3b22319032ff30/
• fix: Helm chart image tag - https://github.com/spiceai/spiceai/commit/7405c8df0db4ecce0ed6d4a4d424553d604b9036/
• Revert "Remove models variant (now default) & Windows builds (use WSL) " by @lukekim in #9063
• fix(cli): Several CLI fixes from the Go to Rust migration by @lukekim in #9157

Announcing the release of Spice v1.11.0-rc.2! ⭐

v1.11.0-rc.2 is the second release candidate for advanced test of v1.11. It brings Spice Cayenne to Beta status with acceleration snapshots support, a new ScyllaDB Data Connector, upgrades to DataFusion v51, Arrow 57.2, and iceberg-rust v0.8.0. It includes significant improvements to distributed query, caching, and observability.

What's New in v1.11.0-rc.2​

Spice Cayenne Accelerator Reaches Beta​

Spice Cayenne has been promoted to Beta status with acceleration snapshots support and numerous stability improvements.

Improved Reliability:

• Fixed timezone database issues in Docker images that caused acceleration panics
• Resolved FuturesUnordered reentrant drop crashes
• Fixed memory growth issues related to Vortex metrics allocation
• Metadata catalog now properly respects cayenne_file_path location
• Added warnings for unparseable configuration values

Example configuration with snapshots:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file

DataFusion v51 Upgrade​

Apache DataFusion has been upgraded to v51, bringing significant performance improvements, new SQL features, and enhanced observability.

Performance Improvements:

• Faster CASE Expression Evaluation: Expressions now short-circuit earlier, reuse partial results, and avoid unnecessary scattering, speeding up common ETL patterns
• Better Defaults for Remote Parquet Reads: DataFusion now fetches the last 512KB of Parquet files by default, typically avoiding 2 I/O requests per file
• Faster Parquet Metadata Parsing: Leverages Arrow 57's new thrift metadata parser for up to 4x faster metadata parsing

New SQL Features:

• SQL Pipe Operators: Support for |> syntax for inline transforms
• DESCRIBE <query>: Returns the schema of any query without executing it
• Named Arguments in SQL Functions: PostgreSQL-style param => value syntax for scalar, aggregate, and window functions
• Decimal32/Decimal64 Support: New Arrow types supported including aggregations like SUM, AVG, and MIN/MAX

Example pipe operator:

SELECT * FROM t
|> WHERE a > 10
|> ORDER BY b
|> LIMIT 5;

Improved Observability:

• Improved EXPLAIN ANALYZE Metrics: New metrics including output_bytes, selectivity for filters, reduction_factor for aggregates, and detailed timing breakdowns

Arrow 57.2 Upgrade​

Spice has been upgraded to Apache Arrow Rust 57.2.0, bringing major performance improvements and new capabilities.

Key Features:

• 4x Faster Parquet Metadata Parsing: A rewritten thrift metadata parser delivers up to 4x faster metadata parsing, especially beneficial for low-latency use cases and files with large amounts of metadata
• Parquet Variant Support: Experimental support for reading and writing the new Parquet Variant type for semi-structured data, including shredded variant values
• Parquet Geometry Support: Read and write support for Parquet Geometry types (GEOMETRY and GEOGRAPHY) with GeospatialStatistics
• New arrow-avro Crate: Efficient conversion between Apache Avro and Arrow RecordBatches with projection pushdown and vectorized execution support

iceberg-rust v0.8.0 Upgrade​

Spice has been upgraded to iceberg-rust v0.8.0, bringing improved Iceberg table support.

Key Features:

• V3 Metadata Support: Full support for Iceberg V3 table metadata format
• INSERT INTO Partitioned Tables: DataFusion integration now supports inserting data into partitioned Iceberg tables
• Improved Delete File Handling: Better support for position and equality delete files, including shared delete file loading and caching
• SQL Catalog Updates: Implement update_table and register_table for SQL catalog
• S3 Tables Catalog: Implement update_table for S3 Tables catalog
• Enhanced Arrow Integration: Convert Arrow schema to Iceberg schema with auto-assigned field IDs, _file column support, and Date32 type support

Acceleration Snapshots​

Acceleration snapshots enable point-in-time recovery and data versioning for accelerated datasets. Snapshots capture the state of accelerated data at specific points, allowing for fast bootstrap recovery and rollback capabilities.

Key Feature Improvements in v1.11:

• Flexible Triggers: Configure when snapshots are created based on time intervals or stream batch counts
• Automatic Compaction: Reduce storage overhead by compacting older snapshots (DuckDB only)
• Bootstrap Integration: Snapshots can reset cache expiry on load for seamless recovery (DuckDB with Caching refresh mode)
• Smart Creation Policies: Only create snapshots when data has actually changed

Example configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_dataset
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      snapshots: enabled
      snapshots_trigger: time_interval
      snapshots_trigger_threshold: 1h
      snapshots_creation_policy: on_changed

Snapshots API and CLI: New API endpoints and CLI commands for managing snapshots programmatically. List, create, and restore snapshots directly from the command line or via HTTP.

For more details, refer to the Acceleration Snapshots Documentation.

ScyllaDB Data Connector​

A new data connector for ScyllaDB, the high-performance NoSQL database compatible with Apache Cassandra. Query ScyllaDB tables directly or accelerate them for faster analytics.

Example configuration:

datasets:
  - from: scylladb:my_keyspace.my_table
    name: scylla_data
    acceleration:
      enabled: true
      engine: duckdb

For more details, refer to the ScyllaDB Data Connector Documentation.

Distributed Query Improvements​

mTLS Verification: Cluster communication between scheduler and executors now supports mutual TLS verification for enhanced security.

Credential Propagation: Azure and GCS credentials are now automatically propagated to executors in cluster mode, enabling access to cloud storage across the distributed query cluster.

Improved Resilience:

• Exponential backoff for scheduler disconnection recovery
• Increased gRPC message size limit from 16MB to 100MB for large query plans
• HTTP health endpoint for cluster executors
• Automatic executor role inference when --scheduler-address is provided

For more details, refer to the Distributed Query Documentation.

Caching Acceleration Mode Improvements​

The Caching Acceleration Mode introduced in v1.10.0 has received significant performance optimizations and reliability fixes in this release.

Performance Optimizations:

• Non-blocking Cache Writes: Cache misses no longer block query responses. Data is written to the cache asynchronously after the query returns, reducing query latency for cache miss scenarios.
• Batch Cache Writes: Multiple cache entries are now written in batches rather than individually, significantly improving write throughput for high-volume cache operations.

Reliability Fixes:

• Correct SWR Refresh Behavior: The stale-while-revalidate (SWR) pattern now correctly refreshes only the specific entries that were accessed instead of refreshing all stale rows in the dataset. This prevents unnecessary source queries and reduces load on upstream data sources.
• Deduplicated Refresh Requests: Fixed an issue where JSON array responses could trigger multiple redundant refresh operations. Refresh requests are now properly deduplicated.
• Fixed Cache Hit Detection: Resolved an issue where queries that didn't include fetched_at in their projection would always result in cache misses, even when cached data was available.
• Unfiltered Query Optimization: SELECT * queries without filters now return cached data directly without unnecessary filtering overhead.

For more details, refer to the Caching Acceleration Mode Documentation.

DynamoDB Connector Enhancements​

• Added JSON nesting for DynamoDB Streams
• Proper batch deletion handling

URL Tables​

Query data sources directly via URL in SQL without prior dataset registration. Supports S3, Azure Blob Storage, and HTTP/HTTPS URLs with automatic format detection and partition inference.

Supported Patterns:

• Single files: SELECT * FROM 's3://bucket/data.parquet'
• Directories/prefixes: SELECT * FROM 's3://bucket/data/'
• Glob patterns: SELECT * FROM 's3://bucket/year=*/month=*/data.parquet'

Key Features:

• Automatic file format detection (Parquet, CSV, JSON, etc.)
• Hive-style partition inference with filter pushdown
• Schema inference from files
• Works with both SQL and DataFrame APIs

Example with hive partitioning:

-- Partitions are automatically inferred from paths
SELECT * FROM 's3://bucket/data/' WHERE year = '2024' AND month = '01'

Enable via spicepod.yml:

runtime:
  params:
    url_tables: enabled

Cluster Mode Async Query APIs (experimental)​

New asynchronous query APIs for long-running queries in cluster mode:

• /v1/queries endpoint: Submit queries and retrieve results asynchronously
• Arrow Flight async support: Non-blocking query execution via Arrow Flight protocol

Observability Improvements​

Enhanced Dashboards: Updated Grafana and Datadog example dashboards with:

• Snapshot monitoring widgets
• Improved accelerated datasets section
• Renamed ingestion lag charts for clarity

Additional Histogram Buckets: Added more buckets to histogram metrics for better latency distribution visibility.

For more details, refer to the Monitoring Documentation.

Additional Improvements​

• Model Listing: New functionality to list available models across multiple AI providers
• DuckDB Partitioned Tables: Primary key constraints now supported in partitioned DuckDB table mode
• Post-refresh Sorting: New on_refresh_sort_columns parameter for DuckDB enables data ordering after writes
• Improved Install Scripts: Removed jq dependency and improved cross-platform compatibility
• Better Error Messages: Improved error messaging for bucket UDF arguments and deprecated OpenAI parameters

Contributors​

• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

New ScyllaDB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See ScyllaDB Data Connector Recipe for details.

New SMB Data Connector Recipe: New recipe demonstrating how to use ScyllaDB Data Connector. See SMB Data Connector Recipe for details.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.0-rc.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:v1.11.0-rc.2 image:

docker pull spiceai/spiceai:v1.11.0-rc.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

Spice is available in the AWS Marketplace.

Dependencies​

• DataFusion: Upgraded to v51 (release notes)
• Arrow: Upgraded to v57 (release notes)
• iceberg-rust: Upgraded to v0.8.0 (release notes)

Changelog​

• Add timezone database to Docker image to fix Cayenne acceleration panic by @sgrebnov in #8799
• Upgrade dependencies by @phillipleblanc in #8801
• Fix table_allowlist for table sampling and NSQL by @Jeadie in #8789
• Cayenne primary key on-conflict handling by @lukekim in #8788
• fix: Update benchmark snapshots by @app/github-actions in #8773
• fix: correctly identify deprecated openai_* parameters by @phillipleblanc in #8809
• fix: Update benchmark snapshots by @app/github-actions in #8812
• Use workspace version for cayenne crate by @phillipleblanc in #8811
• Don't CAST strings which breaks push down optimizer by @lukekim in #8810
• fix: Update benchmark snapshots by @app/github-actions in #8815
• Update async-openai to latest revision 4dcd633aad6f - brings fix for openai compatible model providers by @ewgenius in #8816
• Add auth/iam_role_source to DynamoDB connector by @krinart in #8808
• DynamoDB fixes: JSON nesting for Streams, proper batch deletions by @krinart in #8821
• Rough roadmap for 2026-2027 by @lukekim in #8805
• Release notes for v1.11.0-rc1 by @ewgenius in #8786
• Make S3V integration tests prepare_for_aws_tests more robust by @sgrebnov in #8820
• Bump rsa from 0.9.9 to 0.9.10 in the cargo group across 1 directory by @app/dependabot in #8819
• Add timezone database to Release and CUDA Docker images to fix Cayene panic by @sgrebnov in #8832
• fix: UnionProjectionPushdownOptimizer - Schema change during transform_down breaks parent nodes by @phillipleblanc in #8831
• Update grafana/datadog example dashboards by @krinart in #8833
• Add Dev bird bench as text-to-sql queryset in CI. by @Jeadie in #8753
• Update testoperator scheduler to use release/1.11 branch by @ewgenius in #8829
• Spice Cayenne fixes and test spicepods for Beta & RC by @lukekim in #8787
• testoperator dispatch all bird-bench database variants by @Jeadie in #8835
• feat: Improve column statistics handling with safe access and defaults by @phillipleblanc in #8836
• cluster: mTLS verification by @phillipleblanc in #8837
• fix: 8770: Unsupported ScalarFunctionExpr in ORDER BY by @lukekim in #8838
• Workflow tweaks by @lukekim in #8845
• Cayenne: metadata catalog should respect cayenne_file_path location by @sgrebnov in #8844
• Expand Cayenne feature coverage by @lukekim in #8848
• docs: HA distributed query decisions by @phillipleblanc in #8817
• fix(optimizer): Fix correctness issues in UnionProjectionPushdownOptimizer by @phillipleblanc in #8851
• Pin reqwest to 0.12.24 to fix HuggingFace embedding model download by @ewgenius in #8853
• Fix builds and pin to Ubuntu 22.04 by @lukekim in #8856
• Revert "Fix builds and pin to Ubuntu 22.04" by @lukekim in #8861
• Ensure setup Rust is run by @lukekim in #8862
• fix: Ubuntu 24.04+ renamed libaio1 to libaio1t64 by @lukekim in #8865
• Upgrade to Pulls with Spice v2 by @lukekim in #8866
• Add limit and configuration name to 'testoperator run text-to-sql' by @Jeadie in #8839
• PR check and test optimization by @lukekim in #8868
• Upgrade S3 Vectors SDK and improve test robustness by @lukekim in #8867
• [Testoperator] Query level and improved aggregate level for NSQL by @Jeadie in #8840
• Add docker build for private branches for ghcr.io/spiceai/spiceai-dev by @phillipleblanc in #8873
• Expand the data acceleration round-trip test coverage by @lukekim in #8855
• fix: Provide a better error for improper bucket UDF arguments by @peasee in #8849
• ScyllaDB Data Connector by @lukekim in #8827
• Use tokio-rusqlite for Spice Cayenne SQLite by @lukekim in #8857
• Cayenne: fix FuturesUnordered reentrant drop crash by @sgrebnov in #8863
• Bump github/codeql-action from 4.31.9 to 4.31.10 by @app/dependabot in #8884
• Bump golang.org/x/sys from 0.39.0 to 0.40.0 by @app/dependabot in #8881
• Bump github.com/spiceai/gospice/v8 from 8.0.0 to 8.0.1 by @app/dependabot in #8883
• Bump roaring from 0.11.2 to 0.11.3 by @app/dependabot in #8885
• Bump golang.org/x/mod from 0.31.0 to 0.32.0 by @app/dependabot in #8882
• Bump aws-sdk-s3 from 1.115.0 to 1.119.0 by @app/dependabot in #8887
• Bump libc from 0.2.177 to 0.2.180 by @app/dependabot in #8886
• Bump tokio-util from 0.7.17 to 0.7.18 by @app/dependabot in #8889
• Bump governor from 0.10.2 to 0.10.4 by @app/dependabot in #8888
• fix: flaky test test_concurrent_partition_creation by @phillipleblanc in #8898
• Update Cayenne snapshots for TPC-DS by @lukekim in #8890
• Add more buckets to histogram metrics by @krinart in #8850
• feat: Add HTTP health endpoint for cluster executors by @phillipleblanc in #8899
• feat: Implement model listing functionality for multiple providers by @lukekim in #8901
• feat: Initial HA schedulers distributed query implementation by @phillipleblanc in #8852
• fix: infer executor role from --scheduler-address when --role is omitted by @phillipleblanc in #8903
• Improve install scripts and remove jq dependency by @lukekim in #8847
• Benchmarks: sort PartitionedUnionExec children for deterministic snapshot comparison by @sgrebnov in #8877
• Cayenne: share VortexFileCache across partitions via CayenneContext by @sgrebnov in #8880
• Update ballista to add exponential backoff for scheduler disconnection by @phillipleblanc in #8905
• Configurably add BirdBench evidence to testoperator text-to-SQL. by @Jeadie in #8904
• Helm: Allow command override via values.yaml by @sgrebnov in #8906
• Fix distributed query gRPC message size limit (16MB -> 100MB) by @phillipleblanc in #8900
• OS specific setup actions by @lukekim in #8909
• Cayenne should warn if unable to parse configuration value by @sgrebnov in #8907
• Add snapshots widgets to example dashboard by @krinart in #8910
• Add quality criteria for the features by @krinart in #8897
• Improve Accelerated Datasets section for Grafana/Datadog dashboards by @krinart in #8915
• Use HTTP traceparent in NSQL to support concurrency in 'testoperator run text-to-SQL' by @Jeadie in #8912
• Remove setup for cc from integration_models.yml by @Jeadie in #8917
• Propagate Azure and GCS credentials to executors in cluster mode by @phillipleblanc in #8918
• Cayenne: fix memory growth due to vortex metrics allocation by @sgrebnov in #8908
• fix(caching): Deduplicate refresh requests for JSON array responses by @sgrebnov in #8921
• fix(caching): Return cached data directly for unfiltered queries (SELECT *) by @sgrebnov in #8919
• Correct MinIO path syntax for spiced download by @Jeadie in #8916
• Acceleration snapshots compaction + Improved Snapshots UX by @krinart in #8858
• Change base image from bookworm-slim to trixie-slim by @Jeadie in #8923
• Add testoperator run text-to-sql metrics from LogicalPlan by @Jeadie in #8895
• Fix spicepod dependencies in testoperator by @Jeadie in #8875
• Update copilot instructions for data correctness by @lukekim in #8922
• Add BootstrapStatus + Snapshot bootstrapping parallelization by @krinart in #8926
• fix: add missing feature-gate for AWS Secrets Manager error variant by @phillipleblanc in #8928
• refactor: make ConnectorParams fields public for external connectors by @phillipleblanc in #8929
• fix(caching): SWR refreshes only accessed entry instead of all stale rows by @sgrebnov in #8931
• Cayenne: include cayenne_metadata_dir to known params by @sgrebnov in #8933
• Rename Ingestion Lag chart in example dashboards by @krinart in #8932
• fix(caching): Fix HTTP caching always MISS when projection excludes fetched_at by @sgrebnov in #8930
• Reset expiry after snapshot bootstraping for Caching by @krinart in #8925
• Set use_ssl=false for sccache by @lukekim in #8945
• Hash indexing for Arrow Acceleration by @lukekim in #8924
• [Cayenne] Acceleration snapshots support by @lukekim in #7973
• perf(caching): Non-blocking cache writes on cache miss by @sgrebnov in #8948
• Update NSQL models by @lukekim in #8951
• Hash Index Key verification by @lukekim in #8949
• Add snapshots_creation_policy param by @krinart in #8954
• Remove candle & cudarc from non-models build by @lukekim in #8955
• Acceleration Snapshots API and CLI by @lukekim in #8934
• Ignore test for data_components arrow::indexed::test_primary_key_value_matches_batch by @Jeadie in #8962
• fix: Update benchmark snapshots by @app/github-actions in #8965
• Hash Index secondary index support by @lukekim in #8958
• fix: Support primary key constraints in partitioned DuckDB tables mode by @sgrebnov in #8966
• perf(caching): Batch cache writes by @sgrebnov in #8959
• CI perf optimizations by @lukekim in #8968
• Fix Makefile linting by @Jeadie in #8970
• Fixes in testoperator run text-to-sql. by @Jeadie in #8927
• implement Chat::as_sql for xAI anthropic by @Jeadie in #8957
• Fix duckdb_file_path in search integration test by @Jeadie in #8972
• fix: Update benchmark snapshots by @app/github-actions in #8971
• Maintenance updates to Anthropic API by @Jeadie in #8956
• Add CacheBackend Trait, implement pingora-lru, and add throughput tests by @lukekim in #8080
• fix: Update benchmark snapshots by @app/github-actions in #8974
• fix: Update benchmark snapshots by @app/github-actions in #8975
• Make accelerator shutdown more robust by @lukekim in #8969
• feat(duckdb): Add on_refresh_sort_columns for post-write data ordering (initial version) by @sgrebnov in #8964
• Proper handling for initial snapshot by @krinart in #8911
• fix: Remove --no-default-features from cargo-hack command in features workflow by @phillipleblanc in #8977
• build(deps): bump actions/cache from 5.0.1 to 5.0.2 by @app/dependabot in #8983
• build(deps): bump actions/checkout from 4 to 6 by @app/dependabot in #8982
• build(deps): bump actions/setup-go from 6.1.0 to 6.2.0 by @app/dependabot in #8984
• build(deps): bump github.com/olekukonko/tablewriter from 1.1.2 to 1.1.3 by @app/dependabot in #8979
• build(deps): bump github.com/klauspost/compress from 1.18.2 to 1.18.3 by @app/dependabot in #8980
• Add /v1/queries and Arrow Flight async APIs by @lukekim in #8946
• build(deps): bump Vampire/setup-wsl from 5 to 6 by @app/dependabot in #8981
• fix: Update Search integration test snapshots by @app/github-actions in #8973
• build(deps): bump insta from 1.46.0 to 1.46.1 by @app/dependabot in #8988
• build(deps): bump schemars from 1.1.0 to 1.2.0 by @app/dependabot in #8985
• fix: Update benchmark snapshots by @app/github-actions in #8978
• fix: Data correctness edge cases by @lukekim in #8953
• Correct MinIO path syntax for spiced download (Part 2) by @Jeadie in #8995
• Make .spice/data in search integration tests by @Jeadie in #8992
• fix: Hash index composite keys null values by @lukekim in #9001
• Update Cayenne status to Beta by @lukekim in #9002
• fix: Disable TPC-DS result validation (not yet supported) by @sgrebnov in #9004
• feat: Upgrade to DataFusion v51 and dependencies by @lukekim in #8864
• Improvements for snapshots_creation_policy by @krinart in #9003
• fix(ci): restore cached spicepod-validator binary instead of lookup-only by @phillipleblanc in #9007
• Update version by @krinart in #9010

Announcing the release of Spice v1.10.2! 🔥

v1.10.2 introduces Tiered Caching Acceleration with Localpod for multi-layer acceleration architectures, Periodic Acceleration Snapshots with configurable intervals, DynamoDB JSON Nesting for column consolidation, and Kafka/Debezium Batching for faster data ingestion. This release also includes fixes for SQLite accelerator decimal/date handling and real-time status reporting for the /v1/datasets and /v1/models API endpoints.

What's New in v1.10.2​

Tiered Caching with Localpod​

Multi-Layer Acceleration Architecture: The Localpod connector now supports caching refresh mode, enabling tiered acceleration where a persistent cache (e.g., file-mode DuckDB) feeds a fast in-memory cache (e.g., Arrow, memory-mode DuckDB).

Key Features:

• Automatic Cache Propagation: New cache entries automatically propagate from parent to child accelerators
• Warm Startup: Child accelerators initialize from existing parent data on startup, eliminating cold-start latency
• Flexible Tiering: Combine any accelerator engines (DuckDB, SQLite, Cayenne) across tiers

Example spicepod.yaml configuration:

datasets:
  # Parent: persistent file-mode cache
  - from: https://api.example.com
    name: api_cache
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file

  # Child: fast in-memory cache fed by parent
  - from: localpod:api_cache
    name: api_cache_memory
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: arrow
      mode: memory

For more details, refer to the Localpod Data Connector Documentation.

Periodic Acceleration Snapshots​

Configurable Snapshot Intervals: A new snapshots_create_interval parameter enables periodic snapshot creation for accelerated datasets across all refresh modes. This provides better control over snapshot frequency and ensures consistent recovery points for accelerated data.

Example spicepod.yaml configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: my_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: caching
      snapshots: enabled
      params:
        snapshots_create_interval: 60s # Write a snapshot every 60 seconds

For more details, refer to the Data Acceleration Documentation.

DynamoDB JSON Nesting​

Consolidate Columns into JSON: The DynamoDB Data Connector now supports consolidating columns into a single JSON column using the json_object: "*" metadata option. This is useful when only a few columns are needed as discrete fields while the rest can be accessed as nested JSON.

Example spicepod.yaml configuration:

datasets:
  - from: dynamodb:my_table
    name: my_table
    columns:
      - name: PK
      - name: SK
      - name: data_json
        metadata:
          json_object: '*' # Captures all other columns as JSON

Example Output: Given a DynamoDB table with columns PK, SK, name, email, and status, the resulting table schema consolidates all non-specified columns into the data_json column:

For more details, refer to the DynamoDB JSON Nesting Documentation.

Kafka/Debezium Batching​

Faster Data Ingestion: Configure message batching for Kafka and Debezium connectors to improve data ingestion throughput. Batching reduces processing overhead by grouping multiple messages together before insertion.

Key Features:

• Configurable Batch Size: Control the maximum number of records per batch (default: 10,000)
• Configurable Batch Duration: Set the maximum wait time before flushing a partial batch (default: 1s)

Example spicepod.yaml configuration:

datasets:
  - from: debezium:kafka-server.public.my_table
    name: my_table
    params:
      batch_max_size: 10000 # Max records per batch (default: 10000)
      batch_max_duration: 1s # Max wait time per batch (default: 1s)

For more details, refer to the Kafka Data Connector Documentation and Debezium Data Connector Documentation.

Additional Improvements & Bug Fixes​

• Reliability: Fixed SQLite accelerator decimal and date type handling for improved data type accuracy.
• Reliability: Fixed real-time status reporting for /v1/datasets and /v1/models API endpoints.
• Reliability: Fixed Kafka warning when security.protocol is set to PLAINTEXT.

Contributors​

• @phillipleblanc
• @lukekim
• @krinart

Breaking Changes​

No breaking changes.

Cookbook Updates​

New Cayenne Data Accelerator Recipe: New recipe demonstrating how to accelerate a local copy of the taxi trips dataset using Cayenne as the data accelerator engine. See Cayenne Data Accelerator Recipe for details.

New Dataset Partitioning Recipe: New recipe demonstrating how to partition accelerated datasets to improve query performance. See Dataset Partitioning for details.

The Spice Cookbook includes 84 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.10.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.2 image:

docker pull spiceai/spiceai:1.10.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

• Fix kafka warning when security.protocol is set to PLAINTEXT by @krinart in #8587
• fix: SQLite accelerator decimal/date handling by @phillipleblanc in #8606
• feat: Enable localpod with caching mode accelerator for tiered caching by @phillipleblanc in #8621
• Remove the clippy::too_many_lines lint by @phillipleblanc in #8549
• Add snapshot interval for acceleration snapshots by @phillipleblanc in #8627
• Json Nesting for DynamoDB by @krinart in #8623
• Implement batching for Kafka/Debezium + null Decimal handling by @krinart in #8622
• fix: Status field in /v1/datasets & /v1/models by @lukekim in #8633

Announcing the release of Spice v1.8.1! 🚀

Spice v1.8.1 is a patch release that adds Acceleration Snapshots Indexes, and includes a number of bug fixes and performance improvements.

What's New in v1.8.1​

Acceleration Snapshot Indexes​

• Management of Acceleration Snapshots has been improved by adopting an Iceberg-inspired metadata.json, which now encodes pointer IDs, schema serialization, and robust checksum and size, which is validate before loading the snapshot.

• Acceleration Snapshot Metrics: The following metrics are now available for Acceleration Snapshots:

• dataset_acceleration_snapshot_bootstrap_duration_ms: The time it took the runtime to download the snapshot - only emitted when it initially downloads the snapshot.

• dataset_acceleration_snapshot_bootstrap_bytes: The number of bytes downloaded to bootstrap the acceleration from the snapshot.

• dataset_acceleration_snapshot_bootstrap_checksum: The checksum of the snapshot used to bootstrap the acceleration.

• dataset_acceleration_snapshot_failure_count: Number of failures encountered when writing a new snapshot at the end of the refresh cycle. A snapshot failure does not prevent the refresh from completing.

• dataset_acceleration_snapshot_write_timestamp: Unix timestamp in seconds when the last snapshot was completed.

• dataset_acceleration_snapshot_write_duration_ms: The time it took to write the snapshot to object storage.

• dataset_acceleration_snapshot_write_bytes: The number of bytes written on the last snapshot write.

• dataset_acceleration_snapshot_write_checksum: The SHA256 checksum of the last snapshot write.

To learn more, see the Acceleration Snapshots Documentation and the Metrics Documentation.

Improved Regular Expression for DuckDB acceleration​

Regular expression support has been expanded when using DuckDB acceleration for functions like regexp-like and regexp_match.

For more details, refer to the SQL Reference for the list of available regular expression functions.

Additional Improvements & Bugfixes​

• Reliability: Resolved an issue with partitioning on empty partition sets.
• Validation: Added better validation for incorrectly configured Spicepods.
• Reliability: Fixed partition_by accelerations when a projection is applied on empty partition sets.
• Performance: Ensured ListingTable partitions are pruned when filters are not used.
• Performance: Don't download acceleration snapshots if the acceleration is already present.
• Performance: Refactored some blocking I/O and synchronization in the async codebase by moving operations to tokio::task::spawn_blocking, replacing blocking locks with async-friendly variants.
• Bugfix: Nullable fields are now supported for S3 Vectors index columns.

Contributors​

• @Jeadie
• @mach-kernel
• @lukekim
• @phillipleblanc
• @sgrebnov
• @kczimm
• @krinart
• @peasee
• @ewgenius

Breaking Changes​

No breaking changes.

Cookbook Updates​

• New Accelerated Snapshots Recipe - The recipe shows how to bootstrap DuckDB accelerations from object storage to skip cold starts.

The Spice Cookbook includes 81 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.8.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.8.1 image:

docker pull spiceai/spiceai:1.8.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

• Remove println in datafusion by @phillipleblanc in #7461
• fix: Ensure ListingTable partitions are pruned when filters are not used by @peasee in #7471
• Create runtime-secrets crate by @phillipleblanc in #7474
• Create runtime-parameters crate by @phillipleblanc in #7475
• Don't download the snapshot if the acceleration is present by @phillipleblanc in #7477
• Add support for S3 dataset params by @phillipleblanc in #7476
• Add better snapshot validation for incorrectly configured spicepods by @phillipleblanc in #7487
• Move blocking/sync I/O to spawn blocking by @lukekim in #7462
• Validate spicepod file exists before running tests by @lukekim in #7492
• Make snapshot reading/writing more robust with Iceberg-like metadata.json by @phillipleblanc in #7486
• Create runtime-request-context crate by @Jeadie in #7459
• Two minor fixes for AI udf tests by @krinart in #7503
• Add model response timeout for ai udf tests by @krinart in #7504
• Add sccache for build test operator by @lukekim in #7515
• Fix partition_by accelerations when a projection is applied on empty partition sets by @phillipleblanc in #7526
• Nullable fields for index columns by @Jeadie in #7523

Announcing the release of Spice v1.8.0! 🧊

Spice v1.8.0 delivers major advances in data writes, scalable vector search, and now in preview—managed acceleration snapshots for fast cold starts. This release introduces write support for Iceberg tables using standard SQL INSERT INTO, partitioned S3 Vector indexes for petabyte-scale vector search, and preview of the AI SQL function for direct LLM integration in SQL. Additional improvements include improved reliability, and the v3.0.3 release of the Spice.js Node.js SDK.

What's New in v1.8.0​

Iceberg Table Write Support (Preview)​

Append Data to Iceberg Tables with SQL INSERT INTO: Spice now supports writing to Iceberg tables and catalogs using standard SQL INSERT INTO statements. This enables data ingestion, transformation, and pipeline use cases—no Spark or external writer required.

• Append-only: Initial version targets appends; no overwrite or delete.
• Schema validation: Inserted data must match the target table schema.
• Secure by default: Writes are only enabled for datasets or catalogs explicitly marked with access: read_write.

Example Spicepod configuration:

catalogs:
  - from: iceberg:https://glue.ap-northeast-3.amazonaws.com/iceberg/v1/catalogs/111111/namespaces
    name: ice
    access: read_write

datasets:
  - from: iceberg:https://iceberg-catalog-host.com/v1/namespaces/my_namespace/tables/my_table
    name: iceberg_table
    access: read_write

Example SQL usage:

-- Insert from another table
INSERT INTO iceberg_table
SELECT * FROM existing_table;

-- Insert with values
INSERT INTO iceberg_table (id, name, amount)
VALUES (1, 'John', 100.0), (2, 'Jane', 200.0);

-- Insert into catalog table
INSERT INTO ice.sales.transactions
VALUES (1001, '2025-01-15', 299.99, 'completed');

Note: Only Iceberg datasets and catalogs with access: read_write support writes. Internal Spice tables and other connectors remain read-only.

Learn more in the Iceberg Data Connector documentation.

Acceleration Snapshots for Fast Cold Starts (Preview)​

Bootstrap Managed Accelerations from Object Storage: Spice now supports managed acceleration snapshots in preview, enabling datasets accelerated with file-based engines (DuckDB or SQLite) to bootstrap from a snapshot stored in object storage (such as S3) if the local acceleration file does not exist on startup. This dramatically reduces cold start times and enables ephemeral storage for accelerations with persistent recovery.

Key features:

• Rapid readiness: Datasets can become ready in seconds by downloading a pre-built snapshot, skipping lengthy initial acceleration.
• Hive-style partitioning: Snapshots are organized by month, day, and dataset for easy retention and management.
• Flexible bootstrapping: Configurable fallback and retry behavior if a snapshot is missing or corrupted.

Example Spicepod configuration:

snapshots:
  enabled: true
  location: s3://some_bucket/some_folder/ # Folder for storing snapshots
  bootstrap_on_failure_behavior: warn # Options: warn, retry, fallback
  params:
    s3_auth: iam_role # All S3 dataset params accepted here

datasets:
  - from: s3://some_bucket/some_table/
    name: some_table
    params:
      file_format: parquet
      s3_auth: iam_role
    acceleration:
      enabled: true
      snapshots: enabled # Options: enabled, disabled, bootstrap_only, create_only
      engine: duckdb
      mode: file
      params:
        duckdb_file: /nvme/some_table.db

How it works:

• On startup, if the acceleration file does not exist, Spice checks the snapshot location for the latest snapshot and downloads it.
• Snapshots are stored as: s3://some_bucket/some_folder/month=2025-09/day=2025-09-30/dataset=some_table/some_table_<timestamp>.db
• If no snapshot is found, a new acceleration file is created as usual.
• Snapshots are written after each refresh (unless configured otherwise).

Supported snapshot modes:

• enabled: Download and write snapshots.
• bootstrap_only: Only download on startup, do not write new snapshots.
• create_only: Only write snapshots, do not download on startup.
• disabled: No snapshotting.

Note: This feature is only supported for file-based accelerations (DuckDB or SQLite) with dedicated files.

Why use acceleration snapshots?

• Faster cold starts: Skip waiting for full acceleration on startup.
• Ephemeral storage: Use fast local disks (e.g., NVMe) for acceleration, with persistent recovery from object storage.
• Disaster recovery: Recover from federated source outages by bootstrapping from the latest snapshot.

Partitioned S3 Vector Indexes​

Efficient, Scalable Vector Search with Partitioning: Spice now supports partitioning Amazon S3 Vector indexes and scatter-gather queries using a partition_by expression in the dataset vector engine configuration. Partitioned indexes enable faster ingestion, lower query latency, and scale to billions of vectors.

Example Spicepod configuration:

datasets:
  - name: reviews
    vectors:
      enabled: true
      engine: s3_vectors
      params:
        s3_vectors_bucket: my-bucket
        s3_vectors_index: base-embeddings
      partition_by:
        - 'bucket(50, PULocationID)'
    columns:
      - name: body
        embeddings:
          from: bedrock_titan
      - name: title
        embeddings:
          from: bedrock_titan

See the Amazon S3 Vectors documentation for details.

AI SQL function for LLM Integration (Preview)​

LLMs Directly In SQL: A new asynchronous ai SQL function enables direct calls to LLMs from SQL queries for text generation, translation, classification, and more. This feature is released in preview and supports both default and model-specific invocation.

Example Spicepod model configuration:

models:
  - name: gpt-4o
    from: openai:gpt-4o
    params:
      openai_api_key: ${secrets:openai_key}

Example SQL usage:

-- basic usage with default model
SELECT ai('hi, this prompt is directly from SQL.');

-- basic usage with specified model
SELECT ai('hi, this prompt is directly from SQL.', 'gpt-4o');

-- Using row data as input to the prompt
SELECT ai(concat_ws(' ', 'Categorize the zone', Zone, 'in a single word. Only return the word.')) AS category
FROM taxi_zones
LIMIT 10;

Learn more in the SQL Reference AI documentation.

Remote Endpoint Support for Spice CLI​

Run CLI Commands Remotely: The Spice CLI now supports connecting to remote Spice instances, enabling you to run spice sql, spice search, and spice chat commands from your local machine against a remote spiced daemon or to Spice Cloud. Previously, these commands required running on the same machine as the runtime. Now, new flags allow remote execution:

• --cloud: Connect to a Spice Cloud instance (requires --api-key).
• --endpoint <endpoint>: Connect to a remote Spice instance via HTTP or Arrow Flight SQL (gRPC). Supports http://, https://, grpc://, or grpc+tls:// schemes.

Examples:

# Run SQL queries against a remote Spice instance
spice sql --endpoint http://remote-host:8090

# Use Spice Cloud for chat or search
spice chat --cloud --api-key <your-api-key>
spice search --cloud --api-key <your-api-key>

Supported CLI Commands:

• spice sql --cloud / spice sql --endpoint <endpoint>
• spice search --cloud / spice search --endpoint <endpoint>
• spice chat --cloud / spice chat --endpoint <endpoint>

Additional Flags:

• --headers: Pass custom HTTP headers to the remote endpoint.
• --tls-root-certificate-file: Specify a root certificate for TLS verification.
• --user-agent: Set a custom user agent for requests.

For more details, see the Spice CLI Command Reference.

Spice.js v3.0.3 SDK​

Spice.js v3.0.3 Released: The official Spice.ai Node.js/JavaScript SDK has been updated to v3.0.3, bringing cross-platform support, new APIs, and improved reliability for both Node.js and browser environments.

• Modern Query Methods: Use sql(), sqlJson(), and nsql() for flexible querying, streaming, and natural language to SQL.
• Browser Support: SDK now works in browsers and web applications, automatically selecting the optimal transport (gRPC or HTTP).
• Health Checks & Dataset Refresh: Easily monitor Spice runtime health and trigger dataset refreshes on demand.
• Automatic HTTP Fallback: If gRPC/Flight is unavailable, the SDK falls back to HTTP automatically.
• Migration Guidance: v3 requires Node.js 20+, uses camelCase parameters, and introduces a new package structure.

Example usage:

import { SpiceClient } from '@spiceai/spice'

const client = new SpiceClient(apiKey)
const table = await client.sql('SELECT * FROM my_table LIMIT 10')
console.table(table.toArray())

See Spice.js SDK documentation for full details, migration tips, and advanced usage.

Additional Improvements​

• Reliability: Improved logging, error handling, and network readiness checks across connectors (Iceberg, Databricks, etc.).
• Vector search durability and scale: Refined logging, stricter default limits, safeguards against index-only scans and duplicate results, and always-accessible metadata for robust queryability at scale.
• Cache behavior: Tightened cache logic for modification queries.
• Full-Text Search: FTS metadata columns now usable in projections; max search results increased to 1000.
• RRF Hybrid Search: Reciprocal Rank Fusion (RRF) UDTF enhancements for advanced hybrid search scenarios.

Contributors​

• @Jeadie
• @mach-kernel
• @lukekim
• @phillipleblanc
• @sgrebnov
• @kczimm
• @krinart
• @peasee
• @ewgenius

Breaking Changes​

This release introduces two breaking changes associated with the search observability and tooling.

Firstly, the document_similarity tool has been renamed to search. This has the equivalent change to tracing of these tool calls:

## Old: v1.7.1
>> spice trace tool_use::document_similarity
>> curl -XPOST http://localhost:8090/v1/tools/document_similarity \
  -d '{
    "datasets": ["my_tbl"],
    "text": "Welcome to another Spice release"
  }'

## New: v1.8.0
>> spice trace tool_use::search
>> curl -XPOST http://localhost:8090/v1/tools/search \
  -d '{
    "datasets": ["my_tbl"],
    "text": "Welcome to another Spice release"
  }'

Secondly, the vector_search task in runtime.task_history has been renamed to search.

Cookbook Updates​

• Added new AI SQL function recipe for invoking LLMs within SQL queries.
• Updated Iceberg Catalog Connector recipe for Iceberg Writes.
• Updated Spice.js JavaScript (Node.js) SDK for v3.0.3 with examples and v2 to v3 migration guide.

The Spice Cookbook now includes 80 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.8.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.8.0 image:

docker pull spiceai/spiceai:1.8.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Dependencies​

• iceberg-rust: Upgraded to v0.7.0-rc.1
• mimalloc: Upgraded from 0.1.47 to 0.1.48
• azure_core: Upgraded from 0.27.0 to 0.28.0
• Jimver/cuda-toolkit: Upgraded from 0.2.27 to 0.2.28

Changelog​

• Add #[cfg(feature = "postgres")] to acceleration refresh tests by @Jeadie in #7241
• fix: Update benchmark snapshots by @github-actions[bot] in #7267
• fix: Update benchmark snapshots by @github-actions[bot] in #7268
• fix: Update benchmark snapshots by @github-actions[bot] in #7269
• Update the tpch benchmark snapshots for: federated/databricks[sql_warehouse].yaml by @github-actions[bot] in #7270
• EmbeddingInput cache keys to include model name by @mach-kernel in #7275
• ensure FTS metadata columns can be used in projection by @Jeadie in #7282
• Use 8-core runners for Windows CUDA builds by @sgrebnov in #7284
• Make search test more robust by @krinart in #7283
• Post-release housekeeping by @sgrebnov in #7272
• fix: Use median cached response duration for test search cache by @peasee in #7286
• Bump dirs from 5.0.1 to 6.0.0 by @dependabot[bot] in #7244
• Bump indexmap from 2.11.0 to 2.11.4 by @dependabot[bot] in #7248
• Fix JOIN level filters not having columns in schema by @Jeadie in #7287
• use SessionContext::new_empty in RRF by @kczimm in #7291
• Use rust:1.89-slim-bookworm for build, more places to bump rust version by @sgrebnov in #7293
• Update openapi.json by @github-actions[bot] in #7290
• Enable chunking in SearchIndex by @Jeadie in #7143
• Add index name and remove duplicate records string to S3 Vectors log by @lukekim in #7260
• Use file-based fts index by @Jeadie in #7024
• Remove 'PostApplyCandidateGeneration' by @Jeadie in #7288
• RRF: Rank and recency boosting by @mach-kernel in #7294
• Update ROADMAP.md by removing v1.7 milestone by @sgrebnov in #7297
• RRF: Preserve base ranking when results differ -> FULL OUTER JOIN does not produce time column by @mach-kernel in #7300
• chore: remove unused Dataset methods by @kczimm in #7295
• fix removing embedding column by @Jeadie in #7302
• fix: Add feature flag for using object store in spicepod by @peasee in #7303
• Upgrade to iceberg-rust v0.7.0-rc1 by @sgrebnov in #7296
• Enable DML Update SQL operations for datasets configured as access: read_write by @sgrebnov in #7304
• Create and parse partitioned S3 vector index names by @kczimm in #7198
• RRF: Fix decay for disjoint result sets by @mach-kernel in #7305
• RRF: Project top scores, do not yield duplicate results by @mach-kernel in #7306
• RRF: Case sensitive column/ident handling by @mach-kernel in #7309 in #7309
• For vector_search, use a default limit of 1000 if no limit specified by @lukekim in #7311
• Don’t cache modification queries (DDL, DML, COPY) by @sgrebnov in #7316
• Fix Anthropic model regex and add validation tests by @ewgenius in #7319
• Enhancement: Implement before/after/lag metrics for acceleration refresh by @krinart in #7310
• Refactor chat model health check to lower tokens usage for reasoning models by @ewgenius in #7317
• Add support for writing into Iceberg tables by @sgrebnov in #7315
• Fix lint warnings by @lukekim in #7327
• Use logical plan in SearchQueryProvider by @Jeadie in #7314
• FTS max search results 100 -> 1000 by @Jeadie in #7331
• Improve Databricks SQL Warehouse Error Handling by @sgrebnov in #7332
• Use spicepod embedding model name for model_name() by @Jeadie in #7333
• Handle async queries for Databricks SQL Warehouse API by @phillipleblanc in #7335
• Enable DML (INSERT INTO) operations for catalogs configured as access:read_write by @sgrebnov in #7330
• Bump regex from 1.11.2 to 1.11.3 by @dependabot[bot] in #7336
• Update qa_analytics.csv with 1.7.0 release data by @sgrebnov in #7337
• RRF: Fix ident resolution for struct fields, autohashed join key for varying types by @mach-kernel in #7339
• v1.7.1 release notes by @kczimm in #7348
• Bump Jimver/cuda-toolkit from 0.2.27 to 0.2.28 by @dependabot[bot] in #7343
• Add support for writing into Glue (Iceberg) tables and catalogs by @sgrebnov in #7355
• Bump mimalloc from 0.1.47 to 0.1.48 by @dependabot[bot] in #7342
• Add ai async UDF by @lukekim in #7328
• Use self-hosted and spiceai-macos runners for workflows where possible by @lukekim in #7371
• Several updates for improved search testing by @Jeadie in #7358
• Update supported versions in SECURITY.md by @Jeadie in #7377
• 1.7.1 release analytics by @mach-kernel in #7380
• Add acceleration_file_path helper and refactor spice_sys to use Snafu errors by @phillipleblanc in #7376 in #7376
• fix: Update benchmark snapshots by @github-actions[bot] in #7353
• Robust search test by @Jeadie in #7381
• [bug] Fix ai UDF bug of mismatched column length by @lukekim in #7383
• Add OpenOption to spice_sys acceleration tables by @phillipleblanc in #7379
• Add new snapshots Spicepod configuration by @phillipleblanc in #7384
• Update naming of tool_use::document_similarity and vector_search spans by @Jeadie in #7273
• fix: Update benchmark snapshots by @github-actions[bot] in #7354
• Make ai UDF a models only feature by @lukekim in #7387
• Add new runtime_acceleration crate; create SnapshotManager; implement SnapshotManager::download_latest_snapshot by @phillipleblanc in #7386
• Refactor 'VectorScanTableProvider' to use just 'VectorIndex::list_table_provider' by @Jeadie in #7318
• Fix embed logs by @Jeadie in #7382
• Enable spicepod dependencies in testoperator by @Jeadie in #7334
• ai UDF security and performance optimizations by @lukekim in #7392
• Wire up the snapshot download on dataset startup by @phillipleblanc in #7389
• Implement initial snapshot creation logic in SnapshotManager by @phillipleblanc in #7391
• Make tool_use::table_schema output model-friendly by @krinart in #7393
• Fix minor lint warnings by @lukekim in #7395
• Enable metadata columns in document-based object store datasets by @Jeadie in #7397
• Core dependencies of financebench by @Jeadie in #7400
• Add S3vector variant to financebench by @Jeadie in #7399
• Set PostgreSQL unsupported_spice_action=string by default by @lukekim in #7398
• Use non-blocking connection check for verify_ns_lookup_and_tcp_connect by @phillipleblanc in #7401
• Bump moka from 0.12.10 to 0.12.11 by @dependabot[bot] in #7340
• Bump tokio-postgres from 0.7.13 to 0.7.14 by @dependabot[bot] in #7344
• Bump azure_core from 0.27.0 to 0.28.0 by @dependabot[bot] in #7338
• Forbid INSERT OVERWRITE DML operations by @sgrebnov in #7402
• Make database connection pool sizes consistent by @lukekim in #7403
• Disable vector index only scans by @Jeadie in #7405
• Make CLI --endpoint and --cloud args & table output consistent by @lukekim in #7396
• Write new snapshots at the end of an accelerated refresh by @phillipleblanc in #7410
• Read and write partitioned S3 indexes by @kczimm in #7313
• Fix partial data writes in Iceberg data connector by @sgrebnov in #7411
• Remove nix by @phillipleblanc in #7414
• Use DataFusion JoinSetTracer for async context propagation by @lukekim in #7416
• Implement cache invalidation for DML (INSERT INTO) operations by @sgrebnov in #7394
• Make cleanup disk GH action; use in integration tests by @Jeadie in #7418
• Move S3Vector to 'search' crate by @Jeadie in #7373
• Use LogicalPlan builder API for LogicalPlans by @Jeadie in #7408
• Use hive-style partitioned paths for DB snapshots by @phillipleblanc in #7422
• Limit results from SearchIndex::query_table_provider by @Jeadie in #7421
• Delay initial readiness if snapshots are enabled with an append-mode refresh by @phillipleblanc in #7425
• Disable snapshots by default by @phillipleblanc in #7426
• Rewrite ChunkedNonIndexVectorGeneration to use LogicalPlanBuilder (instead of string formatting) by @Jeadie in #7413
• Fix for search field as metadata for chunked search indexes by @Jeadie in #7429
• Add feature is currently in preview warning for read_write access mode by @sgrebnov in #7440
• Add feature is currently in preview warning for snapshots by @sgrebnov in #7442
• Fix tracing so that ai_completions are parented under sql_query by @lukekim in #7415
• Disable acceleration refresh metrics by @krinart in #7450
• Enable snapshot acceleration by default by @phillipleblanc in #7451
• fix: partition name validation by @kczimm in #7452