Skip to main content

Spice v1.9.0-rc.2 (Nov 11, 2025)

· 32 min read
Sergei Grebnov
Sergei Grebnov
Senior Software Engineer at Spice AI

Announcing the release of Spice v1.9.0-rc.2! 🌶

This is the second release candidate for v1.9.0, which introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0-rc.2 also upgrades to DataFusion v50 and DuckDB v1.4.1 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, includes significant DynamoDB and DuckDB accelerator improvements, expands the HTTP data connector to support endpoints as tables, and delivers many security and reliability improvements.

What's New in v1.9.0-rc.2​

Cayenne Data Accelerator (Beta)​

Introducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.

Cayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:

Key Features:

  • SQLite + Vortex Architecture: All metadata is stored in SQLite tables with standard SQL transactions, while data lives in Vortex's compressed, chunked columnar format designed for zero-copy access and efficient scanning.
  • Simplified Operations: No complex file hierarchies, no JSON/Avro metadata files, no separate catalog servers—just SQL tables and Vortex data files. The entire metadata schema is intentionally simple for maximum reliability.
  • Fast Metadata Access: Single SQL query retrieves all metadata needed for query planning—no multiple round trips to storage, no S3 throttling, no reconstruction of metadata state from scattered files.
  • Efficient Small Changes: Dramatically reduces small file proliferation. Snapshots are just rows in SQLite tables, not new files on disk. Supports millions of snapshots without performance degradation.
  • High Concurrency: Changes consist of two steps: stage Vortex files (if any), then run a single SQL transaction. Much faster conflict resolution and support for many more concurrent updates than file-based formats.
  • Advanced Data Lifecycle: Full ACID transactions, delete support, and retention SQL execution on refresh commit.

Example Spicepod.yml configuration:

datasets:
  - from: s3:my_table
    name: accelerated_data_30d
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      refresh_mode: append
      retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'

Note, the Cayenne Data Accelerator is in Beta with limitations.

For more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.

Multi-Node Distributed Query (Preview)​

Apache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0-rc.2.

Architecture:

A distributed Spice cluster consists of:

  • Scheduler: Responsible for distributed query planning and work queue management for the executor fleet
  • Executors: One or more nodes responsible for running physical query plans

Getting Started:

Start a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:

# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051

Start one or more executors configured with the scheduler's flight URI:

# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051

Query Execution:

Queries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:

EXPLAIN SELECT count(id) FROM my_dataset;

Current Limitations:

  • Accelerated datasets are currently not supported. This feature is designed for querying partitioned data lake formats (Parquet, Delta Lake, Iceberg, etc.)
  • The feature is in preview and may have stability or performance limitations
  • Specific acceleration support is planned for future releases

DataFusion v50 Upgrade​

Spice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:

Performance Improvements 🚀:

  • Dynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.

  • Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.

Apache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.

Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.

See the Apache DataFusion 50.0.0 Release for more details.

DuckDB v1.4.1 Upgrade and Accelerator Improvements​

DuckDB v1.4.1: DuckDB has been upgraded to v1.4.1, which includes several performance optimizations.

Composite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      indexes:
        '(region, product_id)': enabled

Performance example with composite index on 7.5M rows:

SELECT * FROM sales WHERE region = 'US' AND product_id = 12345;

-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index

DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://sales_data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        query_federation: disabled # Required currently for intermediate materialization
      indexes:
        '(region, product_id)': enabled

Performance example:

-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;

-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster

The optimizer automatically rewrites the query to:

WITH _intermediate_materialize AS MATERIALIZED (
  SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;

Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.

Retention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.

UTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.

Example Spicepod.yml configuration:

datasets:
  - from: s3://my_bucket/large_table/
    name: partitioned_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      retention:
        sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'

HTTP Data Connector​

  • Querying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.

  • Query HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table

  • Configurable retry logic, timeouts, and POST request support for more complex API interactions

Example Spicepod.yml configuration:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    params:
      file_format: json
      max_retries: 3
      client_timeout: 10s

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;

If a request_body is supplied it will be posted to the endpoint:

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{"name": "michael"}'
LIMIT 10;

HTTP endpoints can be accelerated using refresh_sql:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    acceleration:
      enabled: true
      refresh_mode: full
      refresh_sql: |
        SELECT request_path, request_query, content 
        FROM tvmaze
        request_path = '/search/people'
          AND request_query IN ('q=michael', 'q=luke')

DynamoDB Data Connector Improvements​

Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: ddb_data
    params:
      scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows

S3 Versioning Support​

Atomic Range Reads for Versioned Files: Spice now supports S3 Versioning for all connectors using object-store (S3, Delta Lake, etc.), ensuring range reads over versioned files are atomically correct. When S3 versioning is enabled, Spice automatically tracks version IDs during file discovery and uses them for all subsequent range reads, preventing inconsistencies from concurrent file modifications.

Current limitations:

  • Multi-file connections (e.g., partitioned datasets) do not yet support version tracking across all files
  • Version tracking is automatic when S3 versioning is enabled on the bucket

Search & Embeddings Enhancements​

Full-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.

Multi-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.

Vector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.

Example Spicepod.yml configuration:

views:
  - name: aggregated_reviews
    sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
    embeddings:
      - column: review_text
        model: openai:text-embedding-3-small

Dedicated Query Thread Pool (Now Enabled by Default)​

Dedicated Query Thread Pool: Query execution and accelerated refreshes now run on their own dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health checks fast and avoiding unnecessary Kubernetes pod restarts under load.

This feature was opt-in in previous releases and is now enabled by default in v1.9.0-rc.2. To disable it and revert to the previous behavior, add the following spicepod.yaml configuration:

runtime:
  params:
    dedicated_thread_pool: none

Query Performance Optimizations​

Stale-While-Revalidate Cache Control: Query results now support "stale-while-revalidate" cache control, allowing stale cached data to be served immediately while asynchronously refreshing the cache entry in the background. This improves response times for frequently-accessed queries while maintaining data freshness. Requires cache key type to be set to "sql (raw)" for proper operation.

Optimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.

Large RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.

Query Result Cache: Stale-While-Revalidate​

HTTP Cache-Control Support: The query result cache now supports the stale-while-revalidate Cache-Control directive, enabling faster response times by serving stale cached results immediately while asynchronously refreshing the cache in the background. This feature is particularly useful for applications that can tolerate slightly stale data in exchange for improved performance.

How it works:

When a cache entry is stale but within the stale-while-revalidate window, Spice will:

  1. Immediately return the stale cached result to the client
  2. Asynchronously re-execute the query in the background to refresh the cache
  3. Future requests will use the refreshed data

Configuration:

Use the Cache-Control HTTP header with the stale-while-revalidate directive:

Cache-Control: max-age=300, stale-while-revalidate=60

This configuration caches results for 5 minutes (300 seconds), and allows serving stale results for an additional 60 seconds while refreshing in the background.

Requirements:

  • Must use plan or raw SQL cache keys (set cache_key_type to sql or plan in results_caching configuration)
  • Background revalidation re-executes queries through the normal query path
  • Timestamp tracking automatically determines cache entry age for staleness checks

Example configuration via HTTP header:

GET /v1/sql
Cache-Control: max-age=600, stale-while-revalidate=120
X-Cache-Key-Type: sql

This feature improves application responsiveness while ensuring data freshness through background updates.

Security & Reliability Improvements​

Enhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.

ODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.

CLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.

Oracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.

AWS Authentication Improvements​

Improved Credential Retry Logic: AWS SDK credential initialization has been significantly improved with more robust retry logic and better error handling. The system now automatically retries transient credential resolution failures using Fibonacci backoff, allowing Spice to tolerate extended AWS outages (up to ~48 hours) without manual intervention.

Key features:

  • Automatic retry with backoff: Implements Fibonacci backoff for transient credential failures (network issues, temporary AWS service disruptions)
  • Configurable retry limits: Supports up to 300 retry attempts with a maximum retry interval of 600 seconds
  • Better error handling: Distinguishes between retryable errors (connector errors) and non-retryable errors (misconfiguration)
  • Unauthenticated access support: Properly supports unauthenticated access to public S3 buckets without requiring credentials
  • Improved error messages: Provides detailed logging with attempt numbers, retry intervals, and error context for better troubleshooting

The improvements ensure more reliable AWS service integration, particularly in environments with intermittent network connectivity or during AWS service degradations.

Observability & Tracing​

DataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.

AI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.

Git Data Connector (Alpha)​

Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.

Example Spicepod.yml configuration:

datasets:
  - from: git:https://github.com/myorg/myrepo
    name: git_metrics
    params:
      file_format: csv

For more details, refer to the Git Data Connector Documentation.

Spice Java SDK 0.4.0​

The Spice Java SDK have been upgraded with support configurable Arrow memory limit: spice-java v0.4.0

SpiceClient client = SpiceClient.builder()
    .withArrowMemoryLimitMB(1024) // 1GB limit
    .build();

CLI Improvements​

Install Specific Versions: The spice install command now supports installing specific versions of the Spice runtime and CLI. This enables easy version management, downgrading, or installation of specific releases for testing or compatibility requirements.

Usage:

# Install a specific version
spice install v1.8.3

# Install a specific version with AI flavor
spice install v1.8.3 ai

# Install latest version (existing behavior)
spice install
spice install ai

Note: Homebrew installations require manual version management via brew install spiceai/spiceai/spice@<version>.

Persistent Query History: The Spice CLI REPL (SQL, search, and chat interfaces) now persists command history to ~/.spice/query_history.txt, making your query history available across sessions. The history file is automatically created if it doesn't exist, with graceful fallback if the home directory cannot be determined.

New REPL Commands:

  • .clear - Clear the screen using ANSI escape codes for a clean workspace
  • .clear history - Clear and persist the query history, removing all stored commands

Tab Completion: Tab completion now includes suggestions based on your command history, making it faster to re-run or modify previous queries.

Example usage:

sql> SELECT * FROM my_table;
sql> .clear          # Clears the screen
sql> .clear history  # Clears command history
sql> # Use arrow keys or tab to access previous commands

Additional Improvements & Bug Fixes​

  • Reliability: Fixed refresh worker panics with recovery handling to prevent runtime crashes during acceleration refreshes.
  • Reliability: Improved error messages for missing or invalid spicepod.yaml files, providing actionable feedback for misconfiguration.
  • Reliability: Fixed DuckDB metadata pointer loading issues for snapshots.
  • Performance: Ensured ListingTable partitions are pruned correctly when filters are not used.
  • Reliability: Fixed vector dimension determination for partitioned indexes.
  • Search: Fixed casing issues in Reciprocal Rank Fusion (RRF) for hybrid search queries.
  • Search: Fixed search field handling as metadata for chunked search indexes.
  • Validation: Added timestamp support for partition expressions.
  • Validation: Fixed regexp_match function for DuckDB datasets.
  • Validation: Fixed partition name validation for improved reliability.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

New HTTP Data Connector Recipe: New recipe demonstrating how to query REST APIs and HTTP(s) endpoints. See HTTP Connector Recipe for details.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.9.0-rc.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.9.0-rc.2 image:

docker pull spiceai/spiceai:1.9.0-rc.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Dependencies​

Changelog​

Spice v1.9.0-rc.1 (Nov 4, 2025)

· 16 min read
William Croxson
William Croxson
Senior Software Engineer at Spice AI

This is the first release candidate for v1.9.0, which introduces Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers DuckDB-comparable performance without scaling limitations. This release also upgrades to DataFusion v50 for improved query performance, expands search capabilities with full-text search on views and multi-column embeddings, includes significant DynamoDB and DuckDB accelerator improvements, and delivers security and reliability enhancements.

What's New in v1.9.0-rc.1​

Cayenne Data Accelerator (Alpha)​

Introducing Cayenne: SQL as an Acceleration Format: A new high-performance data accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format. Cayenne delivers query and ingestion performance comparable or better to DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.

Cayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing actual data in Vortex's compressed columnar format. This architecture provides:

Key Features:

  • SQLite + Vortex Architecture: All metadata is stored in SQLite tables with standard SQL transactions, while data lives in Vortex's compressed, chunked columnar format designed for zero-copy access and efficient scanning.
  • Simplified Operations: No complex file hierarchies, no JSON/Avro metadata files, no separate catalog servers—just SQL tables and Vortex data files. The entire metadata schema is intentionally simple for maximum reliability.
  • Fast Metadata Access: Single SQL query retrieves all metadata needed for query planning—no multiple round trips to storage, no S3 throttling, no reconstruction of metadata state from scattered files.
  • Efficient Small Changes: Dramatically reduces small file proliferation. Snapshots are just rows in SQLite tables, not new files on disk. Supports millions of snapshots without performance degradation.
  • High Concurrency: Changes consist of two steps: stage Vortex files (if any), then run a single SQL transaction. Much faster conflict resolution and support for many more concurrent updates than file-based formats.
  • Advanced Data Lifecycle: Full ACID transactions, delete support, and retention SQL execution on refresh commit.

Example Spicepod.yml configuration:

datasets:
  - from: s3:my_table
    name: accelerated_data
    acceleration:
      enabled: true
      engine: cayenne
      retention:
        sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'

Note, the Cayenne Data Accelerator is in Alpha with limitations.

For more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.

DataFusion v50 Upgrade​

Spice.ai is built on the DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:

Performance Improvements 🚀:

  • Dynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.
  • Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.

Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.

See the Apache DataFusion 50.0.0 Release for more details.

DynamoDB Data Connector Improvements​

Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: ddb_data
    params:
      scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows

Search & Embeddings Enhancements​

Full-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.

Multi-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.

Vector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.

Example Spicepod.yml configuration:

views:
  - name: aggregated_reviews
    sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
    embeddings:
      - column: review_text
        model: openai:text-embedding-3-small

DuckDB Accelerator Improvements​

Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.

Retention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.

UTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.

Example Spicepod.yml configuration:

datasets:
  - from: s3://my_bucket/large_table/
    name: partitioned_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      retention:
        sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'

Query Performance Optimizations​

Optimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.

Large RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.

Security & Reliability Improvements​

Enhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.

ODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.

CLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.

Oracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.

Observability & Tracing​

DataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.

AI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.

Git Data Connector (Alpha)​

Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.

Example Spicepod.yml configuration:

datasets:
  - from: git:https://github.com/myorg/myrepo
    name: git_metrics
    params:
      file_format: csv

For more details, refer to the Git Data Connector Documentation.

Additional Improvements & Bug Fixes​

  • Reliability: Fixed refresh worker panics with recovery handling to prevent runtime crashes during acceleration refreshes.
  • Reliability: Improved error messages for missing or invalid spicepod.yaml files, providing actionable feedback for misconfiguration.
  • Reliability: Fixed DuckDB metadata pointer loading issues for snapshots.
  • Performance: Ensured ListingTable partitions are pruned correctly when filters are not used.
  • Reliability: Fixed vector dimension determination for partitioned indexes.
  • Search: Fixed casing issues in Reciprocal Rank Fusion (RRF) for hybrid search queries.
  • Search: Fixed search field handling as metadata for chunked search indexes.
  • Validation: Added timestamp support for partition expressions.
  • Validation: Fixed regexp_match function for DuckDB datasets.
  • Validation: Fixed partition name validation for improved reliability.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates.

The Spice Cookbook includes 81 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.9.0-rc.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.9.0-rc.1 image:

docker pull spiceai/spiceai:1.9.0-rc.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

Spice v1.8.3 (Oct 27, 2025)

· 5 min read
David Stancu
David Stancu
Principal Software Engineer at Spice AI

Announcing the release of Spice v1.8.3! ⚡

Spice v1.8.3 is a patch release focused on performance, reliability, and observability. This release delivers optimizations for DuckDB acceleration, parameterized queries, and query plans. A new opt-in dedicated thread pool for queries is now in preview.

What's New in v1.8.3​

DuckDB Data Accelerator Improvements​

  • Connection Pool Sizing: The DuckDB accelerator now supports a configurable connection_pool_size parameter, supporting fine-grained control over concurrent query execution. This enables tuning for high-concurrency workloads and improved resource utilization.

Example Spicepod.yaml snippet:

datasets:
  - from: postgres:my_table
    name: my_table
    acceleration:
      enabled: true
      engine: duckdb
      params:
        connection_pool_size: 10
  • Automatic Statistics Recomputation: The new on_refresh_recompute_statistics parameter, on by default, triggers automatic ANALYZE execution after refreshes. This keeps DuckDB optimizer statistics up-to-date, ensuring efficient query plans and optimal performance.

Example Spicepod.yaml snippet:

datasets:
  - from: postgres:my_table
    name: my_table
    acceleration:
      enabled: true
      engine: duckdb
      params:
        on_refresh_recompute_statistics: disabled # default enabled

Task History SQL Query Plan Capture & Configuration​

Spice now supports automated SQL query plan capture and store (via EXPLAIN or EXPLAIN ANALYZE) in the task history, enabling deeper analysis and debugging of query execution. This feature is configurable, supporting control of which queries are included based on duration thresholds and plan type.

  • New Configuration Options:
    • task_history.captured_plan: Controls which plan is captured (none, explain, or explain analyze). Default none.
    • task_history.min_sql_duration: Minimum query duration before a plan is captured.
    • task_history.min_plan_duration: Minimum plan execution duration before a plan is captured.

Example spicepod.yaml snippet:

runtime:
  task_history:
    captured_plan: explain analyze
    min_sql_duration: 5s
    min_plan_duration: 10s

Query plans are captured asynchronously to avoid blocking query execution. The result of the plan is stored in the standard sql_query output in the task history.

Learn more in the Task History Documentation.

Query Performance Optimizations​

  • Optimized Prepared Statements (Parameterized Queries): Prepared statement caching for parameterized SQL queries has been improved, reducing planning overhead for repeated queries with different parameters. This results in faster execution and lower latency for workloads that reuse query structures.

  • Limit Pushdown via BytesProcessedExec: Introduces the BytesProcessedExec physical operator, enabling limit pushdown for large datasets. This optimization reduces the amount of data processed and improves top-k query performance.

Dedicated Query Thread Pool (Opt-In)​

Spice now supports running query execution and accelerated refreshes on a dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health and readiness checks fast. Opt-In for v1.8.3: This feature is opt-in for this release and will become enabled by default (opt-out) in v1.9.

Example Spicepod.yaml snippet:

runtime:
  params:
    dedicated_thread_pool: sql_engine # Default: disabled

Validation & Reliability Improvements​

  • Selective Evaluation Scorer Loading: Evaluation scorers are now loaded only when evaluation is explicitly defined, reducing unnecessary initialization and improving startup performance.

  • Improved Error Reporting: Enhanced error messages for misconfigured full-text search (FTS) on datasets and views, providing actionable feedback for configuration issues.

REPL & Usability​

  • Execution Time Display: The Spice REPL now displays query execution time even when queries return no results, improving user feedback and diagnostics.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates.

The Spice Cookbook includes 81 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.8.3, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.8.3 image:

docker pull spiceai/spiceai:1.8.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

Spice v1.8.2 (Oct 21, 2025)

· 5 min read
Jack Eadie
Jack Eadie
Token Plumber at Spice AI

Announcing the release of Spice v1.8.2! 🔍

Spice v1.8.2 is a patch release focused on reliability, validation, performance, and bug fixes, with improvements across DuckDB acceleration, S3 Vectors, document tables, and HTTP search.

What's New in v1.8.2​

Support Table Relations in /v1/search HTTP Endpoint​

Spice now supports table relations for the additional_columns and where parameters in the /v1/search endpoint. This enables improved search for multi-dataset use cases, where filters and columns can be used on specific datasets.

Example:

curl 'http://localhost:8090/v1/search' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' -d '{
        "text": "hello world",
        "additional_columns": ["tbl1.foo", "tbl2.bar", "baz"],
        "where": "tbl1.foo > 100000",
        "limit": 5
    }'

In this example, search results from the tbl1 dataset will include columns foo and baz, where foo > 100000. For tbl2, columns bar and baz will be returned.

DuckDB Data Accelerator Table Partitioning & Indexing​

  • Configurable DuckDB Index Scan: DuckDB acceleration now supports configurable duckdb_index_scan_percentage and duckdb_index_scan_max_count parameters, supporting fine-tuning of index scan behavior for improved query performance.

Example:

datasets:
  - from: postgres:my_table
    name: my_table
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        # When combined, DuckDB will use an index scan when the number of qualifying rows is less than the maximum of these two thresholds
        duckdb_index_scan_percentage: '0.10' # 10% as decimal
        duckdb_index_scan_max_count: '1000'

  • Hive-Style Partitioning: In file-partitioned mode, the DuckDB data accelerator uses Hive-style partitioning for more efficient file management.

  • Table-Based Partitioning: Spice now supports partitioning DuckDB accelerations within a single file. This approach maintains ACID guarantees for full and append mode refreshes, while optimizing resource usage and improving query performance. Configure via the partition_mode parameter:

datasets:
  - from: file:test_data.parquet
    name: test_data
    params:
      file_format: parquet
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        partition_mode: tables
      partition_by:
        - bucket(100, Field1)

S3 Vectors Reliability​

  • Race Condition Fix: Resolved a race condition in S3 Vectors index and bucket creation. The runtime also now checks if an index or bucket exists after a ConflictException, ensuring robust error handling during index creation and improving reliability for large-scale multi-index vector search.

Document Table Improvements​

  • Primary Key Update: Document tables now use the location column as the primary key, improving performance, consistency, and query reliability.

Additional Improvements & Bugfixes​

  • Reliability: Improved error handling and resource checks for S3 Vectors and DuckDB acceleration.
  • Validation: Expanded validation for partitioning and index creation.
  • Performance: Optimized partition refresh and index scan logic.
  • Bugfix: Don't nullify DuckDB release callbacks for schemas.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates.

The Spice Cookbook includes 81 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.8.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.8.2 image:

docker pull spiceai/spiceai:1.8.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

  • Update mongo config for benchmarks by @krinart in #7546
  • Configurable DuckDB duckdb_index_scan_percentage & duckdb_index_scan_max_count by @lukekim in #7551
  • Fix race condition in S3 Vectors index and bucket creation by @kczimm in #7577
  • Use 'location' as primary key for document tables by @Jeadie in #7567
  • Update official Docker builds to use release binaries by @phillipleblanc in #7597
  • Hive-style partitioning for DuckDB file mode by @kczimm in #7563
  • New Generate Changelog workflow by @krinart in #7562
  • Add support for DuckDB table-based partitioning by @sgrebnov in #7581
  • DuckDB table partitioning: delete partitions that no longer exist after full refresh by @sgrebnov in #7614
  • Rename duckdb_partition_mode to partition_mode param by @sgrebnov in #7622
  • Fix license issue in table-providers by @phillipleblanc in #7620
  • Make DuckDB table partition data write threshold configurable by @sgrebnov in #7626
  • fix: Don't nullify DuckDB release callbacks for schemas by @peasee in #7628
  • Fix integration tests by reverting the use of batch inserts w/ prepared statements by @phillipleblanc in #7630
  • Return TableProvider from CandidateGeneration::search by @Jeadie in #7559
  • Handle table relations in HTTP v1/search by @Jeadie in #7615

Spice v1.8.1 (Oct 13, 2025)

· 5 min read
Viktor Yershov
Viktor Yershov
Senior Software Engineer at Spice AI

Announcing the release of Spice v1.8.1! 🚀

Spice v1.8.1 is a patch release that adds Acceleration Snapshots Indexes, and includes a number of bug fixes and performance improvements.

What's New in v1.8.1​

Acceleration Snapshot Indexes​

  • Management of Acceleration Snapshots has been improved by adopting an Iceberg-inspired metadata.json, which now encodes pointer IDs, schema serialization, and robust checksum and size, which is validate before loading the snapshot.

  • Acceleration Snapshot Metrics: The following metrics are now available for Acceleration Snapshots:

  • dataset_acceleration_snapshot_bootstrap_duration_ms: The time it took the runtime to download the snapshot - only emitted when it initially downloads the snapshot.

  • dataset_acceleration_snapshot_bootstrap_bytes: The number of bytes downloaded to bootstrap the acceleration from the snapshot.

  • dataset_acceleration_snapshot_bootstrap_checksum: The checksum of the snapshot used to bootstrap the acceleration.

  • dataset_acceleration_snapshot_failure_count: Number of failures encountered when writing a new snapshot at the end of the refresh cycle. A snapshot failure does not prevent the refresh from completing.

  • dataset_acceleration_snapshot_write_timestamp: Unix timestamp in seconds when the last snapshot was completed.

  • dataset_acceleration_snapshot_write_duration_ms: The time it took to write the snapshot to object storage.

  • dataset_acceleration_snapshot_write_bytes: The number of bytes written on the last snapshot write.

  • dataset_acceleration_snapshot_write_checksum: The SHA256 checksum of the last snapshot write.

To learn more, see the Acceleration Snapshots Documentation and the Metrics Documentation.

Improved Regular Expression for DuckDB acceleration​

Regular expression support has been expanded when using DuckDB acceleration for functions like regexp-like and regexp_match.

For more details, refer to the SQL Reference for the list of available regular expression functions.

Additional Improvements & Bugfixes​

  • Reliability: Resolved an issue with partitioning on empty partition sets.
  • Validation: Added better validation for incorrectly configured Spicepods.
  • Reliability: Fixed partition_by accelerations when a projection is applied on empty partition sets.
  • Performance: Ensured ListingTable partitions are pruned when filters are not used.
  • Performance: Don't download acceleration snapshots if the acceleration is already present.
  • Performance: Refactored some blocking I/O and synchronization in the async codebase by moving operations to tokio::task::spawn_blocking, replacing blocking locks with async-friendly variants.
  • Bugfix: Nullable fields are now supported for S3 Vectors index columns.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

  • New Accelerated Snapshots Recipe - The recipe shows how to bootstrap DuckDB accelerations from object storage to skip cold starts.

The Spice Cookbook includes 81 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.8.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.8.1 image:

docker pull spiceai/spiceai:1.8.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​