3 posts tagged with "hybrid-search"

Announcing the release of Spice v1.9.0-rc.4! 🌶

This release candidate brings DuckDB v1.4.2, Cayenne partitioning improvements, and comprehensive security hardening across the CLI, data connectors, runtime, and MCP. v1.9.0-rc.4 also includes MySQL and PostgreSQL connector improvements with fixed nullability inferences and full-text search support, DynamoDB consistency improvements, HTTP connector validation and UX enhancements, and numerous reliability and performance optimizations. Significant improvements were also made to test and automation infrastructure to ensure high quality releases.

v1.9.0 introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations, and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0 also upgrades to DataFusion v50 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, and delivers many additional features and improvements.

What's New in v1.9.0​

Cayenne Data Accelerator (Beta)​

Introducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.

Cayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:

Key Features:

• SQLite + Vortex Architecture: All metadata is stored in SQLite tables with standard SQL transactions, while data lives in Vortex's compressed, chunked columnar format designed for zero-copy access and efficient scanning.
• Simplified Operations: No complex file hierarchies, no JSON/Avro metadata files, no separate catalog servers—just SQL tables and Vortex data files. The entire metadata schema is intentionally simple for maximum reliability.
• Fast Metadata Access: Single SQL query retrieves all metadata needed for query planning—no multiple round trips to storage, no S3 throttling, no reconstruction of metadata state from scattered files.
• Efficient Small Changes: Dramatically reduces small file proliferation. Snapshots are just rows in SQLite tables, not new files on disk. Supports millions of snapshots without performance degradation.
• High Concurrency: Changes consist of two steps: stage Vortex files (if any), then run a single SQL transaction. Much faster conflict resolution and support for many more concurrent updates than file-based formats.
• Advanced Data Lifecycle: Full ACID transactions, delete support, and retention SQL execution on refresh commit.

Example Spicepod.yml configuration:

datasets:
  - from: s3:my_table
    name: accelerated_data_30d
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      refresh_mode: append
      retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'

Note, the Cayenne Data Accelerator is in Beta with limitations.

For more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.

Multi-Node Distributed Query (Preview)​

Apache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0-rc.3.

Architecture:

A distributed Spice cluster consists of:

• Scheduler: Responsible for distributed query planning and work queue management for the executor fleet
• Executors: One or more nodes responsible for running physical query plans

Getting Started:

Start a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:

# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051

Start one or more executors configured with the scheduler's flight URI:

# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051

Query Execution:

Queries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:

EXPLAIN SELECT count(id) FROM my_dataset;

Current Limitations:

• Accelerated datasets are currently not supported. This feature is designed for querying partitioned data lake formats (Parquet, Delta Lake, Iceberg, etc.)
• The feature is in preview and may have stability or performance limitations
• Specific acceleration support is planned for future releases

DataFusion v50 Upgrade​

Spice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:

Performance Improvements 🚀:

• Dynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.

• Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.

Apache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.

Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.

See the Apache DataFusion 50.0.3 Release for more details.

DuckDB v1.4.2 Upgrade and Accelerator Improvements​

DuckDB v1.4.2: DuckDB has been upgraded to v1.4.2, which includes several performance optimizations.

Composite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      indexes:
        '(region, product_id)': enabled

Performance example with composite index on 7.5M rows:

SELECT * FROM sales WHERE region = 'US' AND product_id = 12345;

-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index

DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://sales_data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        query_federation: disabled # Required currently for intermediate materialization
      indexes:
        '(region, product_id)': enabled

Performance example:

-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;

-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster

The optimizer automatically rewrites the query to:

WITH _intermediate_materialize AS MATERIALIZED (
  SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;

Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.

Retention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.

UTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.

Example Spicepod.yml configuration:

datasets:
  - from: s3://my_bucket/large_table/
    name: partitioned_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      retention:
        sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'

HTTP Data Connector​

• Querying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.

• Query HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table

• Configurable retry logic, timeouts, and POST request support for more complex API interactions

Example Spicepod.yml configuration:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    params:
      file_format: json
      max_retries: 3
      client_timeout: 10s

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;

If a request_body is supplied it will be posted to the endpoint:

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{"name": "michael"}'
LIMIT 10;

HTTP endpoints can be accelerated using refresh_sql:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    acceleration:
      enabled: true
      refresh_mode: full
      refresh_sql: |
        SELECT request_path, request_query, content 
        FROM tvmaze
        WHERE request_path = '/search/people'
          AND request_query IN ('q=michael', 'q=luke')

DynamoDB Data Connector Improvements​

Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: ddb_data
    params:
      scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows

S3 Versioning Support​

Atomic Range Reads for Versioned Files: Spice now supports S3 Versioning for all connectors using object-store (S3, Delta Lake, etc.), ensuring range reads over versioned files are atomically correct. When S3 versioning is enabled, Spice automatically tracks version IDs during file discovery and uses them for all subsequent range reads, preventing inconsistencies from concurrent file modifications.

Current limitations:

• Multi-file connections (e.g., partitioned datasets) do not yet support version tracking across all files
• Version tracking is automatic when S3 versioning is enabled on the bucket

Search & Embeddings Enhancements​

Full-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.

Multi-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.

Vector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.

Example Spicepod.yml configuration:

views:
  - name: aggregated_reviews
    sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
    embeddings:
      - column: review_text
        model: openai:text-embedding-3-small

Dedicated Query Thread Pool (Now Enabled by Default)​

Dedicated Query Thread Pool: Query execution and accelerated refreshes now run on their own dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health checks fast and avoiding unnecessary Kubernetes pod restarts under load.

This feature was opt-in in previous releases and is now enabled by default. To disable it and revert to the previous behavior, add the following spicepod.yaml configuration:

runtime:
  params:
    dedicated_thread_pool: none

Query Performance Optimizations​

Stale-While-Revalidate Cache Control: Query results now support "stale-while-revalidate" cache control, allowing stale cached data to be served immediately while asynchronously refreshing the cache entry in the background. This improves response times for frequently-accessed queries while maintaining data freshness. Requires cache key type to be set to "sql (raw)" for proper operation.

Optimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.

Large RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.

Query Result Cache: Stale-While-Revalidate​

HTTP Cache-Control Support: The query result cache now supports the stale-while-revalidate Cache-Control directive, enabling faster response times by serving stale cached results immediately while asynchronously refreshing the cache in the background. This feature is particularly useful for applications that can tolerate slightly stale data in exchange for improved performance.

How it works:

When a cache entry is stale but within the stale-while-revalidate window, Spice will:

1. Immediately return the stale cached result to the client
2. Asynchronously re-execute the query in the background to refresh the cache
3. Future requests will use the refreshed data

Configuration:

Use the Cache-Control HTTP header with the stale-while-revalidate directive:

Cache-Control: max-age=300, stale-while-revalidate=60

This configuration caches results for 5 minutes (300 seconds), and allows serving stale results for an additional 60 seconds while refreshing in the background.

Requirements:

• Must use plan or raw SQL cache keys (set cache_key_type to sql or plan in results_caching configuration)
• Background revalidation re-executes queries through the normal query path
• Timestamp tracking automatically determines cache entry age for staleness checks

Example configuration via HTTP header:

GET /v1/sql
Cache-Control: max-age=600, stale-while-revalidate=120
X-Cache-Key-Type: sql

This feature improves application responsiveness while ensuring data freshness through background updates.

Security & Reliability Improvements​

Enhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.

ODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.

CLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.

Oracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.

AWS Authentication Improvements​

Improved Credential Retry Logic: AWS SDK credential initialization has been significantly improved with more robust retry logic and better error handling. The system now automatically retries transient credential resolution failures using Fibonacci backoff, allowing Spice to tolerate extended AWS outages (up to ~48 hours) without manual intervention.

Key features:

• Automatic retry with backoff: Implements Fibonacci backoff for transient credential failures (network issues, temporary AWS service disruptions)
• Configurable retry limits: Supports up to 300 retry attempts with a maximum retry interval of 600 seconds
• Better error handling: Distinguishes between retryable errors (connector errors) and non-retryable errors (misconfiguration)
• Unauthenticated access support: Properly supports unauthenticated access to public S3 buckets without requiring credentials
• Improved error messages: Provides detailed logging with attempt numbers, retry intervals, and error context for better troubleshooting

The improvements ensure more reliable AWS service integration, particularly in environments with intermittent network connectivity or during AWS service degradations.

Observability & Tracing​

DataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.

AI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.

Git Data Connector (Alpha)​

Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.

Example Spicepod.yml configuration:

datasets:
  - from: git:https://github.com/myorg/myrepo
    name: git_metrics
    params:
      file_format: csv

For more details, refer to the Git Data Connector Documentation.

Spice Java SDK 0.4.0​

The Spice Java SDK have been upgraded with support configurable Arrow memory limit: spice-java v0.4.0

SpiceClient client = SpiceClient.builder()
    .withArrowMemoryLimitMB(1024) // 1GB limit
    .build();

CLI Improvements​

Install Specific Versions: The spice install command now supports installing specific versions of the Spice runtime and CLI. This enables easy version management, downgrading, or installation of specific releases for testing or compatibility requirements.

Usage:

# Install a specific version
spice install v1.8.3

# Install a specific version with AI flavor
spice install v1.8.3 ai

# Install latest version (existing behavior)
spice install
spice install ai

Note: Homebrew installations require manual version management via brew install spiceai/spiceai/spice@<version>.

Persistent Query History: The Spice CLI REPL (SQL, search, and chat interfaces) now persists command history to ~/.spice/query_history.txt, making your query history available across sessions. The history file is automatically created if it doesn't exist, with graceful fallback if the home directory cannot be determined.

New REPL Commands:

• .clear - Clear the screen using ANSI escape codes for a clean workspace
• .clear history - Clear and persist the query history, removing all stored commands

Tab Completion: Tab completion now includes suggestions based on your command history, making it faster to re-run or modify previous queries.

Example usage:

sql> SELECT * FROM my_table;
sql> .clear          # Clears the screen
sql> .clear history  # Clears command history
sql> # Use arrow keys or tab to access previous commands

Additional Improvements & Bug Fixes​

• Reliability: Fixed refresh worker panics with recovery handling to prevent runtime crashes during acceleration refreshes.
• Reliability: Improved error messages for missing or invalid spicepod.yaml files, providing actionable feedback for misconfiguration.
• Reliability: Fixed DuckDB metadata pointer loading issues for snapshots.
• Performance: Ensured ListingTable partitions are pruned correctly when filters are not used.
• Reliability: Fixed vector dimension determination for partitioned indexes.
• Search: Fixed casing issues in Reciprocal Rank Fusion (RRF) for hybrid search queries.
• Search: Fixed search field handling as metadata for chunked search indexes.
• Validation: Added timestamp support for partition expressions.
• Validation: Fixed regexp_match function for DuckDB datasets.
• Validation: Fixed partition name validation for improved reliability.

Contributors​

• @Jeadie
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @kczimm
• @krinart
• @mach-kernel

Breaking Changes​

No breaking changes.

Cookbook Updates​

New HTTP Data Connector Recipe: New recipe demonstrating how to query REST APIs and HTTP(s) endpoints. See HTTP Connector Recipe for details.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.9.0-rc.4, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.9.0-rc.4 image:

docker pull spiceai/spiceai:1.9.0-rc.4

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Dependencies​

• DataFusion: Upgraded to v50
• Apache Arrow: Upgraded to v56
• DuckDB: Upgraded to v1.4.2
• Delta Kernel: Upgraded to v0.16.0

Changelog (rc.4)​

• Upgrade shutdown signal Ordering by @krinart in #8017
• Set max-age: 0 during stale by @lukekim in #8018
• Add E2E test release for Helm by @lukekim in #8023
• Update generate_changelog script by @krinart in #8028
• [CLI] Improve auto-complete by @lukekim in #8022
• Improve verify helm workflow by @lukekim in #8024
• fix: Ensure Vortex UncompressedSizeInBytes is calculated by @peasee in #8044
• WW README Update by @wyattwenzel in #8058
• Reenable dynamic federation support by @Jeadie in #8026
• fix: Prevent SortExec from ordering below SchemaCastScanExec by @peasee in #8061
• Skip logging and return OK() on error during shutdown by @krinart in #8057
• Partition pruning with complex expressions by @lukekim in #8040
• Make DynamoDB snapshots consistent by @krinart in #8069
• Add check for error log by @krinart in #8070
• Fix tracing of 's3_vector_query_and_get' by @Jeadie in #8065
• DuckDB v1.4.2 by @mach-kernel in #8073
• Fix failing OpenAI test by @krinart in #8076
• Enable 'test_recency_scoring' by @Jeadie in #8068
• Test operator: avoid duplicate Flight requests when using --http-clients by @sgrebnov in #8071
• Update load tests to use truth percentile values by @sgrebnov in #8079
• Update DynamoDB to RC by @krinart in #8060
• CachedQueryVector to avoid recomputing embedding vector for spilling/partitioned vector indexes. by @Jeadie in #8059
• Fix DuckDB on_commit sink race by @lukekim in #8081
• Add partitioned duckdb by @lukekim in #8083
• [CLI] Security and santization by @lukekim in #8082
• Fix partition_by expression by @lukekim in #8087
• Data Components security fixes and sanitization by @lukekim in #8086
• Runtime security and sanitization by @lukekim in #8088
• Add spicepod-validator tool and fix spicepods by @lukekim in #8089
• Skip data fetches for S3 single file refreshes by @lukekim in #8072
• MCP security and sanitization by @lukekim in #8090
• Add install-dev target back to Makefile by @Jeadie in #8100
• fix 'testoperator run search' by @Jeadie in #8101
• Update datafusion-table-providers - fix nullability inferences for MySQL and PostgreSQL, and fix full text search for PostgreSQL by @ewgenius in #8092
• Remove duplicate install-with-models by @phillipleblanc in #8107
• Improve Cayenne partitioning by @lukekim in #8097
• Testoperator dispatch: respect verify_results dispatch configuration by @sgrebnov in #8106
• Include 'match' column only if chunk offsets found in seach query 'LogicalPlan' by @Jeadie in #8102
• Fix validation path by @lukekim in #8109
• Fix dispatch paths by @lukekim in #8110
• Fix dispatch spicepod paths by @lukekim in #8112
• Update test Spicepods by @lukekim in #8131
• Add validation to reference schema by @lukekim in #8111
• Include root error when failing to find latest timestamp in accelerated table by @sgrebnov in #8132
• fix: HTTP Connector validation, query and body by @lukekim in #8115
• Update nsql model list by @lukekim in #8141
• Update DynamoDB Benchmarks by @krinart in #8135
• Fix Dremio E2E test by @sgrebnov in #8139
• Improve HTTP Connector UX by @lukekim in #8146
• QueryOverrides for DynamoDB benchmarks by @krinart in #8151
• test-framework: add row count validation skipping with TPC-DS defaults by @sgrebnov in #8149
• Suppress delta_kernel::listed_log_files warnings by @phillipleblanc in #8158
• Update table providers to fix warning by @phillipleblanc in #8156
• Suppress MCP limit log by @phillipleblanc in #8159
• Remove incorrect tool name validation by @Jeadie in #8161
• Disable results validation for federated/glue[csv].yaml by @phillipleblanc in #8163
• Fix dynamodb overrides by @phillipleblanc in #8165
• Fix dynamo db overrides again by @phillipleblanc in #8166
• few more dynamodb overrides by @phillipleblanc in #8167
• Add stub release notes for v1.9.0-rc.4 by @phillipleblanc in #8168

Announcing the release of Spice v1.9.0-rc.2! 🌶

This is the second release candidate for v1.9.0, which introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0-rc.2 also upgrades to DataFusion v50 and DuckDB v1.4.1 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, includes significant DynamoDB and DuckDB accelerator improvements, expands the HTTP data connector to support endpoints as tables, and delivers many security and reliability improvements.

What's New in v1.9.0-rc.2​

Cayenne Data Accelerator (Beta)​

Introducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.

Cayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:

Key Features:

• SQLite + Vortex Architecture: All metadata is stored in SQLite tables with standard SQL transactions, while data lives in Vortex's compressed, chunked columnar format designed for zero-copy access and efficient scanning.
• Simplified Operations: No complex file hierarchies, no JSON/Avro metadata files, no separate catalog servers—just SQL tables and Vortex data files. The entire metadata schema is intentionally simple for maximum reliability.
• Fast Metadata Access: Single SQL query retrieves all metadata needed for query planning—no multiple round trips to storage, no S3 throttling, no reconstruction of metadata state from scattered files.
• Efficient Small Changes: Dramatically reduces small file proliferation. Snapshots are just rows in SQLite tables, not new files on disk. Supports millions of snapshots without performance degradation.
• High Concurrency: Changes consist of two steps: stage Vortex files (if any), then run a single SQL transaction. Much faster conflict resolution and support for many more concurrent updates than file-based formats.
• Advanced Data Lifecycle: Full ACID transactions, delete support, and retention SQL execution on refresh commit.

Example Spicepod.yml configuration:

datasets:
  - from: s3:my_table
    name: accelerated_data_30d
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      refresh_mode: append
      retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'

Note, the Cayenne Data Accelerator is in Beta with limitations.

For more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.

Multi-Node Distributed Query (Preview)​

Apache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0-rc.2.

Architecture:

A distributed Spice cluster consists of:

• Scheduler: Responsible for distributed query planning and work queue management for the executor fleet
• Executors: One or more nodes responsible for running physical query plans

Getting Started:

Start a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:

# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051

Start one or more executors configured with the scheduler's flight URI:

# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051

Query Execution:

Queries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:

EXPLAIN SELECT count(id) FROM my_dataset;

Current Limitations:

• Accelerated datasets are currently not supported. This feature is designed for querying partitioned data lake formats (Parquet, Delta Lake, Iceberg, etc.)
• The feature is in preview and may have stability or performance limitations
• Specific acceleration support is planned for future releases

DataFusion v50 Upgrade​

Spice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:

Performance Improvements 🚀:

• Dynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.

• Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.

Apache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.

Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.

See the Apache DataFusion 50.0.0 Release for more details.

DuckDB v1.4.1 Upgrade and Accelerator Improvements​

DuckDB v1.4.1: DuckDB has been upgraded to v1.4.1, which includes several performance optimizations.

Composite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      indexes:
        '(region, product_id)': enabled

Performance example with composite index on 7.5M rows:

SELECT * FROM sales WHERE region = 'US' AND product_id = 12345;

-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index

DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://sales_data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        query_federation: disabled # Required currently for intermediate materialization
      indexes:
        '(region, product_id)': enabled

Performance example:

-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;

-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster

The optimizer automatically rewrites the query to:

WITH _intermediate_materialize AS MATERIALIZED (
  SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;

Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.

Retention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.

UTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.

Example Spicepod.yml configuration:

datasets:
  - from: s3://my_bucket/large_table/
    name: partitioned_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      retention:
        sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'

HTTP Data Connector​

• Querying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.

• Query HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table

• Configurable retry logic, timeouts, and POST request support for more complex API interactions

Example Spicepod.yml configuration:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    params:
      file_format: json
      max_retries: 3
      client_timeout: 10s

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;

If a request_body is supplied it will be posted to the endpoint:

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{"name": "michael"}'
LIMIT 10;

HTTP endpoints can be accelerated using refresh_sql:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    acceleration:
      enabled: true
      refresh_mode: full
      refresh_sql: |
        SELECT request_path, request_query, content 
        FROM tvmaze
        request_path = '/search/people'
          AND request_query IN ('q=michael', 'q=luke')

DynamoDB Data Connector Improvements​

Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: ddb_data
    params:
      scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows

S3 Versioning Support​

Atomic Range Reads for Versioned Files: Spice now supports S3 Versioning for all connectors using object-store (S3, Delta Lake, etc.), ensuring range reads over versioned files are atomically correct. When S3 versioning is enabled, Spice automatically tracks version IDs during file discovery and uses them for all subsequent range reads, preventing inconsistencies from concurrent file modifications.

Current limitations:

• Multi-file connections (e.g., partitioned datasets) do not yet support version tracking across all files
• Version tracking is automatic when S3 versioning is enabled on the bucket

Search & Embeddings Enhancements​

Full-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.

Multi-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.

Vector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.

Example Spicepod.yml configuration:

views:
  - name: aggregated_reviews
    sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
    embeddings:
      - column: review_text
        model: openai:text-embedding-3-small

Dedicated Query Thread Pool (Now Enabled by Default)​

Dedicated Query Thread Pool: Query execution and accelerated refreshes now run on their own dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health checks fast and avoiding unnecessary Kubernetes pod restarts under load.

This feature was opt-in in previous releases and is now enabled by default in v1.9.0-rc.2. To disable it and revert to the previous behavior, add the following spicepod.yaml configuration:

runtime:
  params:
    dedicated_thread_pool: none

Query Performance Optimizations​

Stale-While-Revalidate Cache Control: Query results now support "stale-while-revalidate" cache control, allowing stale cached data to be served immediately while asynchronously refreshing the cache entry in the background. This improves response times for frequently-accessed queries while maintaining data freshness. Requires cache key type to be set to "sql (raw)" for proper operation.

Optimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.

Large RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.

Query Result Cache: Stale-While-Revalidate​

HTTP Cache-Control Support: The query result cache now supports the stale-while-revalidate Cache-Control directive, enabling faster response times by serving stale cached results immediately while asynchronously refreshing the cache in the background. This feature is particularly useful for applications that can tolerate slightly stale data in exchange for improved performance.

How it works:

When a cache entry is stale but within the stale-while-revalidate window, Spice will:

1. Immediately return the stale cached result to the client
2. Asynchronously re-execute the query in the background to refresh the cache
3. Future requests will use the refreshed data

Configuration:

Use the Cache-Control HTTP header with the stale-while-revalidate directive:

Cache-Control: max-age=300, stale-while-revalidate=60

This configuration caches results for 5 minutes (300 seconds), and allows serving stale results for an additional 60 seconds while refreshing in the background.

Requirements:

• Must use plan or raw SQL cache keys (set cache_key_type to sql or plan in results_caching configuration)
• Background revalidation re-executes queries through the normal query path
• Timestamp tracking automatically determines cache entry age for staleness checks

Example configuration via HTTP header:

GET /v1/sql
Cache-Control: max-age=600, stale-while-revalidate=120
X-Cache-Key-Type: sql

This feature improves application responsiveness while ensuring data freshness through background updates.

Security & Reliability Improvements​

Enhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.

ODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.

CLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.

Oracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.

AWS Authentication Improvements​

Improved Credential Retry Logic: AWS SDK credential initialization has been significantly improved with more robust retry logic and better error handling. The system now automatically retries transient credential resolution failures using Fibonacci backoff, allowing Spice to tolerate extended AWS outages (up to ~48 hours) without manual intervention.

Key features:

• Automatic retry with backoff: Implements Fibonacci backoff for transient credential failures (network issues, temporary AWS service disruptions)
• Configurable retry limits: Supports up to 300 retry attempts with a maximum retry interval of 600 seconds
• Better error handling: Distinguishes between retryable errors (connector errors) and non-retryable errors (misconfiguration)
• Unauthenticated access support: Properly supports unauthenticated access to public S3 buckets without requiring credentials
• Improved error messages: Provides detailed logging with attempt numbers, retry intervals, and error context for better troubleshooting

The improvements ensure more reliable AWS service integration, particularly in environments with intermittent network connectivity or during AWS service degradations.

Observability & Tracing​

DataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.

AI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.

Git Data Connector (Alpha)​

Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.

Example Spicepod.yml configuration:

datasets:
  - from: git:https://github.com/myorg/myrepo
    name: git_metrics
    params:
      file_format: csv

For more details, refer to the Git Data Connector Documentation.

Spice Java SDK 0.4.0​

The Spice Java SDK have been upgraded with support configurable Arrow memory limit: spice-java v0.4.0

SpiceClient client = SpiceClient.builder()
    .withArrowMemoryLimitMB(1024) // 1GB limit
    .build();

CLI Improvements​

Install Specific Versions: The spice install command now supports installing specific versions of the Spice runtime and CLI. This enables easy version management, downgrading, or installation of specific releases for testing or compatibility requirements.

Usage:

# Install a specific version
spice install v1.8.3

# Install a specific version with AI flavor
spice install v1.8.3 ai

# Install latest version (existing behavior)
spice install
spice install ai

Note: Homebrew installations require manual version management via brew install spiceai/spiceai/spice@<version>.

Persistent Query History: The Spice CLI REPL (SQL, search, and chat interfaces) now persists command history to ~/.spice/query_history.txt, making your query history available across sessions. The history file is automatically created if it doesn't exist, with graceful fallback if the home directory cannot be determined.

New REPL Commands:

• .clear - Clear the screen using ANSI escape codes for a clean workspace
• .clear history - Clear and persist the query history, removing all stored commands

Tab Completion: Tab completion now includes suggestions based on your command history, making it faster to re-run or modify previous queries.

Example usage:

sql> SELECT * FROM my_table;
sql> .clear          # Clears the screen
sql> .clear history  # Clears command history
sql> # Use arrow keys or tab to access previous commands

Additional Improvements & Bug Fixes​

• Reliability: Fixed refresh worker panics with recovery handling to prevent runtime crashes during acceleration refreshes.
• Reliability: Improved error messages for missing or invalid spicepod.yaml files, providing actionable feedback for misconfiguration.
• Reliability: Fixed DuckDB metadata pointer loading issues for snapshots.
• Performance: Ensured ListingTable partitions are pruned correctly when filters are not used.
• Reliability: Fixed vector dimension determination for partitioned indexes.
• Search: Fixed casing issues in Reciprocal Rank Fusion (RRF) for hybrid search queries.
• Search: Fixed search field handling as metadata for chunked search indexes.
• Validation: Added timestamp support for partition expressions.
• Validation: Fixed regexp_match function for DuckDB datasets.
• Validation: Fixed partition name validation for improved reliability.

Contributors​

• @Jeadie
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @kczimm
• @krinart
• @mach-kernel

Breaking Changes​

No breaking changes.

Cookbook Updates​

New HTTP Data Connector Recipe: New recipe demonstrating how to query REST APIs and HTTP(s) endpoints. See HTTP Connector Recipe for details.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.9.0-rc.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.9.0-rc.2 image:

docker pull spiceai/spiceai:1.9.0-rc.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Dependencies​

• DataFusion: Upgraded to v50
• Apache Arrow: Upgraded to v56
• DuckDB: Upgraded to v1.4.1
• Delta Kernel: Upgraded to v0.16.0

Changelog​

• Fix for search field as metadata for chunked search indexes by @Jeadie in #7429
• Bump object_store from 0.12.3 to 0.12.4 by @app/dependabot in #7433
• Properly respect disabling snapshots by @phillipleblanc in #7431
• Revert "Properly respect disabling snapshots" by @sgrebnov in #7439
• Revert "Disable snapshots by default" by @sgrebnov in #7438
• Add preview warning for write access mode by @sgrebnov in #7440
• fix: regexp_match for DuckDB datasets by @kczimm in #7443
• Add feature is currently in preview warning for snapshots by @sgrebnov in #7442
• [Logger] Also emit Datafusion logs by @mach-kernel in #7441
• add missing snapshot by @kczimm in #7446
• Fix tracing so that ai_completions are parented under sql_query by @lukekim in #7415
• Enable snapshot acceleration by default by @phillipleblanc in #7451
• Disable acceleration refresh metrics by @krinart in #7450
• Add v1.8 release notes by @phillipleblanc in #7430
• fix: partition name validation by @kczimm in #7452
• Fix lint error due to ignore without reasons by @krinart in #7454
• Add models and CUDA support to spiced install script by @lukekim in #7457
• Post-release 1.8 updates by @phillipleblanc in #7455
• Remove println in datafusion by @phillipleblanc in #7461
• Update end_game.md to notify once release is done by @sgrebnov in #7460
• Remove italics from snapshot logging by @phillipleblanc in #7463
• Update openapi.json by @app/github-actions in #7466
• Fix generate spicepod schema by @phillipleblanc in #7464
• Fix generate acknowledements by @phillipleblanc in #7465
• Update spicepod.schema.json by @app/github-actions in #7469
• fix: Ensure ListingTable partitions are pruned when filters are not used by @peasee in #7471
• Create runtime-secrets crate by @phillipleblanc in #7474
• Create runtime-parameters crate by @phillipleblanc in #7475
• Don't download the snapshot if the acceleration is present by @phillipleblanc in #7477
• Fix casing for keywords and additional columns by @Jeadie in #7770
• Bump actions/upload-artifact from 4 to 5 by @app/dependabot in #7750
• Bump criterion from 0.5.1 to 0.7.0 by @app/dependabot in #7740
• Bump rustls-native-certs from 0.8.1 to 0.8.2 by @app/dependabot in #7744
• Git Data Connector (Alpha) by @lukekim in #7772
• Pepper accelerator delete support by @lukekim in #7616
• Update Helm chart instructions for Helm in end_game.md by @sgrebnov in #7776
• Turso data accelerator by @lukekim in #7472
• Apply retention SQL filter to refresh fetch by @phillipleblanc in #7778
• Add Parquet buffering option for DuckDB partitioned writes (tables mode) by @sgrebnov in #7735
• fix: EmptyExec when list indexes is empty by @kczimm in #7784
• 1.8.3 post-release housekeeping by @mach-kernel in #7783
• feat: Upgrade to Datafusion v50 by @peasee in #7777
• fix: Replace vortex datafusion with public crate by @peasee in #7791
• Full-text search on views by @Jeadie in #7733
• Revert "Apply retention SQL filter to refresh fetch (#7778)" by @phillipleblanc in #7796
• fix: Add ingest duration and acceleration size metrics to testoperator by @peasee in #7792
• Set local timezone to UTC for DuckDB by @phillipleblanc in #7797
• add Timestamp support for partition expressions by @kczimm in #7803
• Fix trunk lint by @krinart in #7804
• Add missing mongodb params by @krinart in #7807
• Embedding columns on view components by @Jeadie in #7795
• Add Turso as a Pepper Catalog metastore by @lukekim in #7793
• Run retention_sql on refresh commit for DuckDB by @lukekim in #7785
• docs: Update datafusion upgrade checklist by @peasee in #7812
• Vector engines on views by @Jeadie in #7808
• Handle refresh worker panics and add recovery test by @phillipleblanc in #7815
• chunk large record batches to control memory usage by @kczimm in #7802
• fix: cannot determine vector dimension for partitioned indexes by @kczimm in #7818
• Upgrade to Turso v0.3 by @lukekim in #7821
• fix: Ensure custom *Exec ExecutionPlans push down dynamic filters by @peasee in #7811
• handle casing in RRF by @Jeadie in #7825
• Enable 'turso' for pepper acceleration by default by @sgrebnov in #7826
• Improved DynamoDB Data Connector by @krinart in #7715
• Initial support for llama.cpp as LLM inference backend by @lukekim in #7794
• Pepper: Implement retention SQL on refresh commit by @phillipleblanc in #7814
• Fix Dockerfiles for arm64 by @lukekim in #7834
• [DynamoDB] Handle filter edge-cases by @krinart in #7830
• [DynamoDB] Support parallelization for Scan request by @krinart in #7829
• Don't feature gate Pepper by @lukekim in #7832
• Fix llama.cpp static link by @lukekim in #7835
• fix: docker nightly builds by @kczimm in #7837
• Use GitHub-hosted macOS runner only for tag releases by @lukekim in #7836
• Fix Bug: DuckDB INTERNAL Error: Failed to load metadata pointer by @sgrebnov in #7839
• Fix docker arm64 build to use aegis in pure-rust mode by @lukekim in #7840
• Revert "Use GitHub-hosted macOS runner only for tag releases" by @lukekim in #7843
• Rename Pepper to Cayenne by @lukekim in #7844
• Tighten CLI permissions and install script by @lukekim in #7845
• Set mvcc for Cayenne Turso metastore by @lukekim in #7850
• Optimize Prepared Statements by @lukekim in #7859
• Remove unwrap from ODBC connector, fix secrets, and kuberenetes secre… by @lukekim in #7846
• Improve and secure HTTP client usage by @lukekim in #7847
• Pin Oracle Instant Client download to a SHA by @lukekim in #7851
• Improve experience for missing or invalid Spicepod.yaml by @lukekim in #7849
• chore: Fix PR linting by @peasee in #7865
• Revert FlightIPC issues by @Jeadie in #7870
• Improve error message by adding 'cayenne' to the list of valid accelerator engines by @sgrebnov in #7882
• fix: allow parameter index without dollar signs by @kczimm in #7887
• Temporary disable supports_limit_pushdown for SchemaCastScanExec by @sgrebnov in #7893
• Remove '.embeddings[].metadata' by @Jeadie in #7897
• Optimize macOS and Windows builds by @lukekim in #7863
• fix: Kafka message delivery failed by @kczimm in #7883
• docs: Update component criteria by @peasee in #7891
• fix: Make integration run with no relevant changes, disable makefile targets on PR by @peasee in #7896
• Add Cayenne benchmark and concurrency tests and remove indexes for Turso MVCC by @lukekim in #7879
• Revert llama.cpp engine by @lukekim in #7898
• Make Cayenne snapshotting more robust by @sgrebnov in #7899
• Release notes v1.9.0-rc1 by @Jeadie in #7902
• Fix dataset_acceleration_last_refresh_time_ms unit to milliseconds in description by @ewgenius in #7901
• Fix lint error in record_explain_plan functionality by @sgrebnov in #7906
• Cleanup old snapshots after full refresh by @lukekim in #7908
• Cayenne deletion vector caching support by @lukekim in #7903
• Split filters into partition filters (for pruning) and data filters by @lukekim in #7889
• fix: Update benchmark snapshots by @app/github-actions in #7911
• fix: Update benchmark snapshots by @app/github-actions in #7912
• fix: Update benchmark snapshots by @app/github-actions in #7913
• Update spicepod.schema.json by @app/github-actions in #7916
• fix: Update benchmark snapshots by @app/github-actions in #7917
• Add Cayenne & Turso accelerators to E2E CI test matrix by @lukekim in #7922
• Make preview warnings consistent by @lukekim in #7921
• Filter and write optimizations by @lukekim in #7918
• fix: Set sccache region explicitly by @peasee in #7928
• fix: Enable integration test merge group checks by @peasee in #7927
• Update testoperator release branch from 1.8 to 1.9 by @peasee in #7926
• Update DuckDB to 1.4.1 with composite ART scans by @mach-kernel in #7884
• Don't build Windows on trunk pushes by @lukekim in #7931
• fix: Use correct minio secret in build binary push by @peasee in #7934
• Update test-framework workers to use dedicated Flight client by @sgrebnov in #7938
• Fix financebench, configure s3vectors for appropriate snapshotting by @Jeadie in #7935
• Don't try to initialize accelerator if it is disabled by @lukekim in #7932
• Add spark UDFs to Spice by @Jeadie in #7936
• Fix extra async_trait in cayenne metadata catalog by @phillipleblanc in #7942
• deps: Upgrade to Rust 1.90 by @peasee in #7941
• Add cayenne accelerator to README.md by @ewgenius in #7905
• fix: Update benchmark snapshots by @app/github-actions in #7948
• Run integration tests with AWS_EC2_METADATA_DISABLED flag by @sgrebnov in #7952
• Only retry credentials on ConnectorError by @kczimm in #7944
• fix: Improve join reordering by ensuring JoinSelection is applied by @peasee in #7828
• fix: Remove unused deps, consolidate workspace deps by @peasee in #7953
• bump async-openai commit by @kczimm in #7929
• deps: Use vortex fork by @peasee in #7954
• Enable tracing in delta lake integration tests by @sgrebnov in #7951
• Update datasets in S3 vectors test case by @Jeadie in #7956
• Add spiced metrics scraping to test operator by @lukekim in #7937
• Memoize S3 vectors ListIndex API call with configurable TTL by @kczimm in #7910
• Cayenne performance optimizations by @lukekim in #7907
• Setup HotFix issue template by @ewgenius in #7957
• Fix AWS SDK credential cache retry handling by @phillipleblanc in #7943
• Infer RRF join_key from TableProvider::constraints and implement SearchQueryProvider::constraints. by @Jeadie in #7959
• [Optimizer]: DuckDB intermediate materialization (non-federated) by @mach-kernel in #7964
• 1.7.3 post-release housekeeping by @ewgenius in #7962
• Fix digest_many UDF for ColumnarValue::Array. by @Jeadie in #7960
• Fix spiced metrics reporting as part of benchmark tests by @sgrebnov in #7967
• Avoid pushing down Spice specific UDFs to accelerators during federation by @Jeadie in #7909
• CLI file persisted history with .clear and .clear history commands by @lukekim in #7970
• ResultsCache Cache-Control stale-while-revalidate by @lukekim in #7963
• Use GetVectors API instead of returnData by @kczimm in #7083
• Make DuckDB intermediate materialization logic more robust by @sgrebnov in #7971
• [Cayenne] Configurable target Vortex file size by @lukekim in #7972
• fix: Update benchmark snapshots by @app/github-actions in #7974
• Bump github.com/klauspost/compress from 1.17.11 to 1.18.1 by @app/dependabot in #7872
• fix: Update benchmark snapshots by @app/github-actions in #7978
• fix: Update benchmark snapshots by @app/github-actions in #7982
• Run Integration tests on spiceai-dev-runners by @sgrebnov in #7985
• [CLI] Fix cursor issue due to flush by @lukekim in #7981
• fix: Support S3 versioning, Vortex dynamic filter pushdown by @peasee in #7984
• Make cluster a default feature by @lukekim in #7994
• Optimize DuckDB Intermediate Index Materialization for No-Index Case by @sgrebnov in #7998
• HTTP connector with dynamic filter support by @lukekim in #7969
• Revert federation 'can_execute_plan' by @Jeadie in #7999
• Fix stale caching by @lukekim in #7995
• Fix count(*) for http connector by @krinart in #8001
• [CLI] Install specific version by @lukekim in #8006
• Fix stale with revalidate request/response by @lukekim in #8005
• Fallback RequestContext for cluster queries by @Jeadie in #8007
• Use use_rustls_tls for Spice Cloud /connect by @lukekim in #8008
• Use delta-kernel-rs 0.16x + Parquet reader with object meta API changes by @mach-kernel in #8011
• fix: Update datafusion & arrow-rs with S3 versioning fix by @lukekim in #8012

The latest Spice.ai Open Source release (v1.5.0) brings major improvements to search, including native support for Amazon S3 Vectors. Announced in public preview at AWS Summit New York 2025, Amazon S3 Vectors is a new S3 bucket type purpose-built for vector embeddings, with dedicated APIs for similarity search.

Spice AI was a day 1 launch partner for S3 Vectors, integrating it as a scalable vector index backend. In this post, we explore how S3 Vectors integrates into Spice.ai’s data, search, and AI-inference engine, how Spice manages indexing and lifecycle of embeddings for production vector search, and how this unlocks a powerful hybrid search experience. We’ll also put this in context with industry trends and compare Spice’s approach to other vector database solutions like Qdrant, Weaviate, Pinecone, and Turbopuffer.

Amazon S3 Vectors Overview​

Amazon S3 Vectors extends S3 object storage with native support for storing and querying vectors at scale. As AWS describes, it is “designed to provide the same elasticity, scale, and durability as Amazon S3,” providing storage of billions of vectors and sub-second similarity queries. Crucially, S3 Vectors dramatically lowers the cost of vector search infrastructure – reducing upload, storage, and query costs by up to 90% compared to traditional solutions. It achieves this by separating storage from compute: vectors reside durably in S3, and queries execute on transient, on-demand resources, avoiding the need for always-on, memory-intensive vector database servers. In practice, S3 Vectors exposes two core operations:

1. Upsert vectors – assign a vector (an array of floats) to a given key (identifier) and optionally store metadata alongside it.

2. Vector similarity query – given a new query vector, efficiently find the stored vectors that are closest (e.g. minimal distance) to it, returning their keys (and scores).

This transforms S3 into a massively scalable vector index service. You can store embeddings at petabyte scale and perform similarity search with metrics like cosine or Euclidean distance via a simple API. It’s ideal for AI use cases like semantic search, recommendations, or Retrieval-Augmented Generation (RAG) where large volumes of embeddings need to be queried semantically. By leveraging S3’s pay-for-use storage and ephemeral compute, S3 Vectors can handle infrequent or large-scale queries much more cost-effectively than memory-bound databases, yet still deliver sub-second results.

Vector Search with Embeddings​

Vector similarity search retrieves data by comparing items in a high-dimensional embedding space rather than by exact keywords. In a typical pipeline:

• Data to vectors: We first convert each data item (text, image, etc.) into a numeric vector representation (embedding) using an ML model. For example, a customer review text might be turned into a 768-dimensional embedding that encodes its semantic content. Models like Amazon Titan Embeddings, OpenAI, or Hugging Face sentence transformers handle this step.

• Index storage: These vectors are stored in a specialized index or database optimized for similarity search. This could be a dedicated vector database or, in our case, Amazon S3 Vectors acting as the index. Each vector is stored with an identifier (e.g. the primary key of the source record) and possibly metadata.

• Query by vector: A search query (e.g. a phrase or image) is also converted into an embedding vector. The vector index is then queried to find the closest stored vectors by distance metric (cosine, Euclidean, dot product, etc.). The result is a set of IDs of the most similar items, often with a similarity score.

This process enables semantic search – results are returned based on meaning and similarity rather than exact text matches. It powers features like finding relevant documents by topic even if exact terms differ, recommendation systems (finding similar user behavior or content), and providing knowledge context to LLMs in RAG. With the Spice.ai Open Source integration, this whole lifecycle (embedding data, indexing vectors, querying) is managed by the Spice runtime and exposed via a familiar SQL or HTTP interface.

Amazon S3 Vectors in Spice.ai​

Spice.ai is an open-source data, search and AI compute engine that supports vector search end-to-end. By integrating S3 Vectors as an index, Spice can embed data, store embeddings in S3, and perform similarity queries – all orchestrated through simple configuration and SQL queries. Let’s walk through how you enable and use this in Spice.

Configuring a Dataset with Embeddings​

To use vector search, annotate your dataset schema to specify which column(s) to embed and with which model. Spice supports various embedding models (both local or hosted) via the embeddings section in the configuration. For example, suppose we have a customer reviews table and we want to enable semantic search over the review text (body column):

datasets:
  - from: oracle:"CUSTOMER_REVIEWS"
    name: reviews
    columns:
      - name: body
        embeddings:
          from: bedrock_titan # use an embedding model defined below

embeddings:
  - from: bedrock:amazon.titan-embed-text-v2:0
    name: bedrock_titan
    params:
      aws_region: us-east-2
      dimensions: '256'

In this spicepod.yaml, we defined an embedding model bedrock_titan (in this case AWS's Titan text embedding model) and attached it to the body column. When the Spice runtime ingests the dataset, it will automatically generate a vector embedding for each row’s body text using that model. By default, Spice can either store these vectors in its acceleration layer or compute them on the fly. However, with S3 Vectors, we can offload them to an S3 Vectors index for scalable storage.

To use S3 Vectors, we simply enable the vector engine in the dataset config:

datasets:
  - from: oracle:"CUSTOMER_REVIEWS"
    name: reviews
    vectors:
      enabled: true
      engine: s3_vectors
      params:
        s3_vectors_bucket: my-s3-vector-bucket
        #... (rest of dataset definition as above)

This tells Spice to create or use an S3 Vectors index (in the specified S3 bucket) for storing the body embeddings. Spice manages the entire index lifecycle: it creates the vector index, handles inserting each vector with its primary key into S3, and knows how to query it. The embedding model and data source are as before – the only change is where the vectors are stored and queried. The benefit is that now our vectors reside in S3’s highly scalable storage, and we can leverage S3 Vectors’ efficient similarity search API.

Performing a Vector Search Query​

Once configured, performing a semantic search is straightforward. Spice exposes both an HTTP endpoint and a SQL table-valued function for vector search. For example, using the HTTP API:

curl -X POST http://localhost:8090/v1/search \
 -H "Content-Type: application/json" \
 -d '{
 "datasets": ["reviews"],
 "text": "issues with same day shipping",
 "additional_columns": ["rating", "customer_id"],
 "where": "created_at >= now() - INTERVAL '7 days'",
 "limit": 2
 }'

This JSON query says: search the reviews dataset for items similar to the text "issues with same day shipping", and return the top 2 results, including their rating and customer id, filtered to reviews from the last 7 days. The Spice engine will embed the query text (using the same model as the index), perform a similarity lookup in the S3 Vectors index, filter by the WHERE clause, and return the results. A sample response might look like:

{
  "results": [
    {
      "matches": {
        "body": "Everything on the site made it seem like I’d get it the same day. Still waiting the next morning was a letdown."
      },
      "data": { "rating": 3, "customer_id": 6482 },
      "primary_key": { "review_id": 123 },
      "score": 0.82,
      "dataset": "reviews"
    },
    {
      "matches": {
        "body": "It was marked as arriving 'today' when I paid, but the delivery was pushed back without any explanation. Timing was kind of important for me."
      },
      "data": { "rating": 2, "customer_id": 3310 },
      "primary_key": { "review_id": 24 },
      "score": 0.76,
      "dataset": "reviews"
    }
  ],
  "duration_ms": 86
}

Each result includes the matching column snippet (body), the additional requested fields, the primary key, and a relevance score. In this case, the two reviews shown are indeed complaints about “same day” delivery issues, which the vector search found based on semantic similarity to the query (see how the second result made no mention of "same day" delivery, but rather described a similar issue as the first ).

Developers can also use SQL for the same operation. Spice provides a table function vector_search(dataset, query) that can be used in the FROM clause of a SQL query. For example, the above search could be expressed as:

SELECT review_id, rating, customer_id, body, score
FROM vector_search(reviews, 'issues with same day shipping')
WHERE created_at >= to_unixtime(now() - INTERVAL '7 days')
ORDER BY score DESC
LIMIT 2;

This would yield a result set (with columns like review_id, score, etc.) similar to the JSON above, which you can join or filter just like any other SQL table. This ability to treat vector search results as a subquery/table and combine them with standard SQL filtering is a powerful feature of Spice.ai’s integration – few other solutions let you natively mix vector similarity and relational queries so seamlessly.

See a 2-min demo of it in action:

Managing Embeddings Storage in Spice.ai​

An important design question for any vector search system is where and how to store the embedding vectors. Before introducing S3 Vectors, Spice offered two approaches for managing vectors:

1. Accelerator storage: Embed the data in advance and store the vectors alongside other cached data in a Data Accelerator (Spice’s high-performance materialization layer). This keeps vectors readily accessible in memory or fast storage.

2. Just-in-time computation: Compute the necessary vectors on the fly during a query, rather than storing them persistently. For example, at query time, embed only the subset of rows that satisfy recent filters (e.g. all reviews in the last 7 days) and compare those to the query vector.

Both approaches have trade-offs. Pre-storing in an accelerator provides fast query responses but may not be feasible for very large datasets (which might not fit entirely, or fit affordably in fast storage) and accelerators, like DuckDB or SQLite aren’t optimized for similarity search algorithms on billion-scale vectors. Just-in-time embedding avoids extra storage but becomes prohibitively slow when computing embeddings over large data scans (and for each query), and provides no efficient algorithm for efficiently finding similar neighbours.

Amazon S3 Vectors offers a compelling third option: the scalability of S3 with the efficient retrieval of vector index data structures. By configuring the dataset with engine: s3_vectors as shown earlier, Spice will offload the vector storage and similarity computations to S3 Vectors. This means you can handle very large embedding sets (millions or billions of items) without worrying about Spice’s memory or local disk limits, and still get fast similarity operations via S3’s API. In practice, when Spice ingests data, it will embed each row’s body and PUT it into the S3 Vector index (with the review_id as the key, and possibly some metadata). At query time, Spice calls S3 Vectors’ query API to retrieve the nearest neighbors for the embedded query. All of this is abstracted away; you simply query Spice and it orchestrates these steps.

The Spice runtime manages index creation, updates, and deletion. For instance, if new data comes in or old data is removed, Spice will synchronize those changes to the S3 vector index. Developers don’t need to directly interact with S3 – it’s configured once in YAML. This tight integration accelerates application development: your app can treat Spice like any other database, while behind the scenes Spice leverages S3’s elasticity for the heavy lifting.

Vector Index Usage in Query Execution​

How does a vector index actually get used in Spice’s SQL query planner? To illustrate, consider the simplified SQL we used:

SELECT *
FROM vector_search(reviews, 'issues with same day shipping')
ORDER BY score DESC
LIMIT 5;

Logically, without a vector index, Spice would have to do the following at query time:

1. Embed the query text 'issues with same day shipping' into a vector v.

2. Retrieve or compute all candidate vectors for the searchable column (here every body embedding in the dataset). This could mean scanning every row or at least every row matching other filter predicate.

3. Calculate distances between the query vector v and each candidate vector, compute a similarity score (e.g. score = 1 - distance).

4. Sort all candidates by the score and take the top 5.

For large datasets, steps 2–4 would be extremely expensive (a brute-force scan through potentially millions of vectors for each search, then a full sort operation). A vector index avoiding unnecessary recomputation of embeddings, reduces the number of distance calculations required, and provides in-order candidate neighbors.

With S3 Vectors, step 2 and 3 are pushed down to the S3 service. The vector index can directly return the top K closest matches to v. Conceptually, S3 Vectors gives back an ordered list of primary keys with their similarity scores. For example, it might return something like: {(review_id=123, score=0.82), (review_id=24, score=0.76), ...} up to K results.

Spice then uses these results, logically as a temporary table (let’s call it vector_query_results), joined with the main reviews table to get the full records. In SQL pseudocode, Spice does something akin to:

-- The vector index returns the closest matches for a given query.
CREATE TEMP TABLE vector_query_results (
 review_id BIGINT,
 score FLOAT
);

Imagine this temp table is populated by an efficient vector retrieval operatin in S3 Vectors for the query.

-- Now we join to retrieve full details
SELECT r.review_id, r.rating, r.customer_id, r.body, v.score
FROM vector_query_results v
JOIN reviews r ON r.review_id = v.review_id
ORDER BY v.score DESC
LIMIT 5;

This way, only the top few results (say 50 or 100 candidates) are processed in the database, rather than the entire dataset. The heavy work of narrowing down candidates occurs inside the vector index. Spice essentially treats vector_search(dataset, query) as a table-valued function that produces (id, score) pairs which are then joinable.

Handling Filters Efficiently​

One consideration when using an external vector index is how to handle additional filter conditions (the WHERE clause). In our example, we had a filter created_at >= now() - 7 days. If we simply retrieve the top K results from the vector search and then apply the time filter, we might run into an issue: those top K might not include any recent items, even if there are relevant recent items slightly further down the similarity ranking. This is because S3 Vectors (like most ANN indexes) will return the top K most similar vectors globally, unaware of our date constraint.

If only a small fraction of the data meets the filter, a naive approach could drop most of the top results, leaving fewer than the desired number of final results. For example, imagine the vector index returns 100 nearest reviews overall, but only 5% of all reviews are from the last week – we’d expect only ~5 of those 100 to be recent, possibly fewer than the LIMIT. The query could end up with too few results not because they don’t exist, but because the index wasn’t filter-aware and we truncated the candidate list.

To solve this, S3 Vectors supports metadata filtering at query time. We can store certain fields as metadata with each vector and have the similarity search constrained to vectors where the metadata meets criteria. Spice.ai leverages this by allowing you to mark some dataset columns as “vector filterable”. In our YAML, we could do:

columns:
  - name: created_at
    metadata:
      vectors: filterable

By doing this, Spice's query planner will include the created_at value with each vector it upserts to S3, and it will push down the time filter into the S3 Vectors query. Under the hood, the S3 vector query will then return only nearest neighbors that also satisfy created_at >= now()-7d. This greatly improves both efficiency and result relevance. The query execution would conceptually become:

-- Vector query with filter returns a temp table including the metadata
CREATE TEMP TABLE vector_query_results (
 review_id BIGINT,
 score FLOAT,
 created_at TIMESTAMP
);
-- vector_query_results is already filtered to last 7 days

SELECT r.review_id, r.rating, r.customer_id, r.body, v.score
FROM vector_query_results v
JOIN reviews r ON r.review_id = v.review_id
-- (no need for additional created_at filter here, it’s pre-filtered)
ORDER BY v.score DESC
LIMIT 5;

Now the index itself is ensuring all similar reviews are from the last week, and so if there are at least five results from the last week, it will return a full result (i.e. respecting LIMIT 5).

Including Data to Avoid Joins​

Another optimization Spice supports is storing additional, non-filterable columns in the vector index to entirely avoid the expensive table join back to the main table for certain queries. For example, we might mark rating, customer_id, or even the text body as non-filterable vector metadata. This means these fields are stored with the vector in S3, but not used for filtering (just for retrieval). In the Spice config, it would look like:

columns:
  - name: rating
    metadata:
      vectors: non-filterable
  - name: customer_id
    metadata:
      vectors: non-filterable
  - name: body
    metadata:
      vectors: non-filterable

With this setup, when Spice queries S3 Vectors, the vector index will return not only each match’s review_id and score, but also the stored rating, customer_id, and body values. Thus, the temporary vector_query_results table already has all the information needed to satisfy the query. We don’t even need to join against the reviews table unless we want some column that wasn’t stored. The query can be answered entirely from the index data:

SELECT review_id, rating, customer_id, body, score
FROM vector_query_results
ORDER BY score DESC
LIMIT 5;

This is particularly useful for read-heavy query workloads where hitting the main database adds latency. By storing the most commonly needed fields along with the vector, Spice’s vector search behaves like an index-only query (similar to covering indexes in relational databases). You trade a bit of extra storage in S3 (duplicating some fields, but still managed by Spice) for faster queries that bypass the heavier join.

This extends to WHERE conditions on non-filterable columns, or filter predicate unsupported by S3 vectors. Spice's execution engine can apply these filters, still avoiding any expensive JOIN on the underlying table.

SELECT review_id, rating, customer_id, body, score
FROM vector_query_results
where rating > 3  -- Filter performed in Spice on, with non-filterable data from vector index
ORDER BY score DESC
LIMIT 5;

It’s worth noting that you should choose carefully which fields to mark as metadata – too many or very large fields could increase index storage and query payload sizes. Spice gives you the flexibility to include just what you need for filtering and projection to optimize each use case.

Beyond Basic Vector Search in Spice​

Many real-world search applications go beyond a single-vector similarity lookup. Spice.ai’s strength is that it’s a full database engine. You can compose more complex search workflows, including hybrid search (combining keyword/text search with vector search), multi-vector queries, re-ranking strategies, and more. Spice provides both an out-of-the-box hybrid search API and the ability to write custom SQL to implement advanced retrieval logic.

• Multiple vector fields or multi-modal search: You might have vectors for different aspects of data (e.g. an e-commerce product could have embeddings for both its description and the product's image. Or a document has both a title and body that should be searchable individually and together) that you may want to search across and combine results. Spice lets you do vector search on multiple columns easily, and you can weight the importance of each. For instance, you might boost matches in the title higher than matches in the body.

• Vector and full-text search: Similar to vector search, columns can have text indexes defined that enable full-text BM25 search. Text search can then be performed in SQL with a similar text_search UDTF. The /v1/search HTTP API will perform a hybrid search across both full-text and vector indexes, merging results using Reciprocal Rank Fusion (RRF). This means you get a balanced result set that accounts for direct keyword matches as well as semantic similarity. The example below demonstrates how RRF can be implemented in SQL by combining ranks.

• Hybrid vector + keyword search: Sometimes you want to ensure certain keywords are present while also using semantic similarity. Spice supports hybrid search natively – its default /v1/search HTTP API actually performs both full-text BM25 search and vector search, then merges results using Reciprocal Rank Fusion (RRF). This means you get a balanced result set that accounts for direct keyword matches as well as semantic similarity. In Spice’s SQL, you can also call text_search(dataset, query) for traditional full-text search, and combine it with vector_search results. The example below demonstrates how RRF can be implemented in SQL by combining ranks.

• Two-phase retrieval (re-ranking): A common pattern is to use a fast first-pass retrieval (e.g. a keyword search) to get a larger candidate set, then apply a more expensive or precise ranking (e.g. vector search) on this subset to improve the score of the required final candidate set. With Spice, you can orchestrate this in SQL or in application code. For example, you could query a BM25 index for 100 candidates, then perform a vector search amongst this candidate set(i.e. restricted to those IDs) for a second phase. Since Spice supports standard SQL constructs, you can express these multi-step plans with common table expressions (CTEs) and joins.

To illustrate hybrid search, here’s a SQL snippet that uses the Reciprocal Rank Fusion (RRF) technique to merge vector and text search results for the same query (RRF is used, when needed, in the v1/search HTTP API):

WITH
vector_results AS (
 SELECT review_id, RANK() OVER (ORDER BY score DESC) AS vector_rank
 FROM vector_search(reviews, 'issues with same day shipping')
),
text_results AS (
 SELECT review_id, RANK() OVER (ORDER BY score DESC) AS text_rank
 FROM text_search(reviews, 'issues with same day shipping')
)
SELECT
 COALESCE(v.review_id, t.review_id) AS review_id,
 -- RRF scoring: 1/(60+rank) from each source
 (1.0 / (60 + COALESCE(v.vector_rank, 1000)) +
 1.0 / (60 + COALESCE(t.text_rank, 1000))) AS fused_score
FROM vector_results v
FULL OUTER JOIN text_results t ON v.review_id = t.review_id
ORDER BY fused_score DESC
LIMIT 50;

This takes the vector similarity results and text (BM25) results, assigns each a rank based not on the score, but rather the relative order of candidates, and combines these ranks for an overall order. Spice’s primary key SQL semantics easily enables this document ID join.

For a multi-column vector search example, suppose our reviews dataset has both a title and body with embeddings, and we want to prioritize title matches higher. We could create a combined_score where the title is weighted twice as high as the body:

WITH
body_results AS (
 SELECT review_id, score AS body_score
 FROM vector_search(reviews, 'issues with same day shipping', col => 'body')
),
title_results AS (
 SELECT review_id, score AS title_score
 FROM vector_search(reviews, 'issues with same day shipping', col => 'title')
)
SELECT
 COALESCE(body.review_id, title.review_id) AS review_id,
 COALESCE(body_score, 0) + 2.0 * COALESCE(title_score, 0) AS combined_score
FROM body_results
FULL OUTER JOIN title_results ON body_results.review_id = title_results.review_id
ORDER BY combined_score DESC
LIMIT 5;

These examples scratch the surface of what you can do by leveraging Spice’s SQL-based composition. The key point is that Spice isn’t just a vector database – it’s a hybrid engine that lets you combine vector search with other query logic (text search, filters, joins, aggregations, etc.) all in one place. This can significantly simplify building complex search and AI-driven applications.

(Note: Like most vector search systems, S3 Vectors uses an approximate nearest neighbor (ANN) algorithm under the hood for performance. This yields fast results that are probabilistically the closest, which is usually an acceptable trade-off in practice. Additionally, in our examples we focused on one embedding per row; production systems may use techniques like chunking text into multiple embeddings or adding external context, but the principles above remain the same.)

Industry Context and Comparisons​

The rise of vector databases over the past few years (Pinecone, Qdrant, Weaviate, etc.) has been driven by the need to serve AI applications with semantic search at scale. Each solution takes a slightly different approach in architecture and trade-offs. Spice.ai’s integration with Amazon S3 Vectors represents a newer trend in this space: decoupling storage from compute for vector search, analogous to how data warehouses separated compute and storage in the past. Let’s compare this approach with some existing solutions:

• Traditional Vector Databases (Qdrant, Weaviate, Pinecone): These systems typically run as dedicated services or clusters that handle both the storage of vectors (on disk or in-memory) and the computation of similarity search. For example, Qdrant (an open-source engine in Rust) allows either in-memory storage or on-disk storage (using RocksDB) for vectors and payloads. It’s optimized for high performance and offers features like filtering, quantization, and distributed clustering, but you generally need to provision servers/instances that will host all your data and indexes. Weaviate, another popular open-source vector DB, uses a Log-Structured Merge (LSM) tree based storage engine that persists data to disk and keeps indexes in memory. Weaviate supports hybrid search (it can combine keyword and vector queries) and offers a GraphQL API, with a managed cloud option priced mainly by data volume. Pinecone, a fully managed SaaS, also requires you to select a service tier or pod which has certain memory/CPU allocated for your index – essentially your data lives in Pinecone’s infrastructure, not in your AWS account. These solutions excel at low-latency search for high query throughput scenarios (since data is readily available in RAM or local SSD), but the cost can be high for large datasets. You pay for a lot of infrastructure to be running, even during idle times. In fact, prior to S3 Vectors, vector search engines often stored data in memory at ~$2/GB and needed multiple replicas on SSD, which is “the most expensive way to store data”, as Simon Eskildsen (Turbopuffer’s founder) noted. Some databases mitigate cost by compressing or offloading to disk, but still, maintaining say 100 million embeddings might require a sizable cluster of VMs or a costly cloud plan.

• Spice.ai with Amazon S3 Vectors: This approach flips the script by storing vectors in cheap, durable object storage (S3) and loading/indexing them on demand. As discussed, S3 Vectors keeps the entire vector dataset in S3 at ~$0.02/GB storage , and only spins up transient compute (managed by AWS) to serve queries, meaning you aren’t paying for idle GPU or RAM time. AWS states this design can cut total costs by up to 90% while still giving sub-second performance on billions of vectors. It’s essentially a serverless vector search model – you don’t manage servers or even dedicated indices; you just use the API. Spice.ai’s integration means developers get this cost-efficiency without having to rebuild their application: they can use standard SQL and Spice will push down operations to S3 Vectors as appropriate. This decoupled storage/compute model is ideal for use cases where the data is huge but query volumes are moderate or bursty (e.g., an enterprise semantic search that is used a few times an hour, or a nightly ML batch job). It avoids the “monolithic database” scenario of having a large cluster running 24/7. However, one should note that if you need extremely high QPS (thousands of queries per second at ultra-low latency), a purely object-storage-based solution might not outperform a tuned in-memory vector DB – AWS positions S3 Vectors as complementary to higher-QPS solutions like OpenSearch for real-time needs.

• Turbopuffer: Turbopuffer is a startup that, much like Spice with S3 Vectors, is built from first principles on object storage. It provides “serverless vector and full-text search… fast, 10× cheaper, and extremely scalable,” by leveraging S3 or similar object stores with smart caching. The philosophy is the same: use the durability and low cost of object storage for the bulk of data, and layer a cache (memory/SSD) in front for performance-critical portions. According to Turbopuffer’s founder, moving from memory/SSD-centric architectures to an object storage core can yield 100× cost savings for cold data and 6–20× for warm data, without sacrificing too much performance. Turbopuffer’s engine indexes data incrementally on S3 and uses caching to achieve similar latency to conventional search engines on hot data. The key difference is that Turbopuffer is a standalone search service (with its own API), whereas Spice uses AWS’s S3 Vectors service as the backend. Both approaches validate the industry trend toward disaggregated storage for search. Essentially, they are bringing the cloud data warehouse economics to vector search: store everything cheaply, compute on demand.

In summary, Spice.ai’s integration with S3 Vectors and similar efforts indicate a shift in vector search towards cost-efficient, scalable architectures that separate the concerns of storing massive vector sets and serving queries. Developers now have options: if you need blazing fast, realtime vector search with constant high traffic, dedicated compute infrastructure might be justified. But for many applications – enterprise search, AI assistants with a lot of knowledge but lower QPS, periodic analytics over embeddings – offloading to something like S3 Vectors can save enormously on cost while still delivering sub-second performance at huge scale. And with Spice.ai, you get the best of both worlds: the ease of a unified SQL engine that can do keyword + vector hybrid search on structured data, combined with the power of a cloud-native vector store. It simplifies your stack (no separate vector DB service to manage) and accelerates development since you can join and filter vector search results with your data immediately in one query .

References:

• Spice.ai announcement: “Spice.ai Now Supports Amazon S3 Vectors For Vector Search at Petabyte Scale!”

• Spice.ai Amazon S3 Vectors documentation

• Spice.ai Amazon S3 Vectors Cookbook Recipe Sample

• Amazon S3 Vectors official page

• Pinecone Database Architecture (managed vector database)

• Qdrant documentation (storage modes and features)

• Turbopuffer blog by Simon Eskildsen (cost of search on object storage)

• Weaviate storage architecture discussion