9 posts tagged with "release"

Announcing the release of Spice v1.0.7 🏎️

Spice v1.0.7 improves memory usage when using DuckDB, improves schema inference performance when using object-store based data connectors, and fixes a bug in Dremio schema inference.

Highlights in v1.0.7​

• DuckDB Memory Usage: Memory usage when using DuckDB has been significantly improved for data loads and refreshes through expanded use of zero-copy Arrow and multi-threading for data loads. When a duckdb_memory_limit is specified, disk spilling has been improved for greater-than-memory workloads. In addition, a new temp_directory runtime parameter supports storing temporary files to alternative location than the DuckDB data file for higher throughput. For example, temp_directory could be set to a different high-IOPs IO2 EBS volume that is separate from the duckdb_file_path.

  Automated end-to-end tests for the DuckDB Accelerator coverage has been significantly expanded.

  For configuration details, see the documentation for runtime parameters and the DuckDB Data Accelerator.

• Schema Inference Performance for Object-Store Data Connectors: Schema inference performance has been improved, especially for large numbers of objects (1M+ objects) when using object-store based data connectors by making the object-listing and selection more efficient.

Performance​

When compared to previous versions, Spice v1.0.7 loads DuckDB accelerated datasets significantly faster. When using the TPCH lineitem dataset at Scale Factor 100 (600M rows):

Without Indexes​

5x faster, 28% less memory usage.

With Indexes​

2.5x faster. Higher memory usage in v1.0.7 is due to better resource utilization to achieve faster load times. Use the duckdb_memory_limit parameter to control memory usage.

Documentation​

• DuckDB Data Accelerator: Has been expanded with additional resource usage guidance.
• Memory: A new section for memory considerations has been added to the Reference section.

Contributors​

• @phillipleblanc
• @sgrebnov
• @peasee
• @Sevenannn

Breaking Changes​

No breaking changes.

Upgrading​

To upgrade to v1.0.7, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.7 image:

docker pull spiceai/spiceai:1.0.7

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

• DataFusion Table Providers: Upgraded from 760ece6ac52b7d180d697f347642af403c2e711c to 9ba9dce19a1fdbd5e22cc2e445c5b3ea731944b4.

Changelog​

- fix: Remove on zero results arguments from benchmarks by @peasee in https://github.com/spiceai/spiceai/pull/4533
- Run benchmark tests w/o uploading test results (pending improvements) by @sgrebnov in https://github.com/spiceai/spiceai/pull/4843
- fix: Return BAD_REQUEST when not embeddings are configured by @peasee in https://github.com/spiceai/spiceai/pull/4804
- Fix Dremio schema inference by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5114
- Improve performance of schema inference for object-store data connectors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5124
- Always download spice runtime version matched with spice cli version by @Sevenannn in https://github.com/spiceai/spiceai/pull/4761
- Fix go lint errors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5147
- Make DuckDB acceleration E2E tests more comprehensive by @sgrebnov in https://github.com/spiceai/spiceai/pull/5146
- Enable Spice to load larger than memory datasets into DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5149
- Add `temp_directory` runtime parameter and insert it for DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5152
- Fix Postgres and MySQL installation on macos14-runner (E2E CI) by @sgrebnov in https://github.com/spiceai/spiceai/pull/5155
- Enable E2E for DuckDB full mode acceleration with indexes only in CI by @sgrebnov in https://github.com/spiceai/spiceai/pull/5154

Full Changelog: github.com/spiceai/spiceai/compare/v1.0.6...v1.0.7

Announcing the release of Spice v1.0.6 ⚡

Spice v1.0.6 improves stability for DuckDB acceleration, Iceberg Data/Catalog connector improvements when using AWS Glue, and fixes an issue with the ready_state: on_registration federation fallback when using DuckDB. In addition, redundant data refreshes on startup are avoided for accelerations with persistent data.

Highlights in v1.0.6​

• Iceberg Data/Catalog Connector Improvements: Improves Iceberg data & catalog connector reliability, including bug fixes for AWS Glue API rate-limiting and compatibility, REST API pagination support, explicit AWS credential handling, and support for AWS STS role assumption.

• Fixes On-Registration Fallback when using DuckDB: Previously, when using DuckDB as a data accelerator and the ready_state: on_registration configuration, queries made during the initial data refresh did not properly fallback to the federated source. This is now fixed.

• DuckDB downgraded for Stability: DuckDB has been downgraded to v1.1.3 due to a regression in memory handling tracked by duckdb/duckdb issue #16640. Once resolved and validated, Spice will re-upgrade to v1.2.x.

• Expanded Integration Tests: Additional integration tests covering federated accelerator behavior and graceful shutdown processes have been added.

• Optimized Data Refresh for Persistent Accelerations: Changed behavior in v1.0.6. When using persistent (file-mode) acceleration without a defined refresh interval, Spice performs a full refresh at startup only if no previously accelerated data is available. This ensures efficient startup behavior by avoiding unnecessary refreshes. This logic applies only to full refreshes when no refresh interval is specified.

To maintain the previous behavior and always refresh on every startup, set:

 acceleration:
  refresh_on_startup: always

Contributors​

• @peasee
• @phillipleblanc
• @sgrebnov
• @lukekim
• @Sevenannn

Breaking Changes​

Starting from v1.0.6 when using persistent (file-mode) acceleration without a defined refresh interval, Spice performs a full refresh at startup only if no previously accelerated data is available. To maintain the previous behavior and always refresh on every startup, set:

 acceleration:
  refresh_on_startup: always

Cookbook Updates​

No new recipes.

Upgrading​

To upgrade to v1.0.6, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.6 image:

docker pull spiceai/spiceai:1.0.6

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

• duckdb-rs: Downgraded from 1.2.0 to 1.1.3

Changelog​

• Implement proper ready_state: on_registration for federation enabled accelerators by @phillipleblanc in #5019
• Add indexes and primary keys mismatch detection for DuckDB Acceleration by @sgrebnov in #5045
• Add comprehensive integration tests for the ready_state behavior by @phillipleblanc in #5042
• Add test Spicepod for acceleration with constraints by @sgrebnov in #4891
• Add test Spicepod for DuckDB append acceleration with constraints by @sgrebnov in #4898
• Add DuckDB graceful shutdown test to E2E CI tests by @sgrebnov in #5047
• Update duckdb_append_with_pk_and_indexes.yaml (work for duckdb 1.1.x) by @sgrebnov in #5067
• fix: Downgrade to DuckDB 1.1.3 by @peasee in #5055
• fix: Acceleration federation integration test by @peasee in #5070
• Improvements to Iceberg Catalog/Data Connector by @phillipleblanc in #5071
• Add Results-Cache-Status to indicate query result came from cache by @phillipleblanc in #4809
• fix: Spice.ai schema inference by @peasee in #4674
• Add refresh_on_startup Spicepod configuration param by @phillipleblanc and @sgrebnov in #5086
• Test restart behavior of DuckDB file acceleration against glue iceberg table by @Sevenannn #5075
• Run Iceberg Data Connector - DuckDB File mode integration test by @Sevenannn #5069
• Integration test for glue iceberg catalog by @Sevenannn #5077

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.0.5...v1.0.6

Announcing the release of Spice v1.0.5 🧊

Spice v1.0.5 expands Iceberg support with the introduction of the Iceberg Data Connector, in addition to the existing Iceberg Catalog Connector. This new connector enables direct dataset creation and configuration for specific Iceberg objects, enabling federated and accelerated SQL queries on Apache Iceberg tables.

Performance improvements include object-store optimized Parquet pruning in append mode, where object-store metadata is now leveraged alongside Hive partitioning to optimize file pruning. This results in faster and more efficient queries.

DuckDB has been upgraded to v1.2.0, along with additional stability improvements, including improved graceful shutdown and the ability to configure the DuckDB memory limit.

Additional updates include support for the Arrow Map type.

Highlights in v1.0.5​

• New Iceberg Data Connector: Enables direct dataset creation and querying of Iceberg tables.

  Example usage in spicepod.yaml:

  datasets:
    - from: iceberg:https://iceberg-catalog-host.com/v1/namespaces/my_namespace/tables/my_table
      name: my_table
      params:
        # Same as Iceberg Catalog Connector
      acceleration:
        enabled: true
  

  For detailed setup instructions, authentication options, and configuration parameters, refer to the Iceberg Data Connector documentation.

• Improved Parquet pruning in append mode: Uses object-store metadata for more efficient file pruning.

• DuckDB upgrade to v1.2.0 with improved graceful shutdown: Read the DuckDB v1.2.0 announcement for details, including breaking changes for map and list_reduce. Graceful shutdown of DuckDB has been improved for better stability across restarts.

• Configurable DuckDB memory limit: Use the duckdb_memory_limit parameter to set the DuckDB acceleration memory limit:

  - from: spice.ai:path.to.my_dataset
     name: my_dataset
     acceleration:
       params: 
         duckdb_memory_limit: '2GB'
       enabled: true
       engine: duckdb
       mode: file
  

Contributors​

• @peasee
• @phillipleblanc
• @sgrebnov
• @lukekim

Breaking Changes​

• DuckDB v1.2.0 has breaking changes.

Upgrading​

To upgrade to v1.0.5, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.5 image:

docker pull spiceai/spiceai:1.0.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

• duckdb-rs: Upgraded from 1.1.1 to 1.2.0

Changelog​

• fix: Update OpenAI model health check by @peasee in #4849
• fix: Allow metrics endpoint setting in CLI by @peasee in #4939
• DuckDB acceleration: fix Decimal with zero scale support by @sgrebnov in #4922
• Introduce runtime shutdown state by @sgrebnov in #4917
• Add support for Flight and HTTP endpoints configuration to Spice CLI (run and sql) by @sgrebnov and @lukekim in #4913
• Fix Datafusion resources deallocation during shutdown by @sgrebnov in #4912
• DuckDB: fix error handling during record batch insertion by @sgrebnov in #4894
• DuckDB: add support for Map Arrow type for DuckDB acceleration by @sgrebnov in #4887
• Upgrade to DuckDB v1.2.0 by @sgrebnov in #4842
• Gracefully shutdown the runtime and deallocate static resources by @sgrebnov in #4879
• Implement an Iceberg Data Connector by @phillipleblanc in #4941
• Don't trace canceled dataset refresh during runtime termination by @sgrebnov in #4958
• Use metadata column last_modified when specified as a time_column by @phillipleblanc in #4970
• Add duckdb_memory_limit param support for DuckDB acceleration by @sgrebnov in #4971
• Add Iceberg dataset integration test by @phillipleblanc in #4950

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.0.4...v1.0.5

Announcing the release of Spice v1.0.4 🏎️

Spice v1.0.4 improves partition pruning for Delta Lake tables, significantly increasing scan efficiency and reducing overhead. xAI tool calling is more robust and the spice trace CLI command now provides expanded, detailed output for deeper analysis. Additionally, a bug has been fixed to correctly apply column name case-sensitivity in refresh SQL, indexes, and primary keys.

Highlights in v1.0.4​

• Improved Append-Based Refresh When using an append-based acceleration where the time_column format differs from the physical partition, two new dataset configuration options, time_partition_column and time_partition_format can be configured to improve partition pruning and exclude irrelevant partitions during the refreshes.

For example, when the time_column format is timestamp and the physical data partition is date such as below:

my_delta_table/
├── _delta_log/
├── date_col=2023-12-31/
├── date_col=2024-02-04/
├── date_col=2025-01-01/
└── date_col=2030-06-15/

Partition pruning can be optimized using the configuration:

datasets:
  - from: delta_lake://my_delta_table
    name: my_delta_table
    time_column: created_at # A fine-grained timestamp
    time_format: timestamp
    time_partition_column: date_col # Data is physically partitioned by `date_col`
    time_partition_format: date
sgrebnov marked this conversation as resolved.

• Expanded spice trace output: The spice trace CLI command now includes additional details, such as task status, and optional flags --include-input and --include-output for detailed tracing.

Example spice trace output:

TREE                   STATUS DURATION   TASK
a97f52ccd7687e64       ✅       673.14ms ai_chat
  ├── 4eebde7b04321803 ✅         0.04ms tool_use::list_datasets
  └── 4c9049e1bf1c3500 ✅       671.91ms ai_completion

Example spice trace --include-input --include-output output:

TREE                   STATUS DURATION   TASK                    OUTPUT
a97f52ccd7687e64       ✅       673.14ms ai_chat                 The capital of New York is Albany.
  ├── 4eebde7b04321803 ✅         0.04ms tool_use::list_datasets []
  └── 4c9049e1bf1c3500 ✅       671.91ms ai_completion           [{"content":"The capital of New York is Albany.","refusal":null,"tool_calls":null,"role":"assistant","function_call":null,"audio":null}]

Contributors​

• @Jeadie
• @peasee
• @phillipleblanc
• @Sevenannn
• @sgrebnov
• @lukekim

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new recipes.

Upgrading​

To upgrade to v1.0.4, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.4 image:

docker pull spiceai/spiceai:1.0.4

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

No major dependency changes.

Changelog​

- Do not return underlying content of chunked embedding column by default during tool_use::document_similarity by @Jeadie in https://github.com/spiceai/spiceai/pull/4802
- Fix Snowflake Case-Sensitive Identifiers support by @sgrebnov in https://github.com/spiceai/spiceai/pull/4813
- Prepare for 1.0.4 by @sgrebnov in https://github.com/spiceai/spiceai/pull/4801
- Add support for a time_partition_column by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4784
- Prevent the automatic normalization of refresh_sql columns to lowercase by @sgrebnov in https://github.com/spiceai/spiceai/pull/4787
- Implement partition pruning for Delta Lake tables by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4783
- Fix constraint verification for columns with uppercase letters by @sgrebnov in https://github.com/spiceai/spiceai/pull/4785
- Add truncate command for spice trace by @peasee in https://github.com/spiceai/spiceai/pull/4771
- Implement Cache-Control: no-cache to bypass results cache by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4763
- Prompt user to download runtime when running spice sql by @Sevenannn in https://github.com/spiceai/spiceai/pull/4747
- Add vector search tracing by @peasee in https://github.com/spiceai/spiceai/pull/4757
- Update spice trace output format by @Jeadie in https://github.com/spiceai/spiceai/pull/4750
- Fix tool call arguments in Grok messages by @Jeadie in https://github.com/spiceai/spiceai/pull/4741

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v1.0.3...v1.0.4

Announcing the release of Spice v1.0.3 🛠️

Spice v1.0.3 provides several bug fixes, including a fix for the initial data load period when a retention policy has been set, and a new unsupported_type_action: string parameter to auto-convert unsupported types to strings.

Highlights in v1.0.3​

• PostgreSQL Data Connector: New unsupported_type_action: string parameter that auto-converts unsupported types such as JSONB to strings.

Contributors​

• @phillipleblanc
• @Sevenannn
• @sgrebnov
• @peasee
• @Jeadie
• @lukekim

Breaking Changes​

No breaking changes.

Cookbook Updates​

• Updated Kubernetes Deployment Recipe
• Updated Data Retention Recipe

Upgrading​

To upgrade to v1.0.3, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.3 image:

docker pull spiceai/spiceai:1.0.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

No major dependency changes.

Changelog​

- For local models, use 'content=""' instead of None by @Jeadie and @phillipleblanc in https://github.com/spiceai/spiceai/pull/4646
- Perplexity Sonar LLM component by @Jeadie and @lukekim in https://github.com/spiceai/spiceai/pull/4673
- Update async openai fork & support reasoning effort parameter by @Sevenannn and @phillipleblanc in https://github.com/spiceai/spiceai/pull/4679
- Web search tool by @Jeadie and @lukekim in https://github.com/spiceai/spiceai/pull/4687
- Setup tpc-extension by @ewgenius and @phillipleblanc in https://github.com/spiceai/spiceai/pull/4690
- fix: Use PostgreSQL interval style for Spice.ai by @peasee and @phillipleblanc in https://github.com/spiceai/spiceai/pull/4716
- Fix spice upgrade command by @Sevenannn and @sgrebnov in https://github.com/spiceai/spiceai/pull/4699
- Fix bug: Ensure refresh only retrieves data within the retention period by @sgrebnov and @phillipleblanc in https://github.com/spiceai/spiceai/pull/4717
- Implement unsupported_type_action: string for Postgres JSONB support by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4719
- Fix the get latest release logic by @Sevenannn and @phillipleblanc in https://github.com/spiceai/spiceai/pull/4721
- add 'accelerated_refresh' to 'spice trace' allowlist by @Jeadie and @phillipleblanc in https://github.com/spiceai/spiceai/pull/4711
- Update version to 1.0.3 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4731
- Truncate embedding columns within sampling tool by @Jeadie in https://github.com/spiceai/spiceai/pull/4722
- Validate primary key columns during accelerated dataset initialization by @sgrebnov in https://github.com/spiceai/spiceai/pull/4736

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v1.0.2...v1.0.3

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

• Twitter: @spice_ai
• Discord: https://discord.gg/kZnTfneP5u
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]