Skip to main content
William Croxson
Senior Software Engineer at Spice AI
View all authors

Spice v1.0-stable (Jan 20, 2025)

ยท 8 min read
William Croxson
Senior Software Engineer at Spice AI

๐ŸŽ‰ After 47 releases, Spice.ai OSS has reached production readiness with the 1.0-stable milestone!

The core runtime and features such as query federation, query acceleration, catalog integration, search and AI-inference have all graduated to stable status along with key component graduations across data connectors, data accelerators, catalog connectors, and AI model providers.

Highlights in v1.0-stableโ€‹

Breaking Changesโ€‹

  • Default Runtime Version: The CLI will install the GPU accelerated AI-capable Runtime by default (if supported), when running spice install or spice run. To force-install the non-GPU version, run spice install ai --cpu.

  • Default OpenAI Model: The default OpenAI model has updated to gpt-4o-mini.

  • Identifier Normalization: Unquoted identifiers such as table names are no longer normalized to lowercase. Identifiers will now retain their exact case as provided.

  • Sandboxed Docker Image: The Runtime Docker Image now runs the spiced process as the nobody user in a minimal chroot sandbox.

  • Insecure S3 and ABFS endpoints: The S3 and ABFS connectors now enforce insecure endpoint checks, preventing HTTP endpoints unless allow_http is explicitly enabled. Refer to the documentation for details.

Dependenciesโ€‹

No major dependency changes.

Upgradingโ€‹

To upgrade to v1.0.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.0 image:

docker pull spiceai/spiceai:1.0.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

Contributorsโ€‹

  • @peasee
  • @ewgenius
  • @Jeadie
  • @Sevenannn
  • @lukekim
  • @phillipleblanc
  • @sgrebnov

What's Changedโ€‹

- feat: Update load test criteria, testoperator updates by @peasee in <https://github.com/spiceai/spiceai/pull/4311>
- Update helm for v1.0.0-rc.5 by @ewgenius in <https://github.com/spiceai/spiceai/pull/4313>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4318>
- Bump version to v1.0.0, update SECURITY.md by @ewgenius in <https://github.com/spiceai/spiceai/pull/4314>
- Initial criteria for models, embeddings by @Jeadie in <https://github.com/spiceai/spiceai/pull/4223>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/4321>
- Add dremio param for running load test by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4315>
- Promote Databricks (mode: delta_lake) connector to stable by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4328>
- Handle failed query in load test by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4327>
- feat: Use load test hours for baseline query sets by @peasee in <https://github.com/spiceai/spiceai/pull/4334>
- Fix typo in 1.0.0-rc.5 release notes by @ewgenius in <https://github.com/spiceai/spiceai/pull/4329>
- feat: add testoperator data consistency by @peasee in <https://github.com/spiceai/spiceai/pull/4319>
- docs: Release DuckDB connector stable by @peasee in <https://github.com/spiceai/spiceai/pull/4335>
- Fix DocumentDB -> DynamoDB by @lukekim in <https://github.com/spiceai/spiceai/pull/4339>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/4337>
- fix: Download hits.parquet from MinIO for benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/4338>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4341>
- Remove evil averages by @lukekim in <https://github.com/spiceai/spiceai/pull/4343>
- Don't run builds on non-code changes by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4344>
- Remove streaming requirement from Databricks spark Beta and Spark connector Beta by @ewgenius in <https://github.com/spiceai/spiceai/pull/4345>
- Update s3 tpcds spicepods by @ewgenius in <https://github.com/spiceai/spiceai/pull/4346>
- Explicitly set required scale factor for throughput and load tests by @ewgenius in <https://github.com/spiceai/spiceai/pull/4347>
- Fix s3 tpcds dataset name by @ewgenius in <https://github.com/spiceai/spiceai/pull/4348>
- Promote Iceberg Catalog Connector to Beta by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4350>
- Update s3 clickbench benchmark snapshots by @ewgenius in <https://github.com/spiceai/spiceai/pull/4351>
- fix: DuckDB clickbench on zero results by @peasee in <https://github.com/spiceai/spiceai/pull/4349>
- Add integration test with snapshots for databricks catalog connector by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4353>
- refactor: Remove on zero results from benchmarks, add data consistency workflow by @peasee in <https://github.com/spiceai/spiceai/pull/4354>
- Fix Bug: No field named body_embedding when do vector search with refresh sql containing subset of columns by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4297>
- docs: Update roadmap by @peasee in <https://github.com/spiceai/spiceai/pull/4364>
- feat: Release accelerators stable by @peasee in <https://github.com/spiceai/spiceai/pull/4361>
- Add TPCH/TPCDS test spicepods for MySQL by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4365>
- Catch when an insecure (http) S3 and ABFS data connectors endpoint is used without specifying the `allow_http` parameter by @ewgenius in <https://github.com/spiceai/spiceai/pull/4363>
- Update ROADMAP - Iceberg catalog alpha for v1.0 by @ewgenius in <https://github.com/spiceai/spiceai/pull/4367>
- Promote databricks catalog and databricks (spark_connect) connector to beta by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4369>
- Update Roadmap - Iceberg beta by @ewgenius in <https://github.com/spiceai/spiceai/pull/4373>
- Build CUDA binaries for Linux by @Jeadie in <https://github.com/spiceai/spiceai/pull/4320>
- Promote Nvidia NIM as Alpha by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4380>
- Promote xai to alpha by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4381>
- Update stable criteria for object store based connectors by @ewgenius in <https://github.com/spiceai/spiceai/pull/4383>
- Testoperator: http consistency and overhead tests, fixes and ci by @ewgenius in <https://github.com/spiceai/spiceai/pull/4382>
- Promote S3 Data Connector to Stable by @ewgenius in <https://github.com/spiceai/spiceai/pull/4385>
- Download platform-supported CUDA binary version on Linux by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4356>
- Fix http consistency test workflow, add overhead workflow by @ewgenius in <https://github.com/spiceai/spiceai/pull/4387>
- feat: Add Postgres test spicepods by @peasee in <https://github.com/spiceai/spiceai/pull/4388>
- Fix typos + specific in model criteria; Make explicit alpha/beta tests for LLMS in `crates/llms/tests`. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4377>
- Fix federation bug for correlated subqueries of deeply nested Dremio tables by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4389>
- Fix http overhead workflow by @ewgenius in <https://github.com/spiceai/spiceai/pull/4390>
- Tweak model tests, fix embedding input by @ewgenius in <https://github.com/spiceai/spiceai/pull/4391>
- Promote Dremio to Stable quality by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4392>
- Add beta functionality tests for embedding models. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4352>
- docs: Release postgres connector stable by @peasee in <https://github.com/spiceai/spiceai/pull/4398>
- Increase timeout for model response in E2E tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4399>
- Disable ident normalization (i.e. `SELECT MyColumn from table` works) by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4400>
- Preserve schema metadata by @ewgenius in <https://github.com/spiceai/spiceai/pull/4402>
- Make models integration tests tracing less verbose by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4403>
- Fix `cuda` feature build on Windows by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4404>
- Promote MySQL to Stable by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4406>
- docs: Release Delta Lake and Unity catalog by @peasee in <https://github.com/spiceai/spiceai/pull/4405>
- Use `gpt-4o-mini` as a default model for openai provider by @ewgenius in <https://github.com/spiceai/spiceai/pull/4410>
- Fix streaming for Openai and Anthropic by @Jeadie in <https://github.com/spiceai/spiceai/pull/4409>
- Tweak model loading and missing tool errors messages by @ewgenius in <https://github.com/spiceai/spiceai/pull/4412>
- Spice CLI: fallback to CPU build for unsupported GPU Compute Capability by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4407>
- Build Windows CUDA binaries as part of `build_and_release` workflow by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4386>
- Update docs link by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4416>
- feat: Add CPU models install escape hatch by @peasee in <https://github.com/spiceai/spiceai/pull/4419>
- Handle OpenAI API Errors by @ewgenius in <https://github.com/spiceai/spiceai/pull/4417>
- Update spice cli to use `GH_TOKEN` or `GITHUB_TOKEN` env variables when calling releases api by @ewgenius in <https://github.com/spiceai/spiceai/pull/4175>
- Implement secure sandboxing for Docker image by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4411>
- Automatically install supported CUDA binary on Windows by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4420>
- Metrics for LLMs+ embeddings by @Jeadie in <https://github.com/spiceai/spiceai/pull/4418>
- Jeadie/25 01 17/beta perf by @Jeadie in <https://github.com/spiceai/spiceai/pull/4397>
- Pass GitHub token to all CI steps calling spice run by @ewgenius in <https://github.com/spiceai/spiceai/pull/4423>
- Run the models integration tests on PRs by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4421>
- Run CUDA builds in a separate workflow by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4430>
- Promote OpenAI models and embeddings providers to RC by @ewgenius in <https://github.com/spiceai/spiceai/pull/4432>
- Update link to retrieval-augmented generation (RAG) details by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4433>
- Unity catalog should strip parameter prefix before passing parameters to delta lake factory by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4436>
- Update quickstart traces to match current version by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4435>
- Update Supported Embeddings Providers Readme section by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4434>
- Local models can stream tools by @Jeadie in <https://github.com/spiceai/spiceai/pull/4429>
- fix: Use MetricsCollector::show() for HTTP testoperator commands by @peasee in <https://github.com/spiceai/spiceai/pull/4442>
- Fix run query action by @ewgenius in <https://github.com/spiceai/spiceai/pull/4444>
- Default to AI-enabled runtime for `spice run`/`spice install` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4443>
- Change no spicepod.yaml log to warning by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4447>
- refactor: Update Catalog Connector error messages by @peasee in <https://github.com/spiceai/spiceai/pull/4441>
- Fix panic when converting OTel metrics by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4449>
- refactor: Update model errors by @peasee in <https://github.com/spiceai/spiceai/pull/4446>
- Update spiceai/mistral.rs to silence metadata logs by @ewgenius in <https://github.com/spiceai/spiceai/pull/4452>
- fix xAI; don't use openai defaults by @Jeadie in <https://github.com/spiceai/spiceai/pull/4450>
- Improves the UX of using huggingface models by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4451>
- Add GH Workflow to test `spice ai` runtime installation by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4448>
- fix: Use specific model errors where available by @peasee in <https://github.com/spiceai/spiceai/pull/4454>
- Detect and report unsupported embedding column type during dataset registration by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4456>
- Handle Errors by @Jeadie in <https://github.com/spiceai/spiceai/pull/4455>
- Catch and report negative openai_temperature error by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4453>
- Clarify release check error message if it is caused by wrong GH token by @ewgenius in <https://github.com/spiceai/spiceai/pull/4458>

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v1.0.0-rc.5...v1.0.0>

Resourcesโ€‹

Communityโ€‹

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

Spice v1.0-rc.2 (Dec 16, 2024)

ยท 8 min read
William Croxson
Senior Software Engineer at Spice AI

Announcing the release of Spice v1.0-rc.2 ๐Ÿ”—

Spice v1.0.0-rc.2 is the second release candidate for the first major version of Spice.ai OSS. This release continues to build on the stability of Spice for production use, including key Data Connector graduations, bug fixes, and AI features.

Highlights in v1.0-rc.2โ€‹

  • MS SQL and File Data Connectors: Graduated from Alpha to Beta.

  • GraphQL and Databricks Delta Lake Data Connectors: Graduated from Beta to Release Candidate.

  • gospice SDK Release: The Spice Go SDK has updated to v7.0, adding support for refreshing datasets and upgrading dependencies.

  • Azure AI Support: Added support for both LLMs and embedding models. Example spicepod.yml configuration:

embeddings:
- name: azure
from: azure:text-embedding-3-small
params:
endpoint: https://your-resource-name.openai.azure.com
azure_api_version: 2024-08-01-preview
azure_deployment_name: text-embedding-3-small
azure_api_key: ${ secrets:SPICE_AZURE_API_KEY }
models:
- name: azure
from: azure:gpt-4o-mini
params:
endpoint: https://your-resource-name.openai.azure.com
azure_api_version: 2024-08-01-preview
azure_deployment_name: gpt-4o-mini
azure_api_key: ${ secrets:SPICE_AZURE_TOKEN }

Accelerate subsets of columns: Spice now supports acceleration for specific columns from a federated source. Specify the desired columns directly in the Refresh SQL for more selective and efficient data acceleration.

Example spicepod.yaml configuration:

datasets:
- from: s3://spiceai-demo-datasets/taxi_trips/2024/
name: taxi_trips
params:
file_format: parquet
acceleration:
refresh_sql: SELECT tpep_pickup_datetime, tpep_dropoff_datetime, trip_distance, total_amount FROM taxi_trips

Breaking changesโ€‹

Sharepoint Authentication Parameters: now use access tokens instead of authorization codes, using the sharepoint_bearer_token parameter. The sharepoint_auth_code parameter has been removed.

Data Connector Delimiters: now support / and ://, in addition to : in the from parameter of the dataset configuration. The following examples are equivalent:

  • from: postgres://my_postgres_table
  • from: postgres/my_postgres_table
  • from: postgres:my_postgres_table

Some data connectors, such as s3 which only accepts ://, place further restrictions on the allowed delimiter.

The file data connector has changed how it interprets the :// delimiter to reflect how most other URL parsers work, i.e. file://my_file_path. Previously, the file path was interpreted as /my_file_path. Now, it is interpreted as a relative path, i.e. my_file_path.

Spice Search limit: is now applied to the final search result, instead of previously being applied separately to each dataset involved in a search before aggregation.

Dependenciesโ€‹

  • Rust: Upgraded to 1.83

Contributorsโ€‹

  • @phillipleblanc
  • @ewgenius
  • @Jeadie
  • @sgrebnov
  • @peasee
  • @Sevenannn
  • @Advayp

New Contributorsโ€‹

What's Changedโ€‹

- Fix install scripts to handle the RC release by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3718>
- Update helm chart to v1.0.0-rc.1 by @ewgenius in <https://github.com/spiceai/spiceai/pull/3720>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/3719>
- Add logic to ignore task cancellations due to runtime shutdown by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3717>
- Update to next relese version v1.0.0-rc.2 by @ewgenius in <https://github.com/spiceai/spiceai/pull/3721>
- Handle parsing OTel KeyValues from the `baggage` header by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3722>
- Update `llms` dependencies: `mistralrs`, `async-openai` by @Jeadie in <https://github.com/spiceai/spiceai/pull/3725>
- Support `jsonl` for object store by @Jeadie in <https://github.com/spiceai/spiceai/pull/3726>
- Fix NSQL models integration tests for HF by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3727>
- standardise 'csv_schema_infer_max_records' -> 'schema_infer_max_records'; include deprecation messages for dataset params by @Jeadie in <https://github.com/spiceai/spiceai/pull/3732>
- feat: Add script to generate TPC-H data for file connector by @peasee in <https://github.com/spiceai/spiceai/pull/3737>
- feat: Add file connector integration test by @peasee in <https://github.com/spiceai/spiceai/pull/3735>
- fix: Add explicit message for ODBC connector when not installed by @peasee in <https://github.com/spiceai/spiceai/pull/3736>
- Remove Box::leak in `create_accelerated_table` by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3739>
- docs: Update enhancement and PR template by @peasee in <https://github.com/spiceai/spiceai/pull/3740>
- feat: add file connector benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/3734>
- docs: Release file connector beta by @peasee in <https://github.com/spiceai/spiceai/pull/3738>
- For embeddings, use `sentence_*_config.json`, download HF async, use TEI functions by @Jeadie in <https://github.com/spiceai/spiceai/pull/3724>
- Optimize build & release workflow for trunk builds by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3741>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/3752>
- Skip Spice cloud integration tests by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3755>
- Add `http_requests` metric and deprecate `http_requests_total` by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3748>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/3759>
- fix: Parquet file generation script by @peasee in <https://github.com/spiceai/spiceai/pull/3762>
- fix: Use InvalidConfiguration error for GraphQL query errors by @peasee in <https://github.com/spiceai/spiceai/pull/3763>
- Extend Spice Search integration and E2E tests to cover chunking by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3750>
- test: Add GraphQL integration tests from external sources by @peasee in <https://github.com/spiceai/spiceai/pull/3756>
- docs: Release GraphQL release candidate by @peasee in <https://github.com/spiceai/spiceai/pull/3764>
- Accelerate a subset of columns from source dataset in Refresh SQL by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3765>
- Run TPCDS benchmark for databricks delta mode by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3751>
- Update dependencies by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3747>
- Implement vector search benchmark initialization by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3774>
- Implement InvalidTypeAction for PostgreSQL Data Connector by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3767>
- fix: Check ODBC parameters are positive integers by @peasee in <https://github.com/spiceai/spiceai/pull/3777>
- Fix Delta DataType `Map` type mapping to arrow type by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3776>
- Update Databricks & Delta Lake Connector RC criteria by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3778>
- Add a `/v1/packages/generate` API to generate a Spicepod package from a GitHub repo. by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3782>
- Set `Spice-Target-Source` header for `spice add` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3783>
- Call v1 spicerack API by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3784>
- Run models integration tests on self-hosted macOS runners by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3785>
- Fix OpenAI models integration tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3786>
- Integration test for Databricks delta_lake mode by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3779>
- Add `spice connect` for connecting to existing Spice.ai instances by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3790>
- Add `eval` spicepod component; basic HTTP api to run eval. by @Jeadie in <https://github.com/spiceai/spiceai/pull/3766>
- Release RC for databricks delta_lake mode by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3792>
- Include Huggingface model to E2E models tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3788>
- Enable `trace_id` & `parent_span_id` overrides for `v1/chat/completion` by @Jeadie in <https://github.com/spiceai/spiceai/pull/3791>
- Search benchmark: run search workload and measure result by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3793>
- Search benchmark: measure search precision by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3804>
- Use MinIO instead of S3 for benchmark tests by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3794>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/3814>
- Only verify TPCH / TPCDS official query results for DuckDB by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3816>
- Fixes for the Debezium connector by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3819>
- Fix insert statement when all columns are constraint columns by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3820>
- docs: Move ODBC to Beta for current state of roadmap by @peasee in <https://github.com/spiceai/spiceai/pull/3823>
- Accept `:`, `/` or `://` as the delimiter for the data connector by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3821>
- Update dependencies by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3826>
- Enable `read_write` mode support for Postgres Data Connector by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3813>
- feat: add Databricks ODBC TPCDS benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/3825>
- Change `spice.ai` data connector dataset path format to `<org>/<app>/datasets/<table_reference>` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3828>
- fix: enable tpcds explain snapshotting by @peasee in <https://github.com/spiceai/spiceai/pull/3830>
- Azure AI support for both LLMs & embedding models by @Jeadie in <https://github.com/spiceai/spiceai/pull/3824>
- Add Github Workflow to run Search Benchmark by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3834>
- Fetch access token with Microsoft OAuth, and use access token to initiate Sharepoint data connector graph client by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3836>
- Initialize accelerator for datasets dynamically included by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3714>
- Update cargo.lock by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3838>
- feat: add MS SQL TPCH benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/3833>
- Improve Azure AI models support by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3835>
- Primary key support for Arrow's `Memtable` by @Jeadie in <https://github.com/spiceai/spiceai/pull/3829>
- Update Tokenizer to 0.21 and mistral.rs by @Jeadie in <https://github.com/spiceai/spiceai/pull/3839>
- Fix models integration tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3843>
- Enable `spice login abfs` by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3844>
- update `crates/llms` dependencies to 'spiceai' branch by @Jeadie in <https://github.com/spiceai/spiceai/pull/3846>
- Make eval runs non-blocking; `spice.eval.{results, runs}` tables. by @Jeadie in <https://github.com/spiceai/spiceai/pull/3780>
- fix: Update GraphQL snapshots by @peasee in <https://github.com/spiceai/spiceai/pull/3849>
- Update to Rust 1.83 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3847>
- feat: add mssql integration test by @peasee in <https://github.com/spiceai/spiceai/pull/3848>
- Prepend user-specified user agent in flight repl by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3850>
- fix: trim CHAR in mssql by @peasee in <https://github.com/spiceai/spiceai/pull/3852>
- Fix column quoting for SpiceCloudPlatform dialect by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3857>
- Optimize builds by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3861>
- Endgame template: Add recently added AI/ML quickstarts and samples by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3859>
- docs: Release MS SQL Beta by @peasee in <https://github.com/spiceai/spiceai/pull/3853>
- Fix nsql sampling for tables with embeddings by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3860>
- Make GH workflows with spiceai-macos runners more stable by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3863>
- fix: Remove GraphQL swapi test by @peasee in <https://github.com/spiceai/spiceai/pull/3867>
- create 1 `tokio::test` per test/model by @Jeadie in <https://github.com/spiceai/spiceai/pull/3696>
- handle `max_completion_tokens` vs `max_tokens` for openai vs azure by @Jeadie in <https://github.com/spiceai/spiceai/pull/3869>
- Search benchmark: write results to dataset by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3871>
- Create `evalconverter` that creates spice eval components. by @Jeadie in <https://github.com/spiceai/spiceai/pull/3864>
- Update quickstart in README.md by @ewgenius in <https://github.com/spiceai/spiceai/pull/3876>
- Remove reference to spiceai-smart-demo from the repo home by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3885>
- Trace `evals` accelerated tables updates in debug mode by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3884>
- Clarify confusing log message by @Advayp in <https://github.com/spiceai/spiceai/pull/3862>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/3840>
- Azure OpenAI models: make `endpoint` parameter required by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3883>
- Use spiceai delta kernel fork, actionable message for delta checkpoint errors by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3856>
- Add support for GGUF files in HF by @Jeadie in <https://github.com/spiceai/spiceai/pull/3875>

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v1.0.0-rc.1...v1.0.0-rc.2>
```text

## Resources

- [Getting started with Spice.ai](https://docs.spiceai.org/getting-started/)
- [Documentation](https://docs.spiceai.org/)

## Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

- Twitter: [@spice_ai](https://twitter.com/spice_ai)
- Discord: [https://discord.gg/kZnTfneP5u](https://discord.gg/kZnTfneP5u)
- Telegram: [Spice AI Discussion](https://t.me/spiceaichat)
- Reddit: [https://www.reddit.com/r/spiceai](https://www.reddit.com/r/spiceai)
- Email: [[email protected]](mailto:[email protected])

Spice v0.19-beta (Oct 7, 2024)

ยท 6 min read
William Croxson
Senior Software Engineer at Spice AI

Announcing the release of Spice v0.19-beta ๐Ÿ“ฆ

Spice v0.19.0-beta brings performance improvements for accelerators and expanded TPC-DS coverage. A new Azure Blob Storage data connector has also been added.

Highlights in v0.19.0-betaโ€‹

Improved TPC-DS Coverage: Enhanced support for TPC-DS derived queries.

CLI SQL REPL: The CLI SQL REPL (spice sql) now supports multi-line editing and tab indentation. Note, a terminating semi-colon ';' is now required for each executed SQL block.

Azure Storage Data Connector: A new Azure Blob Storage data connector (abfs://) has been added, enabling federated SQL queries on files stored in Azure Blob-compatible endpoints, including Azure BlobFS (abfss://) and Azure Data Lake (adl://). Supported file formats can be specified using the file_format parameter.

Example spicepod.yml:

datasets:
- from: abfs://foocontainer/taxi_sample.csv
name: azure_test
params:
azure_account: spiceadls
azure_access_key: abc123==
file_format: csv

For a full list of supported files, see the Object Store File Formats documentation.

For more details, see the Azure Blob Storage Data Connector documentation.

Breaking Changesโ€‹

  • Spice.ai Data Connector: The key for the Spice.ai Cloud Platform Data Connector has changed from spiceai to spice.ai. To upgrade, change uses of from: spiceai: to from: spice.ai:.

  • GitHub Data Connector: Pull Requests column login has been renamed to author.

  • CLI SQL REPL: A terminating semi-colon ';' is now required for each executed SQL block.

  • Spicepod Hot-Reload: When running spiced directly, hot-reload of spicepod.yml configuration is now disabled. Run with spice run to use hot-reload.

Contributorsโ€‹

  • @sgrebnov
  • @Jeadie
  • @Sevenannn
  • @peasee
  • @ewgenius
  • @slyons
  • @phillipleblanc
  • @lukekim

Dependenciesโ€‹

What's Changedโ€‹

- Bump tonic from 0.12.2 to 0.12.3 by @dependabot in https://github.com/spiceai/spiceai/pull/2880
- Verify benchmark query results using snapshot testing (s3 connector) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2902
- Fix `paths-ignore:` by @Jeadie in https://github.com/spiceai/spiceai/pull/2906
- Rename `spiceai` data connector to `spice.ai` by @sgrebnov in https://github.com/spiceai/spiceai/pull/2899
- Update ROADMAP.md by @Jeadie in https://github.com/spiceai/spiceai/pull/2907
- Helm update for helm for 0.18.3-beta by @Jeadie in https://github.com/spiceai/spiceai/pull/2910
- Add tpcds queries by @Sevenannn in https://github.com/spiceai/spiceai/pull/2918
- Fix `paths-ignore` for docs. by @Jeadie in https://github.com/spiceai/spiceai/pull/2911
- feat: Support LIKE expressions in GitHub filter pushdown by @peasee in https://github.com/spiceai/spiceai/pull/2903
- feat: Support date comparison pushdown in GitHub connector by @peasee in https://github.com/spiceai/spiceai/pull/2904
- Improve aggregation and union queries unparsing by @sgrebnov in https://github.com/spiceai/spiceai/pull/2925
- Initialize file based accelerators on dataset reload by @Sevenannn in https://github.com/spiceai/spiceai/pull/2923
- Update spiceai/spiceai for next release by @Jeadie in https://github.com/spiceai/spiceai/pull/2928
- Verify TPC-H benchmark query results for arrow acceleration by @sgrebnov in https://github.com/spiceai/spiceai/pull/2927
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/2912
- Use structured output for NSQL by @Jeadie in https://github.com/spiceai/spiceai/pull/2922
- Update TPC-DS queries to use supported date addition format by @sgrebnov in https://github.com/spiceai/spiceai/pull/2930
- Add busy_timeout accelerator param for Sqlite by @Sevenannn in https://github.com/spiceai/spiceai/pull/2855
- Use Cosine Similarity in vector search by @Jeadie in https://github.com/spiceai/spiceai/pull/2932
- Add support for passing `x-spiceai-app-id` metadata in spiceai data connector by @ewgenius in https://github.com/spiceai/spiceai/pull/2934
- docs: update beta accelerator criteria by @peasee in https://github.com/spiceai/spiceai/pull/2905
- Azure Connector implementation by @slyons in https://github.com/spiceai/spiceai/pull/2926
- Local embedding model from relative paths by @Jeadie in https://github.com/spiceai/spiceai/pull/2908
- Add Markdown aware chunker when `params.file_format: md`. by @Jeadie in https://github.com/spiceai/spiceai/pull/2943
- 'spice version' without structured logging by @Jeadie in https://github.com/spiceai/spiceai/pull/2944
- Bump tempfile from 3.12.0 to 3.13.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2878
- feat: GraphQL commit query parameters by @peasee in https://github.com/spiceai/spiceai/pull/2945
- Update OpenAI client and use new request fields by @Jeadie in https://github.com/spiceai/spiceai/pull/2951
- refactor: Rename GitHub pulls login to author by @peasee in https://github.com/spiceai/spiceai/pull/2954
- Run tpcds benchmarks for accelerators by @Sevenannn in https://github.com/spiceai/spiceai/pull/2853
- Add spiced arg `--pods-watcher-enabled`. Watcher disabled by default for spiced. by @ewgenius in https://github.com/spiceai/spiceai/pull/2953
- Add error message when spicepod has embeddings or models without '--features models' by @Jeadie in https://github.com/spiceai/spiceai/pull/2952
- Adding multi-line editing and tab indentation to sql REPL by @slyons in https://github.com/spiceai/spiceai/pull/2949
- Update MySQL ghcr image to include tpcds data by @Sevenannn in https://github.com/spiceai/spiceai/pull/2941
- Document DataFusion limitation: The context only support single SQL Statement, Date Arithmetic like date + 3 not supported by @Sevenannn in https://github.com/spiceai/spiceai/pull/2970
- Bump snafu from 0.8.4 to 0.8.5 by @dependabot in https://github.com/spiceai/spiceai/pull/2876
- Bump async-trait from 0.1.82 to 0.1.83 by @dependabot in https://github.com/spiceai/spiceai/pull/2879
- Bump async-graphql from 7.0.9 to 7.0.11 in the cargo group by @dependabot in https://github.com/spiceai/spiceai/pull/2950
- Verify TPC-H benchmark query results for MySQL by @sgrebnov in https://github.com/spiceai/spiceai/pull/2972
- Verify TPCH benchmark query results for Postgres by @sgrebnov in https://github.com/spiceai/spiceai/pull/2973
- Verify TPCH benchmark query results for sqlite acceleration by @sgrebnov in https://github.com/spiceai/spiceai/pull/2974
- Verify TPCH benchmark query results for duckdb (in-memory) acceleration by @sgrebnov in https://github.com/spiceai/spiceai/pull/2975
- Support for `mdx` file extensions to apply a markdown splitter by @ewgenius in https://github.com/spiceai/spiceai/pull/2977
- Don't assume first vector or content will be non-null/zero by @Jeadie in https://github.com/spiceai/spiceai/pull/2940
- use custom chunk sizers for HF, local and OpenAI models by @Jeadie in https://github.com/spiceai/spiceai/pull/2971
- Ensure we return N unique documents, not N unique chunks by @Jeadie in https://github.com/spiceai/spiceai/pull/2976
- Fix issues parsing `messages[*].tool_calls` for local models by @Jeadie in https://github.com/spiceai/spiceai/pull/2957
- text -> SQL trait to customise per model. by @Jeadie in https://github.com/spiceai/spiceai/pull/2942
- Remove system message from ToolUsingChat. by @Jeadie in https://github.com/spiceai/spiceai/pull/2978
- Make logical plan to sql more robust (improve ORDER BY; support `round` for Postgres) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2984
- Add connection_pool_size parameter for Postgres accelerator by @Sevenannn in https://github.com/spiceai/spiceai/pull/2969
- Fix dataset configure prompt by @sgrebnov in https://github.com/spiceai/spiceai/pull/2991
- Verify TPCH benchmark query results for Databricks(odbc) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2989
- Verify TPCH benchmark query results for Databricks (delta_lake) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2982
- Set log level for anonymous telemetry traces to `trace` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2995
- Improvements to issue templates by @lukekim in https://github.com/spiceai/spiceai/pull/2992
- `spice login` writes to `.env.local` if present by @slyons in https://github.com/spiceai/spiceai/pull/2996

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v0.18.3-beta...v0.19.0-beta>

Resourcesโ€‹

Communityโ€‹

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.