Spice v1.0-rc.2 (Dec 16, 2024)
· 8 min read
Announcing the release of Spice v1.0-rc.2 🔗
Spice v1.0.0-rc.2 is the second release candidate for the first major version of Spice.ai OSS. This release continues to build on the stability of Spice for production use, including key Data Connector graduations, bug fixes, and AI features.
Highlights in v1.0-rc.2
-
MS SQL and File Data Connectors: Graduated from Alpha to Beta.
-
GraphQL and Databricks Delta Lake Data Connectors: Graduated from Beta to Release Candidate.
-
gospice SDK Release: The Spice Go SDK has updated to v7.0, adding support for refreshing datasets and upgrading dependencies.
-
Azure AI Support: Added support for both LLMs and embedding models. Example
spicepod.ymlconfiguration:
embeddings:
- name: azure
from: azure:text-embedding-3-small
params:
endpoint: https://your-resource-name.openai.azure.com
azure_api_version: 2024-08-01-preview
azure_deployment_name: text-embedding-3-small
azure_api_key: ${ secrets:SPICE_AZURE_API_KEY }
models:
- name: azure
from: azure:gpt-4o-mini
params:
endpoint: https://your-resource-name.openai.azure.com
azure_api_version: 2024-08-01-preview
azure_deployment_name: gpt-4o-mini
azure_api_key: ${ secrets:SPICE_AZURE_TOKEN }
Accelerate subsets of columns: Spice now supports acceleration for specific columns from a federated source. Specify the desired columns directly in the Refresh SQL for more selective and efficient data acceleration.
Example spicepod.yaml configuration:
datasets:
- from: s3://spiceai-demo-datasets/taxi_trips/2024/
name: taxi_trips
params:
file_format: parquet
acceleration:
refresh_sql: SELECT tpep_pickup_datetime, tpep_dropoff_datetime, trip_distance, total_amount FROM taxi_trips
Breaking changes
Sharepoint Authentication Parameters: now use access tokens instead of authorization codes, using the sharepoint_bearer_token parameter. The sharepoint_auth_code parameter has been removed.
Data Connector Delimiters: now support / and ://, in addition to : in the from parameter of the dataset configuration. The following examples are equivalent:
from: postgres://my_postgres_tablefrom: postgres/my_postgres_tablefrom: postgres:my_postgres_table
Some data connectors, such as s3 which only accepts ://, place further restrictions on the allowed delimiter.
The file data connector has changed how it interprets the :// delimiter to reflect how most other URL parsers work, i.e. file://my_file_path. Previously, the file path was interpreted as /my_file_path. Now, it is interpreted as a relative path, i.e. my_file_path.
Spice Search limit: is now applied to the final search result, instead of previously being applied separately to each dataset involved in a search before aggregation.
Dependencies
- Rust: Upgraded to 1.83
Contributors
- @phillipleblanc
- @ewgenius
- @Jeadie
- @sgrebnov
- @peasee
- @Sevenannn
- @Advayp
New Contributors
- @Advayp made their first contribution in github.com/spiceai/spiceai/pull/3862
What's Changed
- Fix install scripts to handle the RC release by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3718>
- Update helm chart to v1.0.0-rc.1 by @ewgenius in <https://github.com/spiceai/spiceai/pull/3720>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/3719>
- Add logic to ignore task cancellations due to runtime shutdown by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3717>
- Update to next relese version v1.0.0-rc.2 by @ewgenius in <https://github.com/spiceai/spiceai/pull/3721>
- Handle parsing OTel KeyValues from the `baggage` header by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3722>
- Update `llms` dependencies: `mistralrs`, `async-openai` by @Jeadie in <https://github.com/spiceai/spiceai/pull/3725>
- Support `jsonl` for object store by @Jeadie in <https://github.com/spiceai/spiceai/pull/3726>
- Fix NSQL models integration tests for HF by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3727>
- standardise 'csv_schema_infer_max_records' -> 'schema_infer_max_records'; include deprecation messages for dataset params by @Jeadie in <https://github.com/spiceai/spiceai/pull/3732>
- feat: Add script to generate TPC-H data for file connector by @peasee in <https://github.com/spiceai/spiceai/pull/3737>
- feat: Add file connector integration test by @peasee in <https://github.com/spiceai/spiceai/pull/3735>
- fix: Add explicit message for ODBC connector when not installed by @peasee in <https://github.com/spiceai/spiceai/pull/3736>
- Remove Box::leak in `create_accelerated_table` by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3739>
- docs: Update enhancement and PR template by @peasee in <https://github.com/spiceai/spiceai/pull/3740>
- feat: add file connector benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/3734>
- docs: Release file connector beta by @peasee in <https://github.com/spiceai/spiceai/pull/3738>
- For embeddings, use `sentence_*_config.json`, download HF async, use TEI functions by @Jeadie in <https://github.com/spiceai/spiceai/pull/3724>
- Optimize build & release workflow for trunk builds by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3741>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/3752>
- Skip Spice cloud integration tests by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3755>
- Add `http_requests` metric and deprecate `http_requests_total` by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3748>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/3759>
- fix: Parquet file generation script by @peasee in <https://github.com/spiceai/spiceai/pull/3762>
- fix: Use InvalidConfiguration error for GraphQL query errors by @peasee in <https://github.com/spiceai/spiceai/pull/3763>
- Extend Spice Search integration and E2E tests to cover chunking by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3750>
- test: Add GraphQL integration tests from external sources by @peasee in <https://github.com/spiceai/spiceai/pull/3756>
- docs: Release GraphQL release candidate by @peasee in <https://github.com/spiceai/spiceai/pull/3764>
- Accelerate a subset of columns from source dataset in Refresh SQL by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3765>
- Run TPCDS benchmark for databricks delta mode by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3751>
- Update dependencies by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3747>
- Implement vector search benchmark initialization by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3774>
- Implement InvalidTypeAction for PostgreSQL Data Connector by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3767>
- fix: Check ODBC parameters are positive integers by @peasee in <https://github.com/spiceai/spiceai/pull/3777>
- Fix Delta DataType `Map` type mapping to arrow type by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3776>
- Update Databricks & Delta Lake Connector RC criteria by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3778>
- Add a `/v1/packages/generate` API to generate a Spicepod package from a GitHub repo. by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3782>
- Set `Spice-Target-Source` header for `spice add` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3783>
- Call v1 spicerack API by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3784>
- Run models integration tests on self-hosted macOS runners by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3785>
- Fix OpenAI models integration tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3786>
- Integration test for Databricks delta_lake mode by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3779>
- Add `spice connect` for connecting to existing Spice.ai instances by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3790>
- Add `eval` spicepod component; basic HTTP api to run eval. by @Jeadie in <https://github.com/spiceai/spiceai/pull/3766>
- Release RC for databricks delta_lake mode by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3792>
- Include Huggingface model to E2E models tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3788>
- Enable `trace_id` & `parent_span_id` overrides for `v1/chat/completion` by @Jeadie in <https://github.com/spiceai/spiceai/pull/3791>
- Search benchmark: run search workload and measure result by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3793>
- Search benchmark: measure search precision by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3804>
- Use MinIO instead of S3 for benchmark tests by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3794>
- Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/3814>
- Only verify TPCH / TPCDS official query results for DuckDB by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3816>
- Fixes for the Debezium connector by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3819>
- Fix insert statement when all columns are constraint columns by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3820>
- docs: Move ODBC to Beta for current state of roadmap by @peasee in <https://github.com/spiceai/spiceai/pull/3823>
- Accept `:`, `/` or `://` as the delimiter for the data connector by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3821>
- Update dependencies by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3826>
- Enable `read_write` mode support for Postgres Data Connector by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3813>
- feat: add Databricks ODBC TPCDS benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/3825>
- Change `spice.ai` data connector dataset path format to `<org>/<app>/datasets/<table_reference>` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3828>
- fix: enable tpcds explain snapshotting by @peasee in <https://github.com/spiceai/spiceai/pull/3830>
- Azure AI support for both LLMs & embedding models by @Jeadie in <https://github.com/spiceai/spiceai/pull/3824>
- Add Github Workflow to run Search Benchmark by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3834>
- Fetch access token with Microsoft OAuth, and use access token to initiate Sharepoint data connector graph client by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3836>
- Initialize accelerator for datasets dynamically included by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3714>
- Update cargo.lock by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3838>
- feat: add MS SQL TPCH benchmark by @peasee in <https://github.com/spiceai/spiceai/pull/3833>
- Improve Azure AI models support by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3835>
- Primary key support for Arrow's `Memtable` by @Jeadie in <https://github.com/spiceai/spiceai/pull/3829>
- Update Tokenizer to 0.21 and mistral.rs by @Jeadie in <https://github.com/spiceai/spiceai/pull/3839>
- Fix models integration tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3843>
- Enable `spice login abfs` by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3844>
- update `crates/llms` dependencies to 'spiceai' branch by @Jeadie in <https://github.com/spiceai/spiceai/pull/3846>
- Make eval runs non-blocking; `spice.eval.{results, runs}` tables. by @Jeadie in <https://github.com/spiceai/spiceai/pull/3780>
- fix: Update GraphQL snapshots by @peasee in <https://github.com/spiceai/spiceai/pull/3849>
- Update to Rust 1.83 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3847>
- feat: add mssql integration test by @peasee in <https://github.com/spiceai/spiceai/pull/3848>
- Prepend user-specified user agent in flight repl by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3850>
- fix: trim CHAR in mssql by @peasee in <https://github.com/spiceai/spiceai/pull/3852>
- Fix column quoting for SpiceCloudPlatform dialect by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3857>
- Optimize builds by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3861>
- Endgame template: Add recently added AI/ML quickstarts and samples by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3859>
- docs: Release MS SQL Beta by @peasee in <https://github.com/spiceai/spiceai/pull/3853>
- Fix nsql sampling for tables with embeddings by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3860>
- Make GH workflows with spiceai-macos runners more stable by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3863>
- fix: Remove GraphQL swapi test by @peasee in <https://github.com/spiceai/spiceai/pull/3867>
- create 1 `tokio::test` per test/model by @Jeadie in <https://github.com/spiceai/spiceai/pull/3696>
- handle `max_completion_tokens` vs `max_tokens` for openai vs azure by @Jeadie in <https://github.com/spiceai/spiceai/pull/3869>
- Search benchmark: write results to dataset by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3871>
- Create `evalconverter` that creates spice eval components. by @Jeadie in <https://github.com/spiceai/spiceai/pull/3864>
- Update quickstart in README.md by @ewgenius in <https://github.com/spiceai/spiceai/pull/3876>
- Remove reference to spiceai-smart-demo from the repo home by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3885>
- Trace `evals` accelerated tables updates in debug mode by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3884>
- Clarify confusing log message by @Advayp in <https://github.com/spiceai/spiceai/pull/3862>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/3840>
- Azure OpenAI models: make `endpoint` parameter required by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3883>
- Use spiceai delta kernel fork, actionable message for delta checkpoint errors by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3856>
- Add support for GGUF files in HF by @Jeadie in <https://github.com/spiceai/spiceai/pull/3875>
**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v1.0.0-rc.1...v1.0.0-rc.2>
```text
## Resources
- [Getting started with Spice.ai](https://docs.spiceai.org/getting-started/)
- [Documentation](https://docs.spiceai.org/)
## Community
Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.
- Twitter: [@spice_ai](https://twitter.com/spice_ai)
- Discord: [https://discord.gg/kZnTfneP5u](https://discord.gg/kZnTfneP5u)
- Telegram: [Spice AI Discussion](https://t.me/spiceaichat)
- Reddit: [https://www.reddit.com/r/spiceai](https://www.reddit.com/r/spiceai)
- Email: [[email protected]](mailto:[email protected])
