Skip to main content
Luke Kim

Luke Kim

Founder and CEO of Spice AI
View all authors

Spice v1.3.1 (May 26, 2025)

· 3 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v1.3.1! 🛡️

Spice v1.3.1 includes improvements to Databricks SQL Warehouse support and parameterized query handling, along with several bugfixes.

What's New in v1.3.1​

  • Databricks SQL Warehouse Added support for the STRUCT type, enabled join pushdown for queries within the same SQL Warehouse and added projection to logical plans to force federation with correct SQL dialect.

  • SQL Improvements: Fixed an issue where ILike was incorrectly optimized to string equality in DataFusion/Arrow and aliased the random() function to rand() for better compatibility.

  • Parameterized Queries: Fixed parameter schema ordering for queries with more than 10 parameters and resolved placeholder inference issues in CASE expressions.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook now includes 67 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.3.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.3.1 image:

docker pull spiceai/spiceai:1.3.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

No major dependency changes.

Changelog​

Full Changelog: github.com/spiceai/spiceai/compare/v1.3.0...v1.3.1

Spice v1.1.0 (Mar 31, 2025)

· 20 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Model-Context-Protocol (MCP) support in Spice.ai Open Source

Announcing the release of Spice v1.1.0! 🤖

Spice v1.1.0 introduces full support for the Model-Context-Protocol (MCP), expanding how models and tools connect. Spice can now act as both an MCP Server, with the new /v1/mcp/sse API, and an MCP Client, supporting stdio and SSE-based servers. This release also introduces a new Web Search tool with Perplexity model support, advanced evaluation workflows with custom eval scorers, including LLM-as-a-judge, and adds an IMAP Data Connector for federated SQL queries across email servers. Alongside these features, v1.1.0 includes automatic NSQL query retries, expanded task tracing, request drains for HTTP server shutdowns, delivering improved reliability, flexibility, and observability.

Highlights in v1.1.0​

  • Spice as an MCP Server and Client: Spice now supports the Model Context Protocol (MCP), for expanded tool discovery and connectivity. Spice can:

    1. Run stdio-based MCP servers internally.
    2. Connect to external MCP servers over SSE protocol (Streamable HTTP is coming soon!)

    For more details, see the MCP documentation.

    Usage​

    tools:
      - name: google_maps
        from: mcp:npx
        params:
          mcp_args: -y @modelcontextprotocol/server-google-maps

    Spice as an MCP Server​

    Tools in Spice can be accessed via MCP. For example, connecting from an IDE like Cursor or Windsurf to Spice. Set the MCP Server URL to http://localhost:8090/v1/mcp/sse.

  • Perplexity Model Support: Spice now supports Perplexity-hosted models, enabling advanced web search and retrieval capabilities. Example configuration:

    models:
      - name: webs
        from: perplexity:sonar
        params:
          perplexity_auth_token: ${ secrets:SPICE_PERPLEXITY_AUTH_TOKEN }
          perplexity_search_domain_filter:
            - docs.spiceai.org
            - huggingface.co

    For more details, see the Perplexity documentation.

  • Web Search Tool: The new Web Search Tool enables Spice models to search the web for information using search engines like Perplexity. Example configuration:

    tools:
      - name: the_internet
        from: websearch
        description: 'Search the web for information.'
        params:
          engine: perplexity
          perplexity_auth_token: ${ secrets:SPICE_PERPLEXITY_AUTH_TOKEN }

    For more details, see the Web Search Tool documentation.

  • Eval Scorers: Eval scorers assess model performance on evaluation cases. Spice includes built-in scorers:

    • match: Exact match.
    • json_match: JSON equivalence.
    • includes: Checks if actual output includes expected output.
    • fuzzy_match: Normalized subset matching.
    • levenshtein: Levenshtein distance.

    Custom scorers can use embedding models or LLMs as judges. Example:

    evals:
      - name: australia
        dataset: cricket_questions
        scorers:
          - hf_minilm
          - judge
          - match
    embeddings:
      - name: hf_minilm
        from: huggingface:huggingface.co/sentence-transformers/all-MiniLM-L6-v2
    models:
      - name: judge
        from: openai:gpt-4o
        params:
          openai_api_key: ${ secrets:OPENAI_API_KEY }
          system_prompt: |
            Compare these stories and score their similarity (0.0 to 1.0).
            Story A: {{ .actual }}
            Story B: {{ .ideal }}

    For more details, see the Eval Scorers documentation.

  • IMAP Data Connector: Query emails stored in IMAP servers using federated SQL. Example:

    datasets:
      - from: imap:[email protected]
        name: emails
        params:
          imap_access_token: ${secrets:IMAP_ACCESS_TOKEN}

    For more details, see the IMAP Data Connector documentation.

  • Automatic NSQL Query Retries: Failed NSQL queries are now automatically retried, improving reliability for federated queries. For more details, see the NSQL documentation.

  • Enhanced Task Tracing: Task history now includes chat completion IDs, and runtime readiness is traced for better observability. Use the runtime.task_history table to query task details. See the Task History documentation.

  • Vector Search with Keyword Filtering: The vector search API now includes an optional list of keywords as a parameter, to pre-filter SQL results before performing a vector search. When vector searching via a chat completion, models will automatically generate keywords relevant to the search. See the Vector Search API documentation.

  • Improved Refresh Behavior on Startup: Spice won't automatically refresh an accelerated dataset on startup if it doesn't need to. See the Refresh on Startup documentation.

  • Graceful Shutdown for HTTP Server: The HTTP server now drains requests for graceful shutdowns, ensuring smoother runtime termination.

New Contributors 🎉​

Contributors​

  • @sgrebnov
  • @phillipleblanc
  • @peasee
  • @Jeadie
  • @lukekim
  • @benrussell
  • @Sevenannn
  • @sergey-shandar
  • @Garamda
  • @johnnynunez

Breaking Changes​

No breaking changes.

Cookbook Updates​

The Spice Cookbook now has 74 recipes that make it easy to get started with Spice!

Upgrading​

To upgrade to v1.1.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.1.0 image:

docker pull spiceai/spiceai:1.1.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

  • No major dependency changes.

Changelog​

- release: Bump chart, and versions for next release by @peasee in <https://github.com/spiceai/spiceai/pull/4464>
- feat: Schedule testoperator by @peasee in <https://github.com/spiceai/spiceai/pull/4503>
- fix: Remove on zero results arguments from benchmarks by @peasee in <https://github.com/spiceai/spiceai/pull/4533>
- fix: Don't snapshot clickbench benchmarks by @peasee in <https://github.com/spiceai/spiceai/pull/4534>
- docs: v1.0.1 release note by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4529>
- Update acknowledgements by @github-actions in <https://github.com/spiceai/spiceai/pull/4535>
- In spiced_docker, propagate setup to publish-cuda by @Jeadie in <https://github.com/spiceai/spiceai/pull/4543>
- Upgrade Rust to 1.84 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4541>
- Upgrade dependencies by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4546>
- Revert "Use OpenAI golang client in `spice chat` (#4491)" by @Jeadie in <https://github.com/spiceai/spiceai/pull/4564>
- feat: add schema inference for the Spice.ai Data Connector by @peasee in <https://github.com/spiceai/spiceai/pull/4579>
- Remove 'tools: builtin' by @Jeadie in <https://github.com/spiceai/spiceai/pull/4607>
- feat: Add initial IMAP connector by @peasee in <https://github.com/spiceai/spiceai/pull/4587>
- feat: Add email content loading by @peasee in <https://github.com/spiceai/spiceai/pull/4616>
- feat: Add SSL and Auth parameters for IMAP by @peasee in <https://github.com/spiceai/spiceai/pull/4613>
- Change /v1/models to be OpenAI compatible by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4624>
- Use `pdf-extract` crate to extract text from PDF documents by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4615>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4628>
- Add 1.0.2 release notes by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4627>
- Fix cuda::ffi by @Jeadie in <https://github.com/spiceai/spiceai/pull/4649>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4654>
- fix: Spice.ai schema inference by @peasee in <https://github.com/spiceai/spiceai/pull/4674>
- Add SQL Benchmark with sample eval configuration based on TPCH by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4549>
- Update Helm chart to Spice v1.0.2 by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4655>
- Update v1.0.2 release notes by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4639>
- Fix E2E AI release install test on self-hosted runners (macos) by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4675>
- Main performance metrics calculation for Text to SQL Benchmark by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4681>
- Add eval datasets / test scripts for model grading criteria by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4663>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4684>
- Add testoperator for `evals` running by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4688>
- Add GH Workflow to run Text to SQL benchmark by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4689>
- Add 1.0.2 as supported version to SECURITY.md by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4695>
- Text-To-SQL benchmark: trace failed tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4705>
- Text-To-SQL benchmark: extend list of benchmarking models by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4707>
- Text-To-SQL: increase sql coverage, add more advanced tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4713>
- Use model that supports tools in hf_test by @Jeadie in <https://github.com/spiceai/spiceai/pull/4712>
- Fix Spice.ai E2E test by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4723>
- Return non-existing model for v1/chat endpoint by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4718>
- Update Helm chart for 1.0.3 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4742>
- Update dependencies by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4740>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4744>
- Update SECURITY.md with 1.0.3 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4745>
- Add basic smoke test of perplexity LLM to llm integration tests. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4735>
- Don't run integration tests on PRs when only CLI is changed by @Jeadie in <https://github.com/spiceai/spiceai/pull/4751>
- Prompt user to upgrade through brew / do another clean install when spice is installed through homebrew / at non-standard path by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4746>
- feat: Search with keyword filtering by @peasee in <https://github.com/spiceai/spiceai/pull/4759>
- Fix search benchmark by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4765>
- feat: Add IMAP access token parameter by @peasee in <https://github.com/spiceai/spiceai/pull/4769>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4774>
- Mark trunk builds as unstable by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4776>
- feat: Release Spice.ai RC by @peasee in <https://github.com/spiceai/spiceai/pull/4753>
- fix: Validate columns and keywords in search by @peasee in <https://github.com/spiceai/spiceai/pull/4775>
- Run models E2E tests on PR by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4798>
- fix: models runtime not required for cloud chat by @peasee in <https://github.com/spiceai/spiceai/pull/4781>
- Only open one PR for openapi.json by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4807>
- docs: Release IMAP Alpha by @peasee in <https://github.com/spiceai/spiceai/pull/4797>
- Add Results-Cache-Status to indicate query result came from cache by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4809>
- Initial spice cli e2e tests with spice upgrade tests by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4764>
- Log CLI and Runtime Versions on startup by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4816>
- Sort keys for openai by @Jeadie in <https://github.com/spiceai/spiceai/pull/4766>
- Remove docs index trigger from the endgame template by @ewgenius in <https://github.com/spiceai/spiceai/pull/4832>
- Release notes for v1.0.4 by @Jeadie in <https://github.com/spiceai/spiceai/pull/4827>
- Update SECURITY.md by @Jeadie in <https://github.com/spiceai/spiceai/pull/4829>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4831>
- Don't print URL by @lukekim in <https://github.com/spiceai/spiceai/pull/4838>
- add 'eval_run' to 'spice trace' by @Jeadie in <https://github.com/spiceai/spiceai/pull/4841>
- Run benchmark tests w/o uploading test results (pending improvements) by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4843>
- Fix 'actual" and "output" columns in `eval.results`. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4835>
- Fix string escaping of system prompt by @Jeadie in <https://github.com/spiceai/spiceai/pull/4844>
- update helm chart to v1.0.4 by @Jeadie in <https://github.com/spiceai/spiceai/pull/4828>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4806>
- fix: Skip sccache in PR for external users by @peasee in <https://github.com/spiceai/spiceai/pull/4851>
- fix: Return BAD_REQUEST when not embeddings are configured by @peasee in <https://github.com/spiceai/spiceai/pull/4804>
- Debug log cuda detection failure in spice by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4852>
- fix: Set RUSTC wrapper explicitly by @peasee in <https://github.com/spiceai/spiceai/pull/4854>
- Improve trace UX for `ai_completion`, fix infinite tool calls by @Jeadie in <https://github.com/spiceai/spiceai/pull/4853>
- Allow homebrew spice cli to upgrade the runtime by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4811>
- Add support for MCP tools by @Jeadie in <https://github.com/spiceai/spiceai/pull/4808>
- fix: Rustc wrapper actions by @peasee in <https://github.com/spiceai/spiceai/pull/4867>
- Provide link to supported OS list when user platform is not supported by @Garamda in <https://github.com/spiceai/spiceai/pull/4840>
- Always download spice runtime version matched with spice cli version by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4761>
- Disable flaky integration test by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4871>
- fix: sccache actions setup by @peasee in <https://github.com/spiceai/spiceai/pull/4873>
- Fixing Go installation in the setup script for Linux Arm64 by @sergey-shandar in <https://github.com/spiceai/spiceai/pull/4868>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4864>
- DuckDB acceleration: Use temp table only for append with conflict resolution by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4874>
- Trace the output of streamed `chat/completions` to runtime.task_history. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4845>
- Always pass `X-API-Key` in spice api calls header if detected in env by @ewgenius in <https://github.com/spiceai/spiceai/pull/4878>
- Revert "DuckDB acceleration: Use temp table only for append with conflict resolution" by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4886>
- Allow overriding spicerack base url in the CLI by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4892>
- Add test Spicepod for DuckDB full acceleration with constraints by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4891>
- Refactor Parameter Handling by @Advayp in <https://github.com/spiceai/spiceai/pull/4833>
- Add test Spicepod for DuckDB append acceleration with constraints by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4898>
- Update to latest async-openai fork. Update secrecy by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4911>
- Fix mcp tools build by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4916>
- Add more test spicepods by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4923>
- task: Add more dispatch files by @peasee in <https://github.com/spiceai/spiceai/pull/4933>
- run spiceai benchmark test using test operator by @Sevenannn in <https://github.com/spiceai/spiceai/pull/4920>
- Convert sequential search code block to parallel async by @Garamda in <https://github.com/spiceai/spiceai/pull/4936>
- fix: Throughput metric calculation by @peasee in <https://github.com/spiceai/spiceai/pull/4938>
- Update dependabot dependencies & `cargo update` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4872>
- Improve servers shutdown sequence during runtime termination by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4942>
- Semantic model for views. Views visible in `table_schema` & `list_datasets` tools. by @Jeadie in <https://github.com/spiceai/spiceai/pull/4946>
- update openai-async by @Jeadie in <https://github.com/spiceai/spiceai/pull/4948>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4961>
- fix: Redundant results snapshotting by @peasee in <https://github.com/spiceai/spiceai/pull/4956>
- Create schema for views if not exist by @Jeadie in <https://github.com/spiceai/spiceai/pull/4957>
- Bump Jimver/cuda-toolkit from 0.2.21 to 0.2.22 by @dependabot in <https://github.com/spiceai/spiceai/pull/4969>
- List available operations in `spice trace <operation>` by @Jeadie in <https://github.com/spiceai/spiceai/pull/4953>
- Initial commit of release analytics by @lukekim in <https://github.com/spiceai/spiceai/pull/4975>
- Remove spaces from CSV by @lukekim in <https://github.com/spiceai/spiceai/pull/4977>
- Fix Spice pods watcher by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4984>
- feat: Add appendable data sources for the testoperator by @peasee in <https://github.com/spiceai/spiceai/pull/4949>
- Omit timestamp when warning regarding datasets with hyphens by @Advayp in <https://github.com/spiceai/spiceai/pull/4987>
- Update helm chart to v1.0.5 by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4990>
- docs: Update qa_analytics.csv by @peasee in <https://github.com/spiceai/spiceai/pull/4989>
- Update end_game template by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4991>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/4993>
- Add v1.0.5 release notes by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4994>
- Supported Versions: include v1.0.5 by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4995>
- Dependabot updates by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/4992>
- Switch to basic markdown formatting for vector search by @sgrebnov in <https://github.com/spiceai/spiceai/pull/4934>
- docs: Update qa_analytics.csv by @peasee in <https://github.com/spiceai/spiceai/pull/5001>
- feat: Add TPCDS FileAppendableSource for testoperator by @peasee in <https://github.com/spiceai/spiceai/pull/5002>
- Update `ring` by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/5003>
- docs: Update qa_analytics.csv by @peasee in <https://github.com/spiceai/spiceai/pull/5006>
- feat: Add ClickBench FileAppendableSource for testoperator by @peasee in <https://github.com/spiceai/spiceai/pull/5004>
- feat: Validate append test table counts by @peasee in <https://github.com/spiceai/spiceai/pull/5008>
- feat: Add append spicepods by @peasee in <https://github.com/spiceai/spiceai/pull/5009>
- Improve Vector Search performance for large content w/o primary key defined by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5010>
- Don't try to downgrade Arc in test_acceleration_duckdb_single_instance by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/5014>
- feat: Add an initial testoperator vector search command by @peasee in <https://github.com/spiceai/spiceai/pull/5011>
- feat: Update testoperator workflows for automatic snapshot updates by @peasee in <https://github.com/spiceai/spiceai/pull/5018>
- Fix Vector Search when additional columns include embedding column by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5022>
- Include test for primary key passed as additional column in Vector Search by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5024>
- fix: Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/5020>
- upgrade mistral.rs by @Jeadie in <https://github.com/spiceai/spiceai/pull/4952>
- fix: Indexes for TPCDS SQLite Spicepod by @peasee in <https://github.com/spiceai/spiceai/pull/5038>
- fix: Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/5035>
- Include local files in generated Spicepod package by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5041>
- update mistral.rs to 'spiceai' branch rev by @Jeadie in <https://github.com/spiceai/spiceai/pull/5029>
- Configure spiced as an MCP SSE server by @Jeadie in <https://github.com/spiceai/spiceai/pull/5039>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/5052>
- fix: Disable benchmarks schedule, enable testoperator schedule by @peasee in <https://github.com/spiceai/spiceai/pull/5058>
- fix: Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/5060>
- Update ROADMAP.md March 2025 by @lukekim in <https://github.com/spiceai/spiceai/pull/5061>
- fix: Testoperator data setup by @peasee in <https://github.com/spiceai/spiceai/pull/5068>
- fix: All HTTP endpoints to hang when adding an invalid dataset with --pods-watcher-enabled by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5050>
- fix: Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/5073>
- Integration tests for MCP tooling by @Jeadie in <https://github.com/spiceai/spiceai/pull/5053>
- OpenAPI docs for MCP by @Jeadie in <https://github.com/spiceai/spiceai/pull/5057>
- fix: Acceleration federation test by @peasee in <https://github.com/spiceai/spiceai/pull/5090>
- fix: Allow spiced commit in testoperator dispatch by @peasee in <https://github.com/spiceai/spiceai/pull/5098>
- fix: Use RefreshOverrides for the refresh API definition by @peasee in <https://github.com/spiceai/spiceai/pull/5095>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/5094>
- fix: Increase tries for refresh_status_change_to_ready test by @peasee in <https://github.com/spiceai/spiceai/pull/5099>
- feat: Testoperator reports on max and median memory usage by @peasee in <https://github.com/spiceai/spiceai/pull/5101>
- Update openapi.json by @github-actions in <https://github.com/spiceai/spiceai/pull/5105>
- fix: Fail testoperator on failed queries by @peasee in <https://github.com/spiceai/spiceai/pull/5106>
- Update Helm chart to 1.0.6 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/5107>
- Update SECURITY.md to include 1.0.6 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/5109>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/5108>
- Add QA analytics for 1.0.6 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/5110>
- add env variables to tools, usable in MCP stdio by @Jeadie in <https://github.com/spiceai/spiceai/pull/5097>
- HF downloads obey SIGTERM by @Jeadie in <https://github.com/spiceai/spiceai/pull/5044>
- Add v1.0.6 release notes into trunk by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5111>
- Remove redundant mod name for iceberg integration tests by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5112>
- Use fixed data directory for test operator by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5103>
- Improvements for evals by @Jeadie in <https://github.com/spiceai/spiceai/pull/5040>
- Make McpProxy trait for MCP passthrough by @Jeadie in <https://github.com/spiceai/spiceai/pull/5115>
- Properly handle '/' for tool names. by @Jeadie in <https://github.com/spiceai/spiceai/pull/5116>
- Use retry logic when loading tools by @Jeadie in <https://github.com/spiceai/spiceai/pull/5120>
- Exclude slow tests from regular pr runs by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5119>
- Fix test operator snapshot update by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5130>
- spice init: Fixes windows bug where full path is used for spicepod name by @benrussell in <https://github.com/spiceai/spiceai/pull/5126>
- fix: Update benchmark snapshots by @github-actions in <https://github.com/spiceai/spiceai/pull/5131>
- Implement graceful shutdown for HTTP server by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5102>
- Update enhancement.md by @lukekim in <https://github.com/spiceai/spiceai/pull/5142>
- Add GitHub Workflow and PoC Spicepod configuration to run FinanceBench tests by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5145>
- Fix Postgres and MySQL installation on macos14-runner (E2E CI) by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5155>
- De-duplicate attachments in DuckDBAttachments by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/5156>
- v1.0.7 release note by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5153>
- Update spicepod.schema.json by @github-actions in <https://github.com/spiceai/spiceai/pull/5160>
- Update Helm chart to 1.0.7 by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5159>
- Add github token to macos test release download tasks by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5161>
- update security.md for 1.0.7 by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5162>
- Update roadmap.md by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5163>
- Add a performance comparison section for 1.0.7 by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/5164>
- docs: Add snafu error variant point to style guide by @peasee in <https://github.com/spiceai/spiceai/pull/5167>
- Fix 1.0.7 release note by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5168>
- Adjust DuckDB connection pool size based on DuckDB accelerator instances usage by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5117>
- Add automatic retry for NSQL queries by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5169>
- Include chat completion id to task history by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5170>
- Trace when all runtime components are ready by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5171>
- Update qa_analytics.csv for 1.0.7 by @Sevenannn in <https://github.com/spiceai/spiceai/pull/5165>
- Set default tool recursion limit to 10 to prevent infinite loops by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5173>
- Add support for `schema_source_path` param for object-store data connectors by @sgrebnov in <https://github.com/spiceai/spiceai/pull/5178>
- Run license check and check changes on self-hosted macOS runners by @lukekim in <https://github.com/spiceai/spiceai/pull/5179>
- Add MCP by @lukekim in <https://github.com/spiceai/spiceai/pull/5183>

Full Changelog: github.com/spiceai/spiceai/compare/v1.0.0...release/1.1

Spice v1.0-rc.3 (Dec 30, 2024)

· 9 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v1.0-rc.3 🧊

Spice v1.0.0-rc.3 is the third release candidate for the first major version of Spice.ai OSS. This release continues the focus on production readiness and includes new Iceberg Catalog APIs, DuckDB improvements, and a new Iceberg Catalog Connector.

Highlights in v1.0-rc.3​

  • Iceberg Catalog APIs: Spice now functions as an Iceberg Catalog provider, implementing a core subset of the Iceberg Catalog APIs. This enables Iceberg Catalog clients native discovery of datasets and schemas through Spice APIs.

  • GET /v1/namespaces - List all catalogs registered in Spice.

  • GET /v1/namespaces?parent=catalog - List schemas registered under a given catalog.

  • GET /v1/namespaces/:catalog_schema/tables - List tables registered under a given schema.

  • GET /v1/namespaces/:catalog_schema/tables/:table - Get the schema of a given table.

  • Iceberg Catalog Connector: The Iceberg Catalog Connector is a new integration to discover and query datasets from a remote Iceberg Catalog.

Example connecting to a remote Iceberg Catalog with tables stored in S3:

catalogs:
  - from: iceberg:https://my-iceberg-catalog.com/v1/namespaces
    name: ice
    params:
      iceberg_s3_access_key_id: ${secrets:ICEBERG_S3_ACCESS_KEY_ID}
      iceberg_s3_secret_access_key: ${secrets:ICEBERG_S3_SECRET_ACCESS_KEY}
      iceberg_s3_region: us-east-1

View the Iceberg Catalog Connector documentation for more details.

  • DuckDB Improvements: Added cosine_distance support for DuckDB-backed vector search, improved unnest nested type handling for array_element and lists, and optimized query performance.

  • SQLite Data Accelerator: Graduated to Release Candidate (RC).

  • File Data Accelerator: Graduated to Release Candidate (RC).

Breaking changes​

  • API:v1/datasets/sample has been removed as it is not particularly useful, can be replicated via SQL, and via the tools endpoint POST v1/tools/:name.

Cookbook​

  • New Language Model Evals Recipe showing how to measure the performance of a language model using LLM-as-Judge, configured entirely in the spice runtime.

  • New Iceberg Catalog Recipe showing how to use Spice to query Iceberg tables from an Iceberg catalog.

Dependencies​

  • OpenTelemetry: Upgraded from 0.26.0 to 0.27.1
  • Go: Upgraded from 1.22 to 1.23 (CLI)

Contributors​

  • @sgrebnov
  • @phillipleblanc
  • @peasee
  • @Jeadie
  • @Sevenannn
  • @lukekim
  • @ewgenius

What's Changed​

- Add CI configuration for search benchmark dataset access by @sgrebnov in https://github.com/spiceai/spiceai/pull/3888
- Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/3895
- Upgrade dependencies by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3896
- chore: Update helm chart for RC.2 by @peasee in https://github.com/spiceai/spiceai/pull/3899
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/3903
- chore: Update MacOS test release install to macos-13 by @peasee in https://github.com/spiceai/spiceai/pull/3901
- Add usage to `spice chat` and fix `v1/models?status=true`. by @Jeadie in https://github.com/spiceai/spiceai/pull/3898
- chore: Bump versions for rc3 by @peasee in https://github.com/spiceai/spiceai/pull/3902
- docs: Update endgame with a step to verify dependencies in release notes by @peasee in https://github.com/spiceai/spiceai/pull/3897
- Ensure eval dataset input and ouput of correct length by @Jeadie in https://github.com/spiceai/spiceai/pull/3900
- `spice add/connect/dataset configure` should update spicepod, not overwrite it & upgrade to Go 1.23 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3905
- Bump opentelemetry from 0.26.0 to 0.27.1 by @dependabot in https://github.com/spiceai/spiceai/pull/3879
- Ensure trace_id is overridden for prior written spans by @Jeadie in https://github.com/spiceai/spiceai/pull/3906
- add 'role': 'assistant' for local models by @Jeadie in https://github.com/spiceai/spiceai/pull/3910
- Run tpcds benchmark for file connector by @Sevenannn in https://github.com/spiceai/spiceai/pull/3924
- Update to reference cookbook instead of quickstarts/samples by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3928
- Fix/remove flaky integration tests by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3930
- Implement `/v1/iceberg/namespaces` & `/v1/iceberg/config` APIs by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3923
- Add script for creating tpcds parquet files and spicepod for file connector by @Sevenannn in https://github.com/spiceai/spiceai/pull/3931
- Use `utoipa` to generate openapi.json and swagger for dev by @Jeadie in https://github.com/spiceai/spiceai/pull/3927
- `fuzzy_match`, `json_match`, `includes` scorer by @Jeadie in https://github.com/spiceai/spiceai/pull/3926
- Implement `/v1/iceberg/namespaces/:namespace` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3933
- Implement `GET /v1/iceberg/namespaces/:namespace/tables` API by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3934
- Add custom Spice DuckDB dialect with cosine_distance support by @sgrebnov in https://github.com/spiceai/spiceai/pull/3938
- Fix NSQL error: `all columns in a record batch must have the same length` by @sgrebnov in https://github.com/spiceai/spiceai/pull/3947
- Don't include tools use in hf test model by @Jeadie in https://github.com/spiceai/spiceai/pull/3955
- Implement `GET /v1/namespaces/{namespace}/tables/{table}` API by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3940
- Update dependencies by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3967
- DuckDB: add support for nested types in Lists by @sgrebnov in https://github.com/spiceai/spiceai/pull/3961
- Add script to set up clickbench for file connector by @Sevenannn in https://github.com/spiceai/spiceai/pull/3945
- docs: Add connector stable criteria by @peasee in https://github.com/spiceai/spiceai/pull/3908
- Update Roadmp Dec 23, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/3978
- Improve CI testing for OpenAPI, new tool `spiceschema`, fix broken OpenAPI stuff. by @Jeadie in https://github.com/spiceai/spiceai/pull/3948
- remove `v1/datasets/sample` by @Jeadie in https://github.com/spiceai/spiceai/pull/3981
- feat: add SQLite ClickBench benchmark by @peasee in https://github.com/spiceai/spiceai/pull/3975
- Remove feature 'llms/mistralrs' by @Jeadie in https://github.com/spiceai/spiceai/pull/3984
- Add support for 'params.spice_tools: nsql' by @Jeadie in https://github.com/spiceai/spiceai/pull/3985
- Fix integration tests - add missing `format` query parameter in /v1/status requests by @ewgenius in https://github.com/spiceai/spiceai/pull/3989
- Enhance AI tools sampling logic for robust handling of large fields by @sgrebnov in https://github.com/spiceai/spiceai/pull/3959
- Fix subquery federation by @Sevenannn in https://github.com/spiceai/spiceai/pull/3991
- Fix unnest and add DuckDB support for `array_element` by @sgrebnov in https://github.com/spiceai/spiceai/pull/3995
- Add score value snapshotting to vector similarity search tests by @sgrebnov in https://github.com/spiceai/spiceai/pull/3996
- Use Llama-3.2-3B-Instruct for Hugging Face integration testing by @sgrebnov in https://github.com/spiceai/spiceai/pull/3992
- Simplify `construct_chunk_query_sql` for DuckDB compatibility by @sgrebnov in https://github.com/spiceai/spiceai/pull/3988
- Update TPCH and TPCDS benchmarks for spice.ai connector by @ewgenius in https://github.com/spiceai/spiceai/pull/3982
- Correctly pass Hugging Face token in models integration tests by @sgrebnov in https://github.com/spiceai/spiceai/pull/3997
- Fix: `on_zero_results` causes `TransactionContext Error: Catalog write-write conflict on create with "attachment_0"` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3998
- Add DuckDB acceleration to search benchmarks by @sgrebnov in https://github.com/spiceai/spiceai/pull/4000
- Enable Postgres write via non-default `postgres-write` feature flag by @sgrebnov in https://github.com/spiceai/spiceai/pull/4004
- Allow search benchmark to write test results by @sgrebnov in https://github.com/spiceai/spiceai/pull/4008
- Make Flight DoPut atomic and commit write only on successful stream completion by @sgrebnov in https://github.com/spiceai/spiceai/pull/4002
- Create a `CatalogConnector` abstraction by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4003
- Fix `generate-openapi.yml` and add `.schema/openapi.json`. by @Jeadie in https://github.com/spiceai/spiceai/pull/3983
- Enable spice.ai tpcds bench workflow. Comment failing tpch queries. by @ewgenius in https://github.com/spiceai/spiceai/pull/4001
- feat: Add SQLite ClickBench overrides by @peasee in https://github.com/spiceai/spiceai/pull/4016
- Implement Iceberg Catalog Connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4053
- feat: Datafusion updates for SQLite fixes and release by @peasee in https://github.com/spiceai/spiceai/pull/4054
- docs: Add accelerator stable release criteria by @peasee in https://github.com/spiceai/spiceai/pull/4017
- Add dremio tpch / tpcds benchmark test by @Sevenannn in https://github.com/spiceai/spiceai/pull/4063
- Update docs, and make PR to `spiceai/docs` for new `openapi.json`. by @Jeadie in https://github.com/spiceai/spiceai/pull/4019
- Update openapi.json by @github-actions in https://github.com/spiceai/spiceai/pull/4065
- Fix dremio subquery rewrite by @Sevenannn in https://github.com/spiceai/spiceai/pull/4064
- Update generate-openapi.yml by @Jeadie in https://github.com/spiceai/spiceai/pull/4073
- docs: Add catalog criteria by @peasee in https://github.com/spiceai/spiceai/pull/4052
- fix `distinct_columns` in auto/nsql tool groups by @Jeadie in https://github.com/spiceai/spiceai/pull/4074
- Update openapi.json by @github-actions in https://github.com/spiceai/spiceai/pull/4075
- Update openapi.json by @github-actions in https://github.com/spiceai/spiceai/pull/4076
- Implement window_func_support_window_frame from DremioDialect by @Sevenannn in https://github.com/spiceai/spiceai/pull/4012
- Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/4079
- Promote file connector to rc by @Sevenannn in https://github.com/spiceai/spiceai/pull/4080
- Add Iceberg to README by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4085
- Fix '/v1/status' default format by @Jeadie in https://github.com/spiceai/spiceai/pull/4081

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v1.0.0-rc.2...v1.0.0-rc.3

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.19.1-beta (Oct 14, 2024)

· 5 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v0.19.1-beta 🔥

Spice v0.19.1 brings further performance and stability improvements to data connectors, including improved query push-down for file-based connectors (s3, abfs, file, ftp, sftp) that use Hive-style partitioning.

Highlights in v0.19.1​

TPC-H and TPC-DS Coverage: Expanded coverage for TPC-H and TPC-DS benchmarking suites across accelerators and connectors.

GitHub Connector Array Filter: The GitHub connector now supports filter push down for the array_contains function in SQL queries using search query mode.

NSQL CLI Command: A new spice nsql CLI command has been added to easily query datasets with natural language from the command line.

Breaking changes​

None

Contributors​

  • @peasee
  • @Sevenannn
  • @sgrebnov
  • @karifabri
  • @phillipleblanc
  • @lukekim
  • @Jeadie
  • @slyons

Dependencies​

What's Changed​

- release: Update helm chart for v0.19.0-beta by @peasee in https://github.com/spiceai/spiceai/pull/3024
- Set fail-fast = true for benchmark test by @Sevenannn in https://github.com/spiceai/spiceai/pull/2997
- release: Update next version and ROADMAP by @peasee in https://github.com/spiceai/spiceai/pull/3033
- Verify TPCH benchmark query results for Spark connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/2993
- feat: Add x-spice-user-agent header to Spice REPL by @peasee in https://github.com/spiceai/spiceai/pull/2979
- Update to object store file formats documentation link by @karifabri in https://github.com/spiceai/spiceai/pull/3036
- Use teraswitch-runners for Linux x64 workflows + builds by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3042
- feat: Support array contains in GitHub pushdown by @peasee in https://github.com/spiceai/spiceai/pull/2983
- Bump text-splitter from 0.16.1 to 0.17.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2987
- Revert integration tests back to hosted runner by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3046
- Tune Github runner resources to allow in memory TPCDS benchmark to run by @Sevenannn in https://github.com/spiceai/spiceai/pull/3025
- fix: add winver by @peasee in https://github.com/spiceai/spiceai/pull/3054
- refactor: Use is modifier for checking GitHub state filter by @peasee in https://github.com/spiceai/spiceai/pull/3056
- Enable `merge_group` checks for PR workflows by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3058
- Fix issues with merge group by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3059
- Validate in-memory arrow accelertion TPCDS result correctness by @Sevenannn in https://github.com/spiceai/spiceai/pull/3044
- Fix rev parsing for PR checks by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3060
- Use 'Accept' header for `/v1/sql/` and `/v1/nsql` by @Jeadie in https://github.com/spiceai/spiceai/pull/3032
- Verify Postgres acceleration TPCDS result correctness by @Sevenannn in https://github.com/spiceai/spiceai/pull/3043
- Add NSQL CLI REPL command by @lukekim in https://github.com/spiceai/spiceai/pull/2856
- Preserve query results order and add TPCH benchmark results verification for duckdb:file mode by @sgrebnov in https://github.com/spiceai/spiceai/pull/3034
- Refactor benchmark to include MySQL tpcds bench, tweaks to makefile target for generating mysql tpcds data by @Sevenannn in https://github.com/spiceai/spiceai/pull/2967
- Support runtime parameter for `sql_query_keep_partition_by_columns` & enable by default by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3065
- Document TPC-DS limitations: `EXCEPT`, `INTERSECT`, duplicate names by @sgrebnov in https://github.com/spiceai/spiceai/pull/3069
- Adding ABFS benchmark by @slyons in https://github.com/spiceai/spiceai/pull/3062
- Add support for GitHub app installation auth for GitHub connector by @ewgenius in https://github.com/spiceai/spiceai/pull/3063
- docs: Document stack overflow workaround, add helper script by @peasee in https://github.com/spiceai/spiceai/pull/3070
- Tune MySQL TPCDS image to allow for successful benchmark test run by @Sevenannn in https://github.com/spiceai/spiceai/pull/3067
- Automatically infer partitions for hive-style partitioned files for object store based connectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3073
- Support `hf_token` from params/secrets by @Jeadie in https://github.com/spiceai/spiceai/pull/3071
- Inherit embedding columns from source, when available. by @Jeadie in https://github.com/spiceai/spiceai/pull/3045
- Validate identifiers for component names by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3079
- docs: Add workaround for TPC-DS Q97 in MySQL by @peasee in https://github.com/spiceai/spiceai/pull/3080
- Document TPC-DS Postgres column alias in a CASE statement limitation by @sgrebnov in https://github.com/spiceai/spiceai/pull/3083
- Update plan snapshots for TPC-H bench queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/3088
- Update Datafusion crate to include recent unparsing fixes by @sgrebnov in https://github.com/spiceai/spiceai/pull/3089
- Sample SQL table data tool and API by @Jeadie in https://github.com/spiceai/spiceai/pull/3081
- chore: Update datafusion-table-providers by @peasee in https://github.com/spiceai/spiceai/pull/3090
- Add `hive_infer_partitions` to remaining object store connectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/3086
- deps: Update datafusion-table-providers by @peasee in https://github.com/spiceai/spiceai/pull/3093
- For local embedding models, return usage input tokens. by @Jeadie in https://github.com/spiceai/spiceai/pull/3095
- Update end_game.md with Accelerator/Connector criteria check by @slyons in https://github.com/spiceai/spiceai/pull/3092
- Update TPC-DS Q90 by @sgrebnov in https://github.com/spiceai/spiceai/pull/3094
- docs: Add RC connector criteria by @peasee in https://github.com/spiceai/spiceai/pull/3026
- Update version to 0.19.1-beta by @sgrebnov in https://github.com/spiceai/spiceai/pull/3101

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.19.0-beta...v0.19.1-beta

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.17.4-beta (Sep 9, 2024)

· 5 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v0.17.4-beta.

The v0.17.4-beta release adds compatibility, performance, and reliability improvements to the DuckDB and SQLite accelerators. The GitHub data connector adds a Stargazers table, Snowflake and Clickhouse data connectors have improved resiliency for empty tables, and core data processing and quality has been improved.

Highlights in v0.17.4-beta​

Improved benchmarking, testing, and robustness of data accelerators: Continued compatibility, performance, and reliability improvements for SQLite and DuckDB data accelerators and expanded performance and quality testing.

GitHub Stargazers: The GitHub Data Connector adds support for a /stargazers table making it easy to query GitHub Stargazers using SQL!

Breaking Changes​

None.

Contributors​

  • @phillipleblanc
  • @Jeadie
  • @lukekim
  • @sgrebnov
  • @peasee
  • @eltociear
  • @Sevenannn
  • @ewgenius

New Contributors​

What's Changed​

- Change to sql lang by @ewgenius in https://github.com/spiceai/spiceai/pull/2484
- Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/2487
- Bump rustyline from 13.0.0 to 14.0.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2473
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/2490
- Native schema inference for snowflake (and support timezone_tz, better numeric support) by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2493
- Add checks for GitHub quickstart and docs banner to endgame template by @ewgenius in https://github.com/spiceai/spiceai/pull/2489
- Prepare for v0.18.0-beta by @Jeadie in https://github.com/spiceai/spiceai/pull/2488
- Add logo to README.md by @lukekim in https://github.com/spiceai/spiceai/pull/2497
- Add stargazers to GitHub data connector by @lukekim in https://github.com/spiceai/spiceai/pull/2502
- Enable federation for accelerated queries (sqlite and duckdb) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2511
- Load SQLite decimal extension by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2498
- fix: Support INTERVAL in SQLite by @peasee in https://github.com/spiceai/spiceai/pull/2513
- Add refresh jitter to refreshing dataset acceleration by @Jeadie in https://github.com/spiceai/spiceai/pull/2510
- Update to use DuckDB streaming by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2514
- Add more MySQL types in E2E testing by @sgrebnov in https://github.com/spiceai/spiceai/pull/2512
- Update tpc loading script to support automatic loading into postgres by @Sevenannn in https://github.com/spiceai/spiceai/pull/2509
- docs: update README.md by @eltociear in https://github.com/spiceai/spiceai/pull/2516
- Bump quinn-proto from 0.11.6 to 0.11.8 in the cargo group by @dependabot in https://github.com/spiceai/spiceai/pull/2501
- Script for loading clickbench data into arrow / postgres and run clickbench queries by @Sevenannn in https://github.com/spiceai/spiceai/pull/2500
- Fix run query script to correctly record all errors by @Sevenannn in https://github.com/spiceai/spiceai/pull/2529
- Add support for DuckDB engine to setup-tpc-spicepod.bash by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2530
- Upgrade `datafusion` (fixes subquery alias table unparsing for SQLite) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2532
- Make dataset acceleration delay `period +- jitter` by @Jeadie in https://github.com/spiceai/spiceai/pull/2534
- Add refresh options to `POST /v1/datasets/:name/acceleration/refresh` by @Jeadie in https://github.com/spiceai/spiceai/pull/2515
- Add E2E for GitHub Connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/2505
- Add on-conflict integration test for file based and memory based sqlite by @Sevenannn in https://github.com/spiceai/spiceai/pull/2533
- Upgrade to Rust v.1.81.0 and fix resulting compile error by @Sevenannn in https://github.com/spiceai/spiceai/pull/2539
- Remove unneeded `RwLock` from `EmbeddingModelStore` by @Jeadie in https://github.com/spiceai/spiceai/pull/2541
- Remove unneeded RwLock from LlmModelStore by @Jeadie in https://github.com/spiceai/spiceai/pull/2537
- Add sqlite to the setup tpc benchmark script by @Sevenannn in https://github.com/spiceai/spiceai/pull/2540
- Add sqlite to setup clickbench script by @Sevenannn in https://github.com/spiceai/spiceai/pull/2548
- Update version for v0.17.4-beta release by @ewgenius in https://github.com/spiceai/spiceai/pull/2563
- Sharepoint data connector by @Jeadie in https://github.com/spiceai/spiceai/pull/2294
- Fix predicate/projection push-down for BytesProcessedNode by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2564
- fix out of order projections in sharepoint scans by @Jeadie in https://github.com/spiceai/spiceai/pull/2569
- Use Decimal instead of Float64 for SQLite Decimal columns by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2566
- Add snapshot tests for EXPLAIN plans in integration tests by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2570
- Set refresh.period from `refresh_data_window` by @ewgenius in https://github.com/spiceai/spiceai/pull/2578
- Add snapshot tests for EXPLAIN plans in benchmark tests by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2580
- Disable federation for accelerated queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/2581
- Add manual refresh payload to 'spice refresh...' by @Jeadie in https://github.com/spiceai/spiceai/pull/2565
- Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/2586

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.17.3-beta...v0.17.4-beta

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.16-alpha (July 22, 2024)

· 8 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.16-alpha release is the first candidate release for the beta milestone on a path to finalizing the v1.0 developer and user experience. Upgraders should be aware of several breaking changes designed to improve the Secrets configuration experience and to make authoring spicepod.yml files more consistent. See the [Breaking Changes](#Breaking Changes) section below for details. Additionally, the Spice Java SDK was released, providing Java developers a simple but powerful native experience to query Spice.

Highlights in v0.16-alpha​

secrets:
  - from: env
    name: env
  - from: aws_secrets_manager:my_secret_name
    name: aws_secret

Secrets managed by configured Secret Stores can be referenced in component params using the syntax ${<store_name>:<key>}. E.g.

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass: ${ env:MY_PG_PASS }

  • Java Client SDK: The Spice Java SDK has been released for JDK 17 or greater.

  • Federated SQL Query: Significant stability and reliability improvements have been made to federated SQL query support in most data connectors.

  • ODBC Data Connector: Providing a specific SQL dialect to query ODBC data sources is now supported using the sql_dialect param. For example, when querying Databricks using ODBC, the databricks dialect can be specified to ensure compatibility. Read the ODBC Data Connector documentation for more details.

Breaking Changes​

  • Secret Stores: Secret Stores support has been overhauled including required changes to spicepod.yml schema. File based secrets stored in the ~/.spice/auth file are no longer supported. See Secret Stores Documentation for full reference.

To upgrade Secret Stores, rename any parameters ending in _key to remove the _key suffix and specify a secret inline via the secret replacement syntax (${<secret_store>:<key>}):

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass_key: my_pg_pass

to:

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass: ${secrets:my_pg_pass}

And ensure the MY_PG_PASS environment variable is set.

  • Datasets: The default value of time_format has changed from unix_seconds to timestamp.

To upgrade:

datasets:
  - from:
    name: my_dataset
    # Explicitly define format when not specified.
    time_format: unix_seconds
  • HTTP Port: The default HTTP port has changed from port 3000 to port 8090 to avoid conflicting with frontend apps which typically use the 3000 range. If an SDK is used, upgrade it at the same time as the runtime.

To upgrade and continue using port 3000, run spiced with the --http command line argument:

# Using Dockerfile or spiced directly
spiced --http 127.0.0.1:3000
  • HTTP Metrics Port: The default HTTP Metrics port has changed from port 9000 to 9090 to avoid conflicting with other metrics protocols which typically use port 9000.

To upgrade and continue using port 9000, run spiced with the metrics command line argument:

# Using Dockerfile or spiced directly
spiced --metrics 127.0.0.1:9000

To upgrade, change:

json_path: my.json.path

To:

json_pointer: /my/json/pointer
  • Data Connector Configuration: Consistent connector name prefixing has been applied to connector specific params parameters. Prefixed parameter names helps ensure parameters do not collide.

For example, the Databricks data connector specific params are now prefixed with databricks:

datasets:
  - from: databricks:spiceai.datasets.my_awesome_table # A reference to a table in the Databricks unity catalog
    name: my_delta_lake_table
    params:
      mode: spark_connect
      endpoint: dbc-a1b2345c-d6e7.cloud.databricks.com
      token: MY_TOKEN

To upgrade:

datasets:
  # Example for Spark Connect
  - from: databricks:spiceai.datasets.my_awesome_table # A reference to a table in the Databricks unity catalog
    name: my_delta_lake_table
    params:
      mode: spark_connect
      databricks_endpoint: dbc-a1b2345c-d6e7.cloud.databricks.com # Now prefixed with databricks
      databricks_token: ${secrets:my_token} # Now prefixed with databricks

Refer to the Data Connector documentation for parameter naming changes in this release.

Clickhouse Data Connector: The clickhouse_connection_timeout parameter has been renamed to connection_timeout as it applies to the client and is not Clickhouse configuration itself.

To upgrade, change:

clickhouse_connection_timeout: time

To:

connection_timeout: time

Contributors​

  • @y-f-u
  • @phillipleblanc
  • @ewgenius
  • @github-actions
  • @sgrebnov
  • @lukekim
  • @digadeesh
  • @peasee
  • @Sevenannn

What's Changed​

Dependencies​

No major dependency updates.

Commits​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.15.2-alpha...v0.16.0-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.15.2-alpha (July 15, 2024)

· 5 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.15.2-alpha minor release focuses on enhancing stability, performance, and introduces Catalog Providers for streamlined access to Data Catalog tables. Unity Catalog, Databricks Unity Catalog, and the Spice.ai Cloud Platform Catalog are supported in v0.15.2-alpha. The reliability of federated query push-down has also been improved for the MySQL, PostgreSQL, ODBC, S3, Databricks, and Spice.ai Cloud Platform data connectors.

Highlights in v0.15.2-alpha​

Catalog Providers: Catalog Providers streamline access to Data Catalog tables. Initial catalog providers supported are Databricks Unity Catalog, Unity Catalog and Spice.ai Cloud Platform Catalog.

For example, to configure Spice to connect to tpch tables in the Spice.ai Cloud Platform Catalog use the new catalogs: section in the spicepod.yml:

catalogs:
  - name: spiceai
    from: spiceai
    include:
      - tpch.*
sql> show tables
+---------------+--------------+---------------+------------+
| table_catalog | table_schema | table_name    | table_type |
+---------------+--------------+---------------+------------+
| spiceai       | tpch         | region        | BASE TABLE |
| spiceai       | tpch         | part          | BASE TABLE |
| spiceai       | tpch         | customer      | BASE TABLE |
| spiceai       | tpch         | lineitem      | BASE TABLE |
| spiceai       | tpch         | partsupp      | BASE TABLE |
| spiceai       | tpch         | supplier      | BASE TABLE |
| spiceai       | tpch         | nation        | BASE TABLE |
| spiceai       | tpch         | orders        | BASE TABLE |
| spice         | runtime      | query_history | BASE TABLE |
+---------------+--------------+---------------+------------+

Time: 0.001866958 seconds. 9 rows.

ODBC Data Connector Push-Down: The ODBC Data Connector now supports query push-down for joins, improving performance for joined datasets configured with the same odbc_connection_string.

Improved Spicepod Validation Improved spicepod.yml validation has been added, including warnings when loading resources with duplicate names (datasets, views, models, embeddings).

Breaking Changes​

None.

Contributors​

  • @phillipleblanc
  • @peasee
  • @y-f-u
  • @ewgenius
  • @Sevenannn
  • @sgrebnov
  • @lukekim

What's Changed​

Dependencies​

Commits​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.15.1-alpha...v0.15.2-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.15.1-alpha (July 8, 2024)

· 5 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.15.1-alpha minor release focuses on enhancing stability, performance, and usability. Memory usage has been significantly improved for the postgres and duckdb acceleration engines which now use stream processing. A new Delta Lake Data Connector has been added, sharing a delta-kernel-rs based implementation with the Databricks Data Connector supporting deletion vectors.

Highlights​

Improved memory usage for PostgreSQL and DuckDB acceleration engines: Large dataset acceleration with PostgreSQL and DuckDB engines has reduced memory consumption by streaming data directly to the accelerated table as it is read from the source.

Delta Lake Data Connector: A new Delta Lake Data Connector has been added for using Delta Lake outside of Databricks.

ODBC Data Connector Streaming: The ODBC Data Connector now streams results, reducing memory usage, and improving performance.

GraphQL Object Unnesting: The GraphQL Data Connector can automatically unnest objects from GraphQL queries using the unnest_depth parameter.

Breaking Changes​

None.

Contributors​

What's Changed​

Dependencies​

The MySQL, PostgreSQL, SQLite and DuckDB DataFusion TableProviders developed by Spice AI have been donated to the datafusion-contrib/datafusion-table-providers community repository.

From the v0.15.1-alpha release, a new dependency is taken on datafusion-contrib/datafusion-table-providers

Commits​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.15.0-alpha...v0.15.1-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.15-alpha (July 1, 2024)

· 5 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.15-alpha release introduces support for streaming databases changes with Change Data Capture (CDC) into accelerated tables via a new Debezium connector, configurable retry logic for data refresh, and the release of a new C# SDK to build with Spice in Dotnet.

Highlights in v0.15-alpha​

  • Debezium data connector with Change Data Capture (CDC): Sync accelerated datasets with Debezium data sources over Kafka in real-time.

  • Data Refresh Retries: By default, accelerated datasets attempt to retry data refreshes on transient errors. This behavior can be configured using refresh_retry_enabled and refresh_retry_max_attempts.

  • C# Client SDK: A new C# Client SDK has been released for developing applications in Dotnet.

Debezium data connector with Change Data Capture (CDC)​

Integrating Debezium CDC is straightforward. Get started with the Debezium CDC Sample, read more about CDC in Spice, and read the Debezium data connector documentation.

Example Spicepod using Debezium CDC:

datasets:
  - from: debezium:cdc.public.customer_addresses
    name: customer_addresses_cdc
    params:
      debezium_transport: kafka
      debezium_message_format: json
      kafka_bootstrap_servers: localhost:19092
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: changes

Data Refresh Retries​

Example Spicepod configuration limiting refresh retries to a maximum of 10 attempts:

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      refresh_retry_enabled: true
      refresh_retry_max_attempts: 10
      refresh_check_interval: 30s

Breaking Changes​

None.

New Contributors​

Contributors​

What's Changed​

Dependencies​

No major dependency updates.

Commits​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.14.1-alpha...v0.15.0-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.14.1-alpha (June 24, 2024)

· 5 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.14.1-alpha release is focused on quality, stability, and type support with improvements in PostgreSQL, DuckDB, and GraphQL data connectors.

Highlights​

  • PostgreSQL acceleration and data connector: Support for Composite Types and UUID data types.
  • DuckDB acceleration and data connector: Support for LargeUTF8 and DuckDB functions.
  • GraphQL data connector: Improved error handling on invalid query syntax.
  • Refresh SQL: Improved stability when overwriting STRUCT data types.

Breaking Changes​

None.

New Contributors​

Contributors​

  • @lukekim
  • @y-f-u
  • @ewgenius
  • @phillipleblanc
  • @Jeadie
  • @sgrebnov
  • @gloomweaver
  • @phungleson
  • @peasee
  • @digadeesh

What's Changed​

Dependencies​

No major dependency updates.

Commits​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.14.0-alpha...v0.14.1-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.