2 posts tagged with "accelerators"

Spice v0.19-beta (Oct 7, 2024)

October 7, 2024 · 6 min read

Senior Software Engineer at Spice AI

Announcing the release of Spice v0.19-beta 📦

Spice v0.19.0-beta brings performance improvements for accelerators and expanded TPC-DS coverage. A new Azure Blob Storage data connector has also been added.

Highlights in v0.19.0-beta

Improved TPC-DS Coverage: Enhanced support for TPC-DS derived queries.

CLI SQL REPL: The CLI SQL REPL (spice sql) now supports multi-line editing and tab indentation. Note, a terminating semi-colon ';' is now required for each executed SQL block.

Azure Storage Data Connector: A new Azure Blob Storage data connector (abfs://) has been added, enabling federated SQL queries on files stored in Azure Blob-compatible endpoints, including Azure BlobFS (abfss://) and Azure Data Lake (adl://). Supported file formats can be specified using the file_format parameter.

Example spicepod.yml:

datasets:
  - from: abfs://foocontainer/taxi_sample.csv
    name: azure_test
    params:
      azure_account: spiceadls
      azure_access_key: abc123==
      file_format: csv

For a full list of supported files, see the Object Store File Formats documentation.

For more details, see the Azure Blob Storage Data Connector documentation.

Breaking Changes

Spice.ai Data Connector: The key for the Spice.ai Cloud Platform Data Connector has changed from spiceai to spice.ai. To upgrade, change uses of from: spiceai: to from: spice.ai:.
GitHub Data Connector: Pull Requests column login has been renamed to author.
CLI SQL REPL: A terminating semi-colon ';' is now required for each executed SQL block.
Spicepod Hot-Reload: When running spiced directly, hot-reload of spicepod.yml configuration is now disabled. Run with spice run to use hot-reload.

Contributors

@sgrebnov
@Jeadie
@Sevenannn
@peasee
@ewgenius
@slyons
@phillipleblanc
@lukekim

Dependencies

DataFusion Table Providers: Upgraded to rev 826814ab149aad8ee668454c83a0650fb8b18d60.

What's Changed

- Bump tonic from 0.12.2 to 0.12.3 by @dependabot in https://github.com/spiceai/spiceai/pull/2880
- Verify benchmark query results using snapshot testing (s3 connector) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2902
- Fix `paths-ignore:` by @Jeadie in https://github.com/spiceai/spiceai/pull/2906
- Rename `spiceai` data connector to `spice.ai` by @sgrebnov in https://github.com/spiceai/spiceai/pull/2899
- Update ROADMAP.md by @Jeadie in https://github.com/spiceai/spiceai/pull/2907
- Helm update for helm for 0.18.3-beta by @Jeadie in https://github.com/spiceai/spiceai/pull/2910
- Add tpcds queries by @Sevenannn in https://github.com/spiceai/spiceai/pull/2918
- Fix `paths-ignore` for docs. by @Jeadie in https://github.com/spiceai/spiceai/pull/2911
- feat: Support LIKE expressions in GitHub filter pushdown by @peasee in https://github.com/spiceai/spiceai/pull/2903
- feat: Support date comparison pushdown in GitHub connector by @peasee in https://github.com/spiceai/spiceai/pull/2904
- Improve aggregation and union queries unparsing by @sgrebnov in https://github.com/spiceai/spiceai/pull/2925
- Initialize file based accelerators on dataset reload by @Sevenannn in https://github.com/spiceai/spiceai/pull/2923
- Update spiceai/spiceai for next release by @Jeadie in https://github.com/spiceai/spiceai/pull/2928
- Verify TPC-H benchmark query results for arrow acceleration by @sgrebnov in https://github.com/spiceai/spiceai/pull/2927
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/2912
- Use structured output for NSQL by @Jeadie in https://github.com/spiceai/spiceai/pull/2922
- Update TPC-DS queries to use supported date addition format by @sgrebnov in https://github.com/spiceai/spiceai/pull/2930
- Add busy_timeout accelerator param for Sqlite by @Sevenannn in https://github.com/spiceai/spiceai/pull/2855
- Use Cosine Similarity in vector search by @Jeadie in https://github.com/spiceai/spiceai/pull/2932
- Add support for passing `x-spiceai-app-id` metadata in spiceai data connector by @ewgenius in https://github.com/spiceai/spiceai/pull/2934
- docs: update beta accelerator criteria by @peasee in https://github.com/spiceai/spiceai/pull/2905
- Azure Connector implementation by @slyons in https://github.com/spiceai/spiceai/pull/2926
- Local embedding model from relative paths by @Jeadie in https://github.com/spiceai/spiceai/pull/2908
- Add Markdown aware chunker when `params.file_format: md`. by @Jeadie in https://github.com/spiceai/spiceai/pull/2943
- 'spice version' without structured logging by @Jeadie in https://github.com/spiceai/spiceai/pull/2944
- Bump tempfile from 3.12.0 to 3.13.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2878
- feat: GraphQL commit query parameters by @peasee in https://github.com/spiceai/spiceai/pull/2945
- Update OpenAI client and use new request fields by @Jeadie in https://github.com/spiceai/spiceai/pull/2951
- refactor: Rename GitHub pulls login to author by @peasee in https://github.com/spiceai/spiceai/pull/2954
- Run tpcds benchmarks for accelerators by @Sevenannn in https://github.com/spiceai/spiceai/pull/2853
- Add spiced arg `--pods-watcher-enabled`. Watcher disabled by default for spiced. by @ewgenius in https://github.com/spiceai/spiceai/pull/2953
- Add error message when spicepod has embeddings or models without '--features models' by @Jeadie in https://github.com/spiceai/spiceai/pull/2952
- Adding multi-line editing and tab indentation to sql REPL by @slyons in https://github.com/spiceai/spiceai/pull/2949
- Update MySQL ghcr image to include tpcds data by @Sevenannn in https://github.com/spiceai/spiceai/pull/2941
- Document DataFusion limitation: The context only support single SQL Statement, Date Arithmetic like date + 3 not supported by @Sevenannn in https://github.com/spiceai/spiceai/pull/2970
- Bump snafu from 0.8.4 to 0.8.5 by @dependabot in https://github.com/spiceai/spiceai/pull/2876
- Bump async-trait from 0.1.82 to 0.1.83 by @dependabot in https://github.com/spiceai/spiceai/pull/2879
- Bump async-graphql from 7.0.9 to 7.0.11 in the cargo group by @dependabot in https://github.com/spiceai/spiceai/pull/2950
- Verify TPC-H benchmark query results for MySQL by @sgrebnov in https://github.com/spiceai/spiceai/pull/2972
- Verify TPCH benchmark query results for Postgres by @sgrebnov in https://github.com/spiceai/spiceai/pull/2973
- Verify TPCH benchmark query results for sqlite acceleration by @sgrebnov in https://github.com/spiceai/spiceai/pull/2974
- Verify TPCH benchmark query results for duckdb (in-memory) acceleration by @sgrebnov in https://github.com/spiceai/spiceai/pull/2975
- Support for `mdx` file extensions to apply a markdown splitter by @ewgenius in https://github.com/spiceai/spiceai/pull/2977
- Don't assume first vector or content will be non-null/zero by @Jeadie in https://github.com/spiceai/spiceai/pull/2940
- use custom chunk sizers for HF, local and OpenAI models by @Jeadie in https://github.com/spiceai/spiceai/pull/2971
- Ensure we return N unique documents, not N unique chunks by @Jeadie in https://github.com/spiceai/spiceai/pull/2976
- Fix issues parsing `messages[*].tool_calls` for local models by @Jeadie in https://github.com/spiceai/spiceai/pull/2957
- text -> SQL trait to customise per model. by @Jeadie in https://github.com/spiceai/spiceai/pull/2942
- Remove system message from ToolUsingChat. by @Jeadie in https://github.com/spiceai/spiceai/pull/2978
- Make logical plan to sql more robust (improve ORDER BY; support `round` for Postgres) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2984
- Add connection_pool_size parameter for Postgres accelerator by @Sevenannn in https://github.com/spiceai/spiceai/pull/2969
- Fix dataset configure prompt by @sgrebnov in https://github.com/spiceai/spiceai/pull/2991
- Verify TPCH benchmark query results for Databricks(odbc) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2989
- Verify TPCH benchmark query results for Databricks (delta_lake) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2982
- Set log level for anonymous telemetry traces to `trace` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2995
- Improvements to issue templates by @lukekim in https://github.com/spiceai/spiceai/pull/2992
- `spice login` writes to `.env.local` if present by @slyons in https://github.com/spiceai/spiceai/pull/2996

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v0.18.3-beta...v0.19.0-beta>

Resources

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

Twitter: @spice_ai
Discord: https://discord.gg/kZnTfneP5u
Telegram: Spice AI Discussion
Reddit: https://www.reddit.com/r/spiceai
Email: [email protected]

Spice v0.12-alpha (April 29, 2024)

April 29, 2024 · 4 min read

Luke Kim

Founder and CEO of Spice AI

The v0.12-alpha release introduces Clickhouse and Apache Spark data connectors, adds support for limiting refresh data periods for temporal datasets, and includes upgraded Spice Client SDKs compatible with Spice OSS.

Highlights

Clickhouse data connector: Use Clickhouse as a data source with the clickhouse: scheme.
Apache Spark Connect data connector: Use Apache Spark Connect connections as a data source using the spark: scheme.
Refresh data window: Limit accelerated dataset data refreshes to the specified window, as a duration from now configuration setting, for faster and more efficient refreshes.
ODBC data connector: Use ODBC connections as a data source using the odbc: scheme. The ODBC data connector is currently optional and not included in default builds. It can be conditionally compiled using the odbc cargo feature when building from source.
Spice Client SDK Support: The official Spice SDKs have been upgraded with support for Spice OSS.

Breaking Changes

Refresh interval: The refresh_interval acceleration setting and been changed to refresh_check_interval to make it clearer it is the check versus the data interval.

Contributors

@phillipleblanc
@Jeadie
@ewgenius
@sgrebnov
@y-f-u
@lukekim
@digadeesh
@gloomweaver
@edmondop
@mach-kernel

New Contributors

Thanks to @mach-kernel who made their first contribution in https://github.com/spiceai/spiceai/pull/1204 by adding the ODBC data connector!

What's Changed

Update helm version by @Jeadie in https://github.com/spiceai/spiceai/pull/1167
Handle and trace errors in secret stores by @ewgenius in https://github.com/spiceai/spiceai/pull/1149
bump the release versions to 0.12.0 by @y-f-u in https://github.com/spiceai/spiceai/pull/1171
Don't fail acknowledgments flow if no changes detected by @ewgenius in https://github.com/spiceai/spiceai/pull/1170
Allow Spice CLI to control runtime installation on Windows by @sgrebnov in https://github.com/spiceai/spiceai/pull/1173
Allow SELECT count(*) for Sqlite Data Accelerator by @sgrebnov in https://github.com/spiceai/spiceai/pull/1166
add refresh_period param in acceleration by @y-f-u in https://github.com/spiceai/spiceai/pull/1180
Properly support Spark Connect filter pushdown by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1186
Avoid rate-limiting on arduino/setup-protoc@v3 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1189
Clickhouse DataConnector base implementation by @gloomweaver in https://github.com/spiceai/spiceai/pull/1168
rename refresh_interval to refresh_check_interval by @y-f-u in https://github.com/spiceai/spiceai/pull/1190
Fix timestamp & add support for Decimal to Databricks/Spark by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1194
Convert temporal column and refresh period to datafusion expr by @y-f-u in https://github.com/spiceai/spiceai/pull/1187
Hot reload accelerated table on dataset update by @ewgenius in https://github.com/spiceai/spiceai/pull/1195
Upgrade DataFusion to 37.1 & DuckDB to 10.2 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1200
Update version.txt for 0.11.2 release by @digadeesh in https://github.com/spiceai/spiceai/pull/1199
Clickhouse E2E by @gloomweaver in https://github.com/spiceai/spiceai/pull/1193
Clickhouse: fix darwin ci pipeline by @gloomweaver in https://github.com/spiceai/spiceai/pull/1201
Add table_type to show tables in Spice SQL & update next version to v0.12.0-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1206
print WARN if time_column does not exists in federated schema by @y-f-u in https://github.com/spiceai/spiceai/pull/1207
Add FallbackOnZeroResultsScanExec for executing an input ExecutionPlan and optionally falling back to a TableProvider.scan() if the input has zero results by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1196
Clickhouse refactor connection code and set secure option by @gloomweaver in https://github.com/spiceai/spiceai/pull/1198
E2E: reusable Spice installation by @sgrebnov in https://github.com/spiceai/spiceai/pull/1205
Clickhouse block_to_arrow unit test by @gloomweaver in https://github.com/spiceai/spiceai/pull/1202
rename refresh_period to refresh_data_period by @y-f-u in https://github.com/spiceai/spiceai/pull/1210
Refactor E2E tests: dataset verification and PostgreSQL installation by @sgrebnov in https://github.com/spiceai/spiceai/pull/1211
Add BI dashboard acceleration video to README.md by @lukekim in https://github.com/spiceai/spiceai/pull/1219
Improve clarity and consistency of output messages by @lukekim in https://github.com/spiceai/spiceai/pull/1214
Update ROADMAP Apr 29, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/1220
Stand-alone Spark Connect: Isolate Spark Connect from Databricks Connect to make it reusable by @edmondop in https://github.com/spiceai/spiceai/pull/1213
Optimize build time in dev mode by @gloomweaver in https://github.com/spiceai/spiceai/pull/1215
Feature: Support ODBC reads using unixodbc by @mach-kernel in https://github.com/spiceai/spiceai/pull/1204
Use non-fork deltalake by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1223
Support Date32 & Date64 in arrow_sql_gen by @Jeadie in https://github.com/spiceai/spiceai/pull/1217
Update REPL output to be consistent with the latest Spice version by @sgrebnov in https://github.com/spiceai/spiceai/pull/1231
rename refresh_data_period to refresh_data_window by @y-f-u in https://github.com/spiceai/spiceai/pull/1233
Update README.md to include ODBC, Spark Connect, and Clickhouse data connectors in support data connector matrix. by @digadeesh in https://github.com/spiceai/spiceai/pull/1234

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.11.1-alpha...v0.12.0-alpha

Resources

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

Twitter: @spice_ai
Discord: https://discord.gg/kZnTfneP5u
Telegram: Spice AI Discussion
Reddit: https://www.reddit.com/r/spiceai
Email: [email protected]

Highlights in v0.19.0-beta​

Breaking Changes​

Contributors​

Dependencies​

What's Changed​

Resources​

Community​

Highlights​

Breaking Changes​

Contributors​

New Contributors​

What's Changed​

Resources​

Community​

Highlights in v0.19.0-beta

Breaking Changes

Contributors

Dependencies

What's Changed

Resources

Community

Highlights

Breaking Changes

Contributors

New Contributors

What's Changed

Resources

Community