Spice v0.19.4-beta (Oct 30, 2024)
Announcing the release of Spice v0.19.4-beta ☘️
Spice v0.19.4-beta introduces a new localpod Data Connector, improvements to accelerator resiliency and control, and a new configuration to control when accelerated datasets are considered ready.
Highlights in v0.19.4
localpod Connector: Implement a "tiered" acceleration strategy with a new localpod Data Connector that can be used to accelerate datasets from other datasets registered in Spice.
datasets:
- from: s3://my_bucket/my_dataset
name: my_dataset
acceleration:
enabled: true
engine: duckdb
mode: file
refresh_check_interval: 60s
- from: localpod:my_dataset
name: my_localpod_dataset
acceleration:
enabled: true
Refreshes on the localpod's parent dataset will automatically be synchronized with the localpod dataset.
Improved Accelerator Resiliency: When Spice is restarted, if the federated source for a dataset configured with a file-based accelerator is not available, the dataset will still load from the existing file data and will attempt to connect to the federated source in the background for future refreshes.
Accelerator Ready State: Control when an accelerated dataset is considered "ready" by the runtime with the new ready_state parameter.
datasets:
- from: s3://my_bucket/my_dataset
name: my_dataset
acceleration:
enabled: true
ready_state: on_load # or on_registration
ready_state: on_load: Default. The dataset is considered ready after the initial load of the accelerated data. For file-based accelerated datasets that have existing data, this means the dataset is ready immediately.ready_state: on_registration: The dataset is considered ready when the dataset is registered in Spice. Queries against this dataset before the data is loaded will fallback to the federated source.
Breaking changes
Accelerated datasets configured with ready_state: on_load (the default behavior) that are not ready will return an error instead of returning zero results.
Contributors
- @Sevenannn
- @peasee
- @phillipleblanc
- @sgrebnov
- @barracudarin
- @Jeadie
- @ewgenius
What's Changed
- Update helm for v0.19.3-beta by @ewgenius in <https://github.com/spiceai/spiceai/pull/3274>
- docs: Mark GitHub as Beta in README.md by @peasee in <https://github.com/spiceai/spiceai/pull/3272>
- Fix docker publish by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3273>
- Add SQLite TPC-DS Limitations: `ROLLUP` and `GROUPING` by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3277>
- Update version to 1.0.0-rc.1 by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3276>
- Synchronize localpod acceleration with parent acceleration refreshes by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3264>
- feat: Update Datafusion, promote DuckDB and MySQL by @peasee in <https://github.com/spiceai/spiceai/pull/3278>
- Add SQLite TPC-DS Limitations: `stddev` by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3279>
- fix indentation issue with service annotations by @barracudarin in <https://github.com/spiceai/spiceai/pull/3281>
- fix: Expose GitHub ratelimit errors by @peasee in <https://github.com/spiceai/spiceai/pull/3258>
- Revert Datafusion parquet changes by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3286>
- Promote arrow accelerator to beta by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3287>
- Add SQLite TPC-DS Limitations: casting to DECIMAL by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3282>
- Accelerated datasets can fallback to federated source while loading by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3280>
- Enable overlap_size correctly by @Jeadie in <https://github.com/spiceai/spiceai/pull/3229>
- Avoid duplicated filter conditions in rewritten SQL by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3284>
- Fix SQLite records conversion with NULL in first row by @sgrebnov in <https://github.com/spiceai/spiceai/pull/3295>
- fix: Update datafusion by @peasee in <https://github.com/spiceai/spiceai/pull/3297>
- Display shorter name for benchmark workflow matrix by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3299>
- Update `spice_sys_dataset_checkpoint` to store federated table schema by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3303>
- Update postgres connector/accelerator snapshot by @Sevenannn in <https://github.com/spiceai/spiceai/pull/3298>
- Accelerated tables with existing file data can load without a connection to the federated source by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3306>
- Ensure synchronized tables complete their insertion at the same time by @phillipleblanc in <https://github.com/spiceai/spiceai/pull/3307>
**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v0.19.3-beta...v0.19.4-beta>
Resources
Community
Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.
- Twitter: @spice_ai
- Discord: https://discord.gg/kZnTfneP5u
- Telegram: Spice AI Discussion
- Reddit: https://www.reddit.com/r/spiceai
- Email: [email protected]
