Skip to main content

4 posts tagged with "spark"

View All Tags

Spice v1.0-rc.5 (Jan 13, 2025)

· 7 min read
Evgenii Khramkov
Evgenii Khramkov
Senior Software Engineer at Spice AI

Announcing the release of Spice v1.0-rc.5 🛠️

Spice v1.0.0-rc.5 is the fifth release candidate for the first major version of Spice.ai OSS. This release focuses production readiness and critical bug fixes. In addition, a new DynamoDB data connector has been added along with automatic detection for GPU acceleration when running Spice using the CLI.

Highlights in v1.0-rc.5​

  • Automatic GPU Acceleration Detection: Automatically detect and utilize GPU acceleration when running by CLI. Install AI components locally using the CLI command spice install ai. Currently supports NVIdia CUDA and Apple Metal (M-series).

  • DynamoDB Data Connector: Query AWS DynamoDB tables using SQL with the new DynamoDB Data Connector.

datasets:
  - from: dynamodb:users
    name: users
    params:
      dynamodb_aws_region: us-west-2
      dynamodb_aws_access_key_id: ${secrets:aws_access_key_id}
      dynamodb_aws_secret_access_key: ${secrets:aws_secret_access_key}
    acceleration:
      enabled: true
sql> describe users;
+----------------+-----------+-------------+
| column_name    | data_type | is_nullable |
+----------------+-----------+-------------+
| created_at     | Utf8      | YES         |
| date_of_birth  | Utf8      | YES         |
| email          | Utf8      | YES         |
| account_status | Utf8      | YES         |
| updated_at     | Utf8      | YES         |
| full_name      | Utf8      | YES         |
| ...                                      |
+----------------+-----------+-------------+

  • File Data Connector: Graduated to Stable.

  • Dremio Data Connector: Graduated to Release Candidate (RC).

  • Spice.ai, Spark, and Snowflake Data Connectors: Graduated to Beta.

Dependencies​

No major dependency changes.

Contributors​

  • @Jeadie
  • @phillipleblanc
  • @ewgenius
  • @peasee
  • @Sevenannn
  • @lukekim

What's Changed​

* Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/4190
* Ensure non-nullity of primary keys in `MemTable`; check validity of initial data. by @Jeadie in https://github.com/spiceai/spiceai/pull/4158
* Bump version to v1.0.0 stable by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4191
* Fix metal + models download by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4193
* Update spice.ai connector beta roadmap by @ewgenius in https://github.com/spiceai/spiceai/pull/4194
* feat: verify on zero results snapshots by @peasee in https://github.com/spiceai/spiceai/pull/4195
* Add throughput test module to `test-framework` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4196
* Update Spice.ai TPCH snapshots by @ewgenius in https://github.com/spiceai/spiceai/pull/4202
* Replace all usage of `lazy_static!` with `LazyLock` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4199
* Fix model + metal download by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4200
* Run Clickbench for Dremio by @Sevenannn in https://github.com/spiceai/spiceai/pull/4138
* Update openapi.json by @github-actions in https://github.com/spiceai/spiceai/pull/4205
* Fix the typo in connector stable criteria by @Sevenannn in https://github.com/spiceai/spiceai/pull/4213
* feat: Add throughput test example by @peasee in https://github.com/spiceai/spiceai/pull/4214
* feat: calculate throughput test query percentiles by @peasee in https://github.com/spiceai/spiceai/pull/4215
* feat: Add throughput test to actions by @peasee in https://github.com/spiceai/spiceai/pull/4217
* Implement DynamoDB Data Connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4218
* 1.0 doc updates by @lukekim in https://github.com/spiceai/spiceai/pull/4181
* Improve clarity and concison of use-cases by @lukekim in https://github.com/spiceai/spiceai/pull/4220
* Remove macOS Intel build by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4221
* fix: Test operator throughput test workflow by @peasee in https://github.com/spiceai/spiceai/pull/4222
* DynamoDB: Automatically load AWS credentials from IAM roles if access key not provided by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4226
* File connector clickbench snapshots results by @ewgenius in https://github.com/spiceai/spiceai/pull/4225
* Spice.ai Catalog Connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4204
* feat: Add test framework metrics collection by @peasee in https://github.com/spiceai/spiceai/pull/4227
* Add badges for build/test status on README.md by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4228
* Release Dremio to RC by @Sevenannn in https://github.com/spiceai/spiceai/pull/4224
* feat: Add more test spicepods by @peasee in https://github.com/spiceai/spiceai/pull/4229
* feat: Add load test to testoperator by @peasee in https://github.com/spiceai/spiceai/pull/4231
* Add TSV format to all `object_store`-based connectors by @Jeadie in https://github.com/spiceai/spiceai/pull/4192
* Move test-framework to dev-dependencies for Runtime by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4230
* Document limitation for correlated subqueries in TPCH for Spice.ai connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4235
* Changes for CUDA by @Jeadie in https://github.com/spiceai/spiceai/pull/4130
* fix: Collect batches from test framework, load test updates by @peasee in https://github.com/spiceai/spiceai/pull/4234
* Suppress opentelemetry_sdk warnings - they aren't useful by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4243
* fix: Set dataset status first, update test framework by @peasee in https://github.com/spiceai/spiceai/pull/4244
* feat: Re-enable defaults on test spicepods by @peasee in https://github.com/spiceai/spiceai/pull/4248
* Add usage for streaming local models; Fix spice chat usage bar TPS expansion by @Jeadie in https://github.com/spiceai/spiceai/pull/4232
* refactor: Use composite testoperator setup, add query overrides by @peasee in https://github.com/spiceai/spiceai/pull/4246
* Enable expand_views_at_output for DF optimizer and transform schema to expanded view types by @ewgenius in https://github.com/spiceai/spiceai/pull/4237
* Add throughput test spicepod for databricks delta mode connector by @Sevenannn in https://github.com/spiceai/spiceai/pull/4241
* Spark data connector - update and enable TPCH and TPCDS benchmarks by @ewgenius in https://github.com/spiceai/spiceai/pull/4240
* Increase the timeout minutes of load test to 10 hours by @Sevenannn in https://github.com/spiceai/spiceai/pull/4254
* Improve partition column counts error for delta table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4247
* Add e2e test for databricks catalog connector (mode: delta_lake) by @Sevenannn in https://github.com/spiceai/spiceai/pull/4255
* Spark connector integration tests by @ewgenius in https://github.com/spiceai/spiceai/pull/4256
* Run benchmark test with the new test framework by @Sevenannn in https://github.com/spiceai/spiceai/pull/4245
* Configure databricks delta secrets to run load test by @Sevenannn in https://github.com/spiceai/spiceai/pull/4257
* Support `properties` for emitted telemetry by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4249
* feat: Add `ready_wait` test operator workflow input by @peasee in https://github.com/spiceai/spiceai/pull/4259
* Handle 'LargeStringArray' for embedding tables by @Jeadie in https://github.com/spiceai/spiceai/pull/4263
* `llms` tests for alpha/beta model criteria  by @Jeadie in https://github.com/spiceai/spiceai/pull/4261
* Configurable runner type for load and throughput tests by @ewgenius in https://github.com/spiceai/spiceai/pull/4262
* Handle NULL partition columns for Delta Lake tables by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4264
* Add integration test for Snowflake by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4266
* Add Snowflake TPCH queries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4268
* Handle `LargeStringArray` in `v1/search`. by @Jeadie in https://github.com/spiceai/spiceai/pull/4265
* Fix `build_cuda` in Update spiced_docker.yml by @Jeadie in https://github.com/spiceai/spiceai/pull/4269
* Run Snowflake benchmark in GitHub Actions by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4270
* Allow Snowflake query override for CI tests by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4271
* Don't run GPU builds for trunk by @Jeadie in https://github.com/spiceai/spiceai/pull/4272
* Fix InvalidTypeAction not working by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4273
* Add xAI key to llm integration tests by @Jeadie in https://github.com/spiceai/spiceai/pull/4274
* Update openai snapshots by @Jeadie in https://github.com/spiceai/spiceai/pull/4275
* Fix federation bug for correlated subqueries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4276
* Update end_game.md by @ewgenius in https://github.com/spiceai/spiceai/pull/4278
* Promote Snowflake to Beta by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4277
* Set version to 1.0.0-rc.5 by @ewgenius in https://github.com/spiceai/spiceai/pull/4283
* Update cargo.lock by @ewgenius in https://github.com/spiceai/spiceai/pull/4285
* Update spice.ai data connector snapshots by @ewgenius in https://github.com/spiceai/spiceai/pull/4281
* Promote the Spice.ai Data Connector to Beta by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4282
* Revert change to  `integration_models__models__search__openai_chunking_response.snap` by @Jeadie in https://github.com/spiceai/spiceai/pull/4279
* Allow for a subset of build artifacts to be published to minio by @Jeadie in https://github.com/spiceai/spiceai/pull/4280
* Promote File Data Connector to Stable by @ewgenius in https://github.com/spiceai/spiceai/pull/4286
* Add Iceberg to Supported Catalogs by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4287
* Update openapi.json by @github-actions in https://github.com/spiceai/spiceai/pull/4289
* Fix Spark benchmark credentials, add back overrides by @ewgenius in https://github.com/spiceai/spiceai/pull/4295
* Promote Spark Data Connector to Beta by @ewgenius in https://github.com/spiceai/spiceai/pull/4296
* Add Dremio throughput test spicepod by @Sevenannn in https://github.com/spiceai/spiceai/pull/4233
* Add error message for invalid databricks mode parameter by @Sevenannn in https://github.com/spiceai/spiceai/pull/4299
* Fix pre-release check to look for `build` string by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4300
* Promote databricks catalog connector (mode: delta_lake) to beta by @Sevenannn in https://github.com/spiceai/spiceai/pull/4301
* Properly delegate `load_table` to Rest Catalog by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4303
* Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/4302
* docs: Update ROADMAP.md by @peasee in https://github.com/spiceai/spiceai/pull/4306
* v1.0.0-rc.5 Release Notes by @ewgenius in https://github.com/spiceai/spiceai/pull/4298

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v1.0.0-rc.4...v1.0.0-rc.5

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

Spice v0.12-alpha (April 29, 2024)

· 4 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.12-alpha release introduces Clickhouse and Apache Spark data connectors, adds support for limiting refresh data periods for temporal datasets, and includes upgraded Spice Client SDKs compatible with Spice OSS.

Highlights​

  • Clickhouse data connector: Use Clickhouse as a data source with the clickhouse: scheme.

  • Apache Spark Connect data connector: Use Apache Spark Connect connections as a data source using the spark: scheme.

  • Refresh data window: Limit accelerated dataset data refreshes to the specified window, as a duration from now configuration setting, for faster and more efficient refreshes.

  • ODBC data connector: Use ODBC connections as a data source using the odbc: scheme. The ODBC data connector is currently optional and not included in default builds. It can be conditionally compiled using the odbc cargo feature when building from source.

  • Spice Client SDK Support: The official Spice SDKs have been upgraded with support for Spice OSS.

Breaking Changes​

  • Refresh interval: The refresh_interval acceleration setting and been changed to refresh_check_interval to make it clearer it is the check versus the data interval.

Contributors​

  • @phillipleblanc
  • @Jeadie
  • @ewgenius
  • @sgrebnov
  • @y-f-u
  • @lukekim
  • @digadeesh
  • @gloomweaver
  • @edmondop
  • @mach-kernel

New Contributors​

What's Changed​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.11.1-alpha...v0.12.0-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

Spice v0.11.1-alpha (April 22, 2024)

· 3 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.11.1-alpha release introduces retention policies for accelerated datasets, native Windows installation support, and integration of catalog and schema settings for the Databricks Spark connector. Several bugs have also been fixed for improved stability.

Highlights​

  • Retention Policies for Accelerated Datasets: Automatic eviction of data from accelerated time-series datasets when a specified temporal column exceeds the retention period, optimizing resource utilization.

  • Windows Installation Support: Native Windows installation support, including upgrades.

  • Databricks Spark Connect Catalog and Schema Settings: Improved translation between DataFusion and Spark, providing better Spark Catalog support.

Contributors​

  • @phillipleblanc
  • @Jeadie
  • @ewgenius
  • @sgrebnov
  • @y-f-u
  • @lukekim
  • @digadeesh
  • @Sevenannn
  • @gloomweaver

New in this release​

What's Changed​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.11.0-alpha...v0.11.1-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

Spice v0.11-alpha (April 15, 2024)

· 4 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The Spice v0.11-alpha release significantly improves the Databricks data connector with Databricks Connect (Spark Connect) support, adds the DuckDB data connector, and adds the AWS Secrets Manager secret store. In addition, enhanced control over accelerated dataset refreshes, improved SSL security for MySQL and PostgreSQL connections, and overall stability improvements have been added.

Highlights in v0.11-alpha​

DuckDB data connector: Use DuckDB databases or connections as a data source.

AWS Secrets Manager Secret Store: Use AWS Secrets Managers as a secret store.

Custom Refresh SQL: Specify a custom SQL query for dataset refresh using refresh_sql.

Dataset Refresh API: Trigger a dataset refresh using the new CLI command spice refresh or via API.

Expanded SSL support for Postgres: SSL mode now supports disable, require, prefer, verify-ca, verify-full options with the default mode changed to require. Added pg_sslrootcert parameter for setting a custom root certificate and the pg_insecure parameter is no longer supported.

Databricks Connect: Choose between using Spark Connect or Delta Lake when using the Databricks data connector for improved performance.

Improved SSL support for Postgres: ssl mode now supports disable, require, prefer, verify-ca, verify-full options with default mode changed to require. Added pg_sslrootcert parameter to allow setting custom root cert for postgres connector, pg_insecure parameter is no longer supported as redundant.

Internal architecture refactor: The internal architecture of spiced was refactored to simplify the creation data components and to improve alignment with DataFusion concepts.

New Contributors​

@edmondop's first contribution github.com/spiceai/spiceai/pull/1110!

Contributors​

  • @phillipleblanc
  • @Jeadie
  • @ewgenius
  • @sgrebnov
  • @y-f-u
  • @lukekim
  • @digadeesh
  • @Sevenannn
  • @gloomweaver
  • @ahirner

New in this release​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.10.2-alpha...v0.11.0-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.