One post tagged with "client sdk"

The v0.15-alpha release introduces support for streaming databases changes with Change Data Capture (CDC) into accelerated tables via a new Debezium connector, configurable retry logic for data refresh, and the release of a new C# SDK to build with Spice in Dotnet.

Highlights in v0.15-alpha​

• Debezium data connector with Change Data Capture (CDC): Sync accelerated datasets with Debezium data sources over Kafka in real-time.

• Data Refresh Retries: By default, accelerated datasets attempt to retry data refreshes on transient errors. This behavior can be configured using refresh_retry_enabled and refresh_retry_max_attempts.

• C# Client SDK: A new C# Client SDK has been released for developing applications in Dotnet.

Debezium data connector with Change Data Capture (CDC)​

Integrating Debezium CDC is straightforward. Get started with the Debezium CDC Sample, read more about CDC in Spice, and read the Debezium data connector documentation.

Example Spicepod using Debezium CDC:

datasets:
  - from: debezium:cdc.public.customer_addresses
    name: customer_addresses_cdc
    params:
      debezium_transport: kafka
      debezium_message_format: json
      kafka_bootstrap_servers: localhost:19092
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: changes

Data Refresh Retries​

Example Spicepod configuration limiting refresh retries to a maximum of 10 attempts:

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      refresh_retry_enabled: true
      refresh_retry_max_attempts: 10
      refresh_check_interval: 30s

Breaking Changes​

None.

New Contributors​

• @rupurt made their first contribution in https://github.com/spiceai/spiceai/pull/1791

Contributors​

• @ewgenius
• @phillipleblanc
• @sgrebnov
• @y-f-u
• @rupurt
• @Jeadie
• @peasee

What's Changed​

Dependencies​

No major dependency updates.

Commits​

• Update version to 0.15.0-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1784
• Update helm for v0.14.1-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1786
• Run PR checks on PRs merging into feature-- branches by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1788
• Enable retries for accelerated table refresh by @sgrebnov in https://github.com/spiceai/spiceai/pull/1762
• enable more tpch benchmark queries as a result of decimal unparsing by @y-f-u in https://github.com/spiceai/spiceai/pull/1790
• add nix flake by @rupurt in https://github.com/spiceai/spiceai/pull/1791
• Support local and HF embedding models by @Jeadie in https://github.com/spiceai/spiceai/pull/1789
• fix(bin/spice): Implement custom Unmarshaller for DatasetOrReference by @peasee in https://github.com/spiceai/spiceai/pull/1787
• For windows, move symlink -> symlink_file. by @Jeadie in https://github.com/spiceai/spiceai/pull/1793
• docs: Add PULL_REQUEST_TEMPLATE.md by @peasee in https://github.com/spiceai/spiceai/pull/1794
• Fix Unsupported DataType: conversion for time predicates by @sgrebnov in https://github.com/spiceai/spiceai/pull/1795
• Use incremental backoff for initial dataset registration retries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1805
• Basic HTTP/S connector by @Jeadie in https://github.com/spiceai/spiceai/pull/1792
• Scale support for Snowflake fixed-point numbers by @sgrebnov in https://github.com/spiceai/spiceai/pull/1804
• bump datafusion federation to resolve the join query failures by @y-f-u in https://github.com/spiceai/spiceai/pull/1806
• fix: Stream PostgreSQL data in by @peasee in https://github.com/spiceai/spiceai/pull/1798
• Remove clippy::module_name_repetitions lint by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1812
• Improve Snowflake fixed-point numbers casting by @sgrebnov in https://github.com/spiceai/spiceai/pull/1809
• Case insensitive secret getter by @ewgenius in https://github.com/spiceai/spiceai/pull/1813
• refactor: Format TOML with Taplo by @peasee in https://github.com/spiceai/spiceai/pull/1808
• feat: Update PR template, add label enforcement in PR by @peasee in https://github.com/spiceai/spiceai/pull/1815
• fix bug that append may miss updates when the incremental changes are not able to be contained in one record batch by @y-f-u in https://github.com/spiceai/spiceai/pull/1817
• add integration test for inner join across federated table and accelerated table by @y-f-u in https://github.com/spiceai/spiceai/pull/1811
• Unify spicepod.llms into spicepod.models and refactor UX of spicepod.models by @Jeadie in https://github.com/spiceai/spiceai/pull/1818
• Fix issue with querying accelerated tables where the dataset name has a schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1823
• Fix schema support for refresh_sql and improve e2e tests by @sgrebnov in https://github.com/spiceai/spiceai/pull/1826
• feat: Add GraphQL unnesting by @peasee in https://github.com/spiceai/spiceai/pull/1822
• fix: Allow kind/optimization labels, increase Postgres test timeout by @peasee in https://github.com/spiceai/spiceai/pull/1830
• Implement Real-time acceleration updates via Debezium CDC by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1832
• Remove println statement from PG Connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/1835
• Don't try to "hot reload" Debezium accelerated datasets by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1837
• Create v1/search that performs vector search. by @Jeadie in https://github.com/spiceai/spiceai/pull/1836
• Align spicepod UX of embeddings with models by @Jeadie in https://github.com/spiceai/spiceai/pull/1829
• Add "cmake-build" feature to rdkafka for windows by @Jeadie in https://github.com/spiceai/spiceai/pull/1840
• Add a better error message when trying to configure refresh_mode=changes on a data connector that doesn't support it. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1839

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.14.1-alpha...v0.15.0-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

• Twitter: @spice_ai
• Discord: https://discord.gg/kZnTfneP5u
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]