2 posts tagged with "operability"

The v0.13.3-alpha release is focused on quality and stability with improvements to metrics, telemetry, and operability.

Highlights​

Ready API: - Add /v1/ready API that returns success once all datasets and models are loaded and ready.

Enhanced Grafana dashboard: The dashboard now includes charts for query duration and failures, the last update time of accelerated datasets, the count of refresh errors, and the last successful time the runtime was able to access federated datasets

Contributors​

• @Jeadie
• @ewgenius
• @phillipleblanc
• @sgrebnov
• @gloomweaver
• @y-f-u
• @mach-kernel

What's Changed​

Dependencies​

• DuckDB 1.0.0: Upgrades embedded DuckDB to 1.0.0.

Commits​

• Scalar UDF array_distance as euclidean distance between Float32[] by @Jeadie in https://github.com/spiceai/spiceai/pull/1601
• Update version to v0.14.0-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1614
• Update helm for v0.13.2-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1618
• Upgrade duckdb-rs to DuckDB 1.0.0 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1615
• initial idea for 'POST v1/assist' by @Jeadie in https://github.com/spiceai/spiceai/pull/1585
• openai server trait and move HTTP endpoints to crates/runtime/src/http/v1/ by @Jeadie in https://github.com/spiceai/spiceai/pull/1619
• Add branching policy & updated endgame instructions by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1620
• Update Cargo.lock & add CI check for updated Cargo.lock by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1627
• Add first-class support for views by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1622
• Add /v1/ready API that returns 200 when all datasets have loaded by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1629
• Separate NQL logic from LLM Chat messages, and add OpenAI compatiblility per LLM trait. by @Jeadie in https://github.com/spiceai/spiceai/pull/1628
• Log queries failing on get_flight_info step (Flight Api) by @sgrebnov in https://github.com/spiceai/spiceai/pull/1626
• Graphql Data Connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1624
• GraphQL improved Error formatting, proper format request body by @gloomweaver in https://github.com/spiceai/spiceai/pull/1637
• Fix v1/assist response and panic bug. Include primary keys in response too by @Jeadie in https://github.com/spiceai/spiceai/pull/1635
• skip integration test if no secret by @y-f-u in https://github.com/spiceai/spiceai/pull/1638
• [append] Refresher::get_latest_timestamp / get_df to add refresh_sql predicates to scan by @mach-kernel in https://github.com/spiceai/spiceai/pull/1636
• GraphQL integration test by @gloomweaver in https://github.com/spiceai/spiceai/pull/1600
• Add err_code to query_failures metric by @sgrebnov in https://github.com/spiceai/spiceai/pull/1639
• use epoch_ms to replace epoch to work with timestamptz by @y-f-u in https://github.com/spiceai/spiceai/pull/1641
• fix the schema mismatch issue on the fallback plan use schema casting by @y-f-u in https://github.com/spiceai/spiceai/pull/1642
• bug report template update by @y-f-u in https://github.com/spiceai/spiceai/pull/1640
• Add query duration, failures and accelerated dataset metrics to Grafana dashboard by @sgrebnov in https://github.com/spiceai/spiceai/pull/1598
• Fix FTP/sftp support for ObjectStoreMetadataTable & ObjectStoreTextTable by @Jeadie in https://github.com/spiceai/spiceai/pull/1649
• Support accelerated embedding tables in v1/assist by @Jeadie in https://github.com/spiceai/spiceai/pull/1648
• GraphQL pagination, limit pushdown and refactor by @gloomweaver in https://github.com/spiceai/spiceai/pull/1643
• Support indexes in accelerated tables by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1644
• Federated datasets availability monitoring by @sgrebnov in https://github.com/spiceai/spiceai/pull/1650
• Reset federated dataset availability during dataset registration by @sgrebnov in https://github.com/spiceai/spiceai/pull/1661
• Change to v0.13.3-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1666
• Add Time Since Offline chart to Grafana dashboard by @sgrebnov in https://github.com/spiceai/spiceai/pull/1664

• readme fix to correct the number of rows for show tables by @y-f-u in https://github.com/spiceai/spiceai/pull/1667
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1668
• Add missing dependency on arrow_sql_gen from duckdb data_component by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1669

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.2-alpha...v0.13.3-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

• Twitter: @spice_ai
• Discord: https://discord.gg/kZnTfneP5u
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]

The v0.13.1-alpha release of Spice is a minor update focused on stability, quality, and operability. Query result caching provides protection against bursts of queries and schema support for datasets has been added logical grouping. An issue where Refresh SQL predicates were not pushed down underlying data sources has been resolved along with improved Acceleration Refresh logging.

Highlights in v0.13.1-alpha​

• Results Caching: Introduced query results caching to handle bursts of requests and support caching of non-accelerated results, such as refresh data returned on zero results. Results caching is enabled by default with a 1s item time-to-live (TTL). Learn more.

• Query History Logging: Recent queries are now logged in the new spice.runtime.query_history dataset with a default retention of 24-hours. Query history is initially enabled for HTTP queries only (not Arrow Flight queries).

• Dataset Schemas: Added support for dataset schemas, allowing logical grouping of datasets by separating the schema name from the table name with a .. E.g.

  datasets:
    - from: mysql:app1.identities
      name: app.users
  
    - from: postgres:app2.purchases
      name: app.purchases
  

  In this example, queries against app.users will be federated to my_schema.my_table, and app.purchases will be federated to app2.purchases.

Contributors​

@y-f-u @Jeadie @sgrebnov @ewgenius @phillipleblanc @lukekim @gloomweaver @Sevenannn

New in this release​

• Add more type support on mysql connector by @y-f-u in https://github.com/spiceai/spiceai/pull/1449
• Add in-memory caching support for Arrow Flight queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1450
• Fix the table reference to use the full table reference, not just the table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1460
• Make file_format parameter required for S3/FTP/SFTP connector by @ewgenius in https://github.com/spiceai/spiceai/pull/1455
• Add more verbose logging when acceleration refresh update is finished by @y-f-u in https://github.com/spiceai/spiceai/pull/1453
• Fix snowflake dataset path when using federation query by @y-f-u in https://github.com/spiceai/spiceai/pull/1474
• Update cargo to use spiceai datafusion fork by @y-f-u in https://github.com/spiceai/spiceai/pull/1475
• Enable in-memory results caching by default by @sgrebnov in https://github.com/spiceai/spiceai/pull/1473
• Add basic integration test for MySQL federation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1477
• Update results_cache config names per final spec by @sgrebnov in https://github.com/spiceai/spiceai/pull/1487
• Add DuckDB quickstart to E2E tests by @lukekim in https://github.com/spiceai/spiceai/pull/1461
• Add X-Cache header for http queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1472
• Add telemetry for in-memory caching by @sgrebnov in https://github.com/spiceai/spiceai/pull/1456
• Pin Git dependencies to a specific commit hash by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1490
• Detect file_format from dataset path by @ewgenius in https://github.com/spiceai/spiceai/pull/1489
• Add file_format to helm chart sample dataset by @ewgenius in https://github.com/spiceai/spiceai/pull/1493
• Improve duckdb data connector error messages by @Sevenannn in https://github.com/spiceai/spiceai/pull/1486
• Add file_format prompt for s3 and ftp datasets in Dataset Configure CLI if no extension detected by @ewgenius in https://github.com/spiceai/spiceai/pull/1494
• Add llms to the spicepod definition and use throughout by @Jeadie in https://github.com/spiceai/spiceai/pull/1447
• Fix duckdb acceleration converting null into default values. by @y-f-u in https://github.com/spiceai/spiceai/pull/1500
• Separate runtime Dataset from spicepod Dataset by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1503
• Duckdb e2e test OSX support by @y-f-u in https://github.com/spiceai/spiceai/pull/1505
• Use TableReference for dataset name by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1506
• Tweak Results Cache naming and output by @lukekim in https://github.com/spiceai/spiceai/pull/1509
• Fix refresh_sql not properly passing down filters by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1510
• Allow datasets to specify a schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1507
• Cache invalidation for accelerated tables by @sgrebnov in https://github.com/spiceai/spiceai/pull/1498
• Improve spark data connector error messages by @Sevenannn in https://github.com/spiceai/spiceai/pull/1497
• Parse postgres table schema from prepare statement to support empty tables by @ewgenius in https://github.com/spiceai/spiceai/pull/1445
• Improve clarity of README and add FAQ by @lukekim in https://github.com/spiceai/spiceai/pull/1512
• Use binary data transfer for ftp by @gloomweaver in https://github.com/spiceai/spiceai/pull/1517
• Add support for time64 for SQL insertion statement by @y-f-u in https://github.com/spiceai/spiceai/pull/1519
• Add Spice Extensions PoC by @ewgenius in https://github.com/spiceai/spiceai/pull/1476
• Add results cache metrics, pod and quantile filters to Grafana dashboard by @sgrebnov in https://github.com/spiceai/spiceai/pull/1513
• Add unit tests for results caching utils by @sgrebnov in https://github.com/spiceai/spiceai/pull/1514
• Add E2E tests for results caching by @sgrebnov in https://github.com/spiceai/spiceai/pull/1515
• Pass table_reference full string into spark_session table so it can query across schemas or catalogs by @y-f-u in https://github.com/spiceai/spiceai/pull/1521
• Trace on debug level for tables in runtime schema by @ewgenius in https://github.com/spiceai/spiceai/pull/1524
• Update SparkSessionBuilder::remote and update spark fork hash by @Sevenannn in https://github.com/spiceai/spiceai/pull/1495
• Fix federation push-down for datasets with schemas by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1526
• Store history of queries in 'spice.runtime.query_history' by @Jeadie in https://github.com/spiceai/spiceai/pull/1501
• Disable cache for system queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1528
• Register runtime tables with runtime schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1532
• Fix acknowledgments workflow to include all cargo features by @Jeadie in https://github.com/spiceai/spiceai/pull/1531

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.0-alpha...v0.13.1-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

• Twitter: @spice_ai
• Discord: https://discord.gg/kZnTfneP5u
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]