One post tagged with "unity catalog"

The v0.15.2-alpha minor release focuses on enhancing stability, performance, and introduces Catalog Providers for streamlined access to Data Catalog tables. Unity Catalog, Databricks Unity Catalog, and the Spice.ai Cloud Platform Catalog are supported in v0.15.2-alpha. The reliability of federated query push-down has also been improved for the MySQL, PostgreSQL, ODBC, S3, Databricks, and Spice.ai Cloud Platform data connectors.

Highlights in v0.15.2-alpha​

Catalog Providers: Catalog Providers streamline access to Data Catalog tables. Initial catalog providers supported are Databricks Unity Catalog, Unity Catalog and Spice.ai Cloud Platform Catalog.

For example, to configure Spice to connect to tpch tables in the Spice.ai Cloud Platform Catalog use the new catalogs: section in the spicepod.yml:

catalogs:
  - name: spiceai
    from: spiceai
    include:
      - tpch.*

sql> show tables
+---------------+--------------+---------------+------------+
| table_catalog | table_schema | table_name    | table_type |
+---------------+--------------+---------------+------------+
| spiceai       | tpch         | region        | BASE TABLE |
| spiceai       | tpch         | part          | BASE TABLE |
| spiceai       | tpch         | customer      | BASE TABLE |
| spiceai       | tpch         | lineitem      | BASE TABLE |
| spiceai       | tpch         | partsupp      | BASE TABLE |
| spiceai       | tpch         | supplier      | BASE TABLE |
| spiceai       | tpch         | nation        | BASE TABLE |
| spiceai       | tpch         | orders        | BASE TABLE |
| spice         | runtime      | query_history | BASE TABLE |
+---------------+--------------+---------------+------------+

Time: 0.001866958 seconds. 9 rows.

ODBC Data Connector Push-Down: The ODBC Data Connector now supports query push-down for joins, improving performance for joined datasets configured with the same odbc_connection_string.

Improved Spicepod Validation Improved spicepod.yml validation has been added, including warnings when loading resources with duplicate names (datasets, views, models, embeddings).

Breaking Changes​

None.

Contributors​

• @phillipleblanc
• @peasee
• @y-f-u
• @ewgenius
• @Sevenannn
• @sgrebnov
• @lukekim

What's Changed​

Dependencies​

• Upgraded Apache DataFusion to v40.0.0.

Commits​

• Update to next release version v0.15.2-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1901
• release: Update helm 0.15.1-alpha by @peasee in https://github.com/spiceai/spiceai/pull/1902
• fix: Detect and error on duplicate component names on spiced (re)load by @peasee in https://github.com/spiceai/spiceai/pull/1905
• fix: flaky test - test_refresh_status_change_to_ready by @y-f-u in https://github.com/spiceai/spiceai/pull/1908
• Add support for parsing catalog from Spicepod. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1903
• Add catalog component to Runtime by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1906
• Adds a RuntimeBuilder and make most items on Runtime private by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1913
• Bump zerovec-derive from 0.10.2 to 0.10.3 by @dependabot in https://github.com/spiceai/spiceai/pull/1914
• Add separate tagged image with enabled models feature by @ewgenius in https://github.com/spiceai/spiceai/pull/1909
• Update datafusion-table-providers to use newest head by @Sevenannn in https://github.com/spiceai/spiceai/pull/1927
• Add MySQL support for TPC-H test data generation script by @sgrebnov in https://github.com/spiceai/spiceai/pull/1932
• fix: Expose ODBC task errors if error is before data stream begins by @peasee in https://github.com/spiceai/spiceai/pull/1924
• Use public.ecr.aws/docker/library/{postgres/mysql}:latest for integration test images by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1934
• Implement spice.ai CatalogProvider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1925
• fix: validate time column and time format when constructing accelerated table refresh by @y-f-u in https://github.com/spiceai/spiceai/pull/1926
• Add support for filtering tables included by a catalog by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1933
• Add UnityCatalog catalog provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1940
• Implement Databricks catalog provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1941
• Copy params into dataset_params by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1947
• Make integration tests more stable by using logged-in registry during CI by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1955
• Add integration test for Spice.ai catalog provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1956
• Add GET /v1/catalogs API and catalogs CMD by @lukekim in https://github.com/spiceai/spiceai/pull/1957
• feat: Enable ODBC JoinPushDown with hashed connection string by @peasee in https://github.com/spiceai/spiceai/pull/1954
• Fix bug: arrow acceleration reports zero results during refresh by @sgrebnov in https://github.com/spiceai/spiceai/pull/1962
• Revert "fix: validate time column and time format when constructing accelerated table refresh" by @y-f-u in https://github.com/spiceai/spiceai/pull/1964
• fix: Update arrow-odbc to use our fork for pending fixes by @peasee in https://github.com/spiceai/spiceai/pull/1965
• Upgrade to DataFusion 40 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1963
• Do exchange shouldn't require table to be writable by @Sevenannn in https://github.com/spiceai/spiceai/pull/1958
• Use custom dialect rule for flight federated request by @y-f-u in https://github.com/spiceai/spiceai/pull/1946
• upgrade datafusion federation to have the table rewrite fix for tpch-q9 by @y-f-u in https://github.com/spiceai/spiceai/pull/1970
• Create v0.15.2-alpha.md Release notes by @digadeesh in https://github.com/spiceai/spiceai/pull/1969
• Fix Unity Catalog API response for Azure Databricks by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1973
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1976

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.15.1-alpha...v0.15.2-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Discord or by email to get involved.

• Twitter: @spice_ai
• Discord: https://discord.gg/kZnTfneP5u
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: [email protected]