Skip to main content
Version: Next

The catalogs section of a Spicepod defines connections to external data catalogs, such as Databricks Unity Catalog or Spice.ai Cloud. Catalogs expose multiple schemas and tables through a single configuration, making it easier to work with large numbers of datasets.

catalogs

Example:

spicepod.yaml

catalogs:
- from: spice.ai
name: spiceai
include:
- 'tpch.*' # Include only the "tpch" tables.

from​

The from field is a string that represents the Uniform Resource Identifier (URI) for the catalog provider. This URI is composed of two parts: a prefix indicating the Catalog Connector to use, and the catalog path within the source.

The syntax for the from field is as follows:

from: <catalog_connector>:<path>

Where:

  • <catalog_connector>: The Catalog Connector to use to connect to the dataset

    Currently supported catalog connectors:

    If the Data Connector is not explicitly specified, it defaults to spiceai.

  • <path>: The path to the catalog within the provider.

ref​

An alternative to adding the catalog definition inline in the spicepod.yaml file. ref can be use to point to a directory with a catalog defined in a catalog.yaml file. For example, a catalog configured in a catalog.yaml in the "catalogs/sample" directory can be referenced with the following:

catalogs/sample/catalog.yaml

from: spice.ai
name: spiceai
include:
- 'tpch.*' # Include only the "tpch" tables.

ref used in spicepod.yaml

version: v1
kind: Spicepod
name: duckdb
catalogs:
- ref: catalogs/sample

name​

The name of the catalog to register in Spice. The schema hierarchy of the external catalog is preserved in Spice. It doesn't need to match the name of the catalog in the external provider.

include​

Optional. The include field is used to specify which tables to include from the catalog. The include field supports glob patterns to match multiple tables. For example, *.my_table_name would include all tables with the name my_table_name in the catalog from any schema. Multiple include patterns are OR'ed together and can be specified to include multiple tables.

exclude​

Optional. The exclude field specifies tables to omit from the catalog, matched against the same table name as include and using the same glob syntax. Multiple exclude patterns are OR'ed together, and exclude takes precedence over include — a table matched by both is omitted. Every Catalog Connector applies it. A common use is to keep tables that cannot be CDC-accelerated out of an accelerated catalog's scope.

access​

Optional. Specifies the access level for the catalog. Supported values are:

  • read (default): Read-only access.
  • read_write: Enables both read and write operations. Only supported for write-capable catalogs.
  • read_write_create: Enables read and write operations and DDL statements such as CREATE TABLE and DROP TABLE. Each catalog's page lists the access levels it supports, such as DuckLake.

A Spicepod reload that lowers a catalog's access withdraws what the new level does not include at once: a change from read_write_create to read_write refuses DDL and keeps writes, and a change to read refuses writes and DDL. Removing the catalog from the Spicepod refuses writes and DDL, and its tables stay readable until Spice restarts. Access that a reload adds takes effect once the catalog has loaded with the new configuration. A reload does not wait for catalogs to load, so a catalog whose source does not answer does not hold back the reload's other changes or later reloads.

In v2.4.3 and earlier, a reload never withdrew write access, so writes reached the catalog until Spice restarted, and a reload that added a catalog whose source did not answer never finished.

params​

Optional. Parameters to pass to the catalog connector for retrieving the metadata on the schemas and tables to be included. The parameters are specific to the connector used.

dataset_params​

Optional. Parameters used when constructing the individual datasets that are registered in Spice from the catalog. The parameters are specific to the connector used.

acceleration​

Optional. Bootstraps and accelerates every table discovered by the catalog (subject to include/exclude), with no per-table configuration. Currently supported for the PostgreSQL catalog connector only.

catalogs:
- from: pg
name: my_pg
acceleration:
engine: cayenne # optional; cayenne is the only supported engine
refresh_mode: changes # required
  • engine: Optional. The accelerator engine used for every table. Defaults to cayenne, currently the only supported value.
  • refresh_mode: Required. The only supported value is changes (CDC); there is no catalog-level full mode.
  • mode: Optional. The storage mode for every table the catalog accelerates, with the same values and meaning as a dataset's acceleration.mode. Defaults to memory, which is not durable: the accelerator starts empty on every restart, and each table runs its initial snapshot from the source again. Use mode: file or mode: file_update, with params.cayenne_file_path, to keep the acceleration across restarts and resume from the replication slot. mode: file_create deletes the acceleration on every startup, so like memory it snapshots each table again.
  • params: Optional. Engine parameters applied to every table the catalog accelerates, with the same meaning as a dataset's acceleration.params. With cayenne_file_path, each table gets its own subdirectory under that path.

Per-table-only acceleration settings (primary_key, on_conflict, indexes, and other per-dataset overrides) are not configurable at the catalog level — they remain on an individual dataset's acceleration block.

An accelerated catalog is included in the runtime's memory budget: the engine, refresh mode, storage mode and params above are the configuration every table it accelerates is built from, and the runtime classifies that same configuration when it derives runtime.query.memory_limit. A Spicepod whose only Cayenne acceleration is a catalog therefore gets the reduced query pool and the compaction reservation, and the projected off-pool reservation counts one table per catalog rather than one per discovered table — see Accelerated catalogs for what that means when sizing.