Skip to main content
Version: Next

Semantic Model

The semantic model is the layer that gives every dataset, view, and column a human-readable description plus structured metadata, and exposes that context uniformly to SQL queries, LLM tools, and language models.

Every description ends up in two places at once:

Defining a Semantic Model​

Semantic data models are defined within the spicepod.yaml file under the datasets section. Each dataset supports description, metadata, and a columns field where individual columns are described with metadata and features for utility and clarity.

Example Configuration​

datasets:
- name: taxi_trips
description: NYC taxi trip rides
metadata:
instructions: Always provide citations with reference URLs.
reference_url_template: https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_<YYYY-MM>.parquet
columns:
- name: tpep_pickup_time
description: 'The time the passenger was picked up by the taxi'
- name: notes
description: 'Optional notes about the trip'
embeddings:
- from: hf_minilm # A defined Spice Model
chunking:
enabled: true
target_chunk_size: 512
overlap_size: 128
trim_whitespace: true

Dataset Metadata​

Datasets can be defined with the following metadata:

  • instructions: Optional. Instructions to provide to a language model when using this dataset.
  • reference_url_template: Optional. A URL template for citation links.

Arbitrary additional keys may be added under metadata: and are passed through to the Arrow schema unchanged, so anything an LLM tool or downstream consumer expects to find there (e.g. governance labels, owning team, source-of-truth links) can be attached without code changes.

For detailed metadata configuration, see the Dataset Reference.

Column Definitions​

Each column can be defined with the following attributes:

  • description: Optional. A description of the column's contents and purpose.
  • type (alias data_type): Optional. Declared column type (Postgres-style or Arrow display form).
  • nullable: Optional. Override column nullability.
  • embeddings: Optional. Vector embeddings configuration for this column.
  • full_text_search: Optional. Full-text-search configuration.
  • metadata: Optional. Arbitrary key/value metadata attached to the column.

For detailed columns configuration, see the Dataset Reference.

Source-Side Comments​

When a dataset is loaded from a source that exposes table or column comments, Spice automatically imports those comments into the Arrow schema metadata under the description key. This means database-native COMMENT ON TABLE and COMMENT ON COLUMN annotations show up alongside Spicepod-defined description values, giving the semantic model the same context that already lives in the source database.

Source-side comments are imported automatically from:

SourceWhat is imported
PostgreSQLTable and column comments via obj_description / col_description in pg_catalog.
MySQLinformation_schema.tables.table_comment and information_schema.columns.column_comment.
Snowflakeinformation_schema.tables.comment and information_schema.columns.comment.
Databricks SQL WarehouseTable and column comments returned by the SQL Warehouse driver's metadata API.
BigQuery (via ADBC catalog)INFORMATION_SCHEMA.TABLE_OPTIONS (description) for the table and INFORMATION_SCHEMA.COLUMN_FIELD_PATHS.description for top-level columns.

Connectors not listed above (object stores, file formats, HTTP, etc.) do not have a comment concept on the source side; for those datasets, attach descriptions in the Spicepod.

Precedence​

When both a Spicepod description and a source-side comment exist for the same table or column, the Spicepod value wins. This lets you override unhelpful or out-of-date source comments without round-tripping a DDL change through the upstream database.

If the Spicepod does not set a description, the source-side comment passes through unchanged.

Reading descriptions from SQL​

Two PostgreSQL-compatible scalar UDFs read the description metadata at query time:

-- Table-level description
SELECT obj_description('public.taxi_trips');

-- Column-level description (by name or 1-based ordinal)
SELECT col_description('public.taxi_trips', 'fare_amount');
SELECT col_description('public.taxi_trips', 3);

Full signatures, including PostgreSQL-shaped (oid, 'pg_class') variants and four-part (catalog, schema, table, column) forms, are documented in obj_description and col_description.

Because these UDFs read the same metadata that source-side comment extraction populates, queries against PostgreSQL or Snowflake datasets that were authored with COMMENT ON … will return those comments without any additional configuration.

How LLMs see the semantic model​

The built-in LLM tools surface descriptions and metadata into the model's context automatically.

Dataset list in the system message​

A model that has the list_datasets tool is told which datasets it can query, with each dataset's Spicepod description, in a system message. When the list is too long for the system message, the model finds datasets with list_datasets instead. See Datasets and Schemas.

list_datasets​

Returns the datasets, views, and catalog tables the runtime can query as JSON, 50 at a time (limit, up to 200), with offset for the next page. search finds the datasets whose names, descriptions, columns, or column descriptions match, best match first. A catalog table matches on its columns only when the runtime has its schema at hand. Each entry has the fully qualified table name, plus the Spicepod description, metadata, and can_search_documents when they apply:

{"datasets": [{"table": "spice.public.taxi_trips", "description": "NYC taxi trip rides"}], "total": 1, "has_more": false}

table_schema​

Lists each table's columns, one per line, with its SQL type and NOT NULL. With output: full (the default), the first line carries the table's description, each column line ends with the column's description, and the table's keys follow the columns. Descriptions come from the Spicepod, falling back to source-side comments when no Spicepod description is set. With output: minimal, only column names, types, and NOT NULL are listed. search lists only the columns whose names or descriptions match.

spice.public.taxi_trips: NYC taxi trip rides
tpep_pickup_time TIMESTAMP -- The time the passenger was picked up
notes VARCHAR -- Optional notes about the trip

In v2.4.x and earlier, list_datasets returned every dataset at once, and table_schema returned a Markdown table.

Because both list_datasets and table_schema are part of the auto and all tool groups, no extra configuration is required — every model declared with tools: auto (or any explicit list that includes table_schema / list_datasets) gets the semantic model in its context.

When the Tool Registry is active, dataset descriptions also feed the hybrid search index used by tool_search to match user questions to relevant dataset-bound tools, so well-described datasets are easier for the model to discover at scale.

End-to-end flow​

spicepod.yaml description source COMMENT ON TABLE / COLUMN
| |
| (Spicepod wins on overlap) |
+---------------+----------------+
v
Arrow schema metadata (`description` key)
|
+---------------+-----------------+
v v v
obj_description table_schema list_datasets
col_description (LLM context) (LLM context)
(SQL)