Version: v1.10

Large Language Models

Spice provides a high-performance, OpenAI API-compatible AI Gateway optimized for managing and scaling large language models (LLMs). It offers tools for Enterprise Retrieval-Augmented Generation (RAG), such as SQL query across federated datasets and an advanced search feature (see [Search(../search)).

Spice supports full OpenTelemetry observability, helping with detailed tracking of model tool use, recursion, data flows and requests for full transparency and easier debugging.

Configuring Language Models

Spice supports a variety of LLMs (see [Model Providers(../../components/models/index.md)).

Core Features

SQL Integration: Invoke LLMs directly within SQL queries using the ai() function for text generation tasks. See [SQL Reference: ai function(../../reference/sql/scalar_functions#ai).
Custom Tools: Provide models with tools to interact with the Spice runtime. See [Tools(../large-language-models/tools).
System Prompts: Customize system prompts and override defaults for [v1/chat/completion(../../api/HTTP/post-chat-completions). See [Parameter Overrides(../large-language-models/parameter_overrides.md).
Memory: Provide LLMs with memory persistence tools to store and retrieve information across conversations. See [Memory(../large-language-models/memory).
Vector Search: Perform advanced vector-based searches using embeddings. See [Vector Search(../search/vector-search).
Evals: Evaluate, track, compare, and improve language model performance for specific tasks. See [Evals(../large-language-models/evals).
Local Models: Load and serve models locally from various sources, including local filesystems and Hugging Face. See [Local Models(../large-language-models/serving).