Large Language Models

Spice provides a high-performance, OpenAI API-compatible AI Gateway optimized for managing and scaling large language models (LLMs). It offers tools for Enterprise Retrieval-Augmented Generation (RAG), such as SQL query across federated datasets and an advanced search feature (see Search).

Spice supports full OpenTelemetry observability, helping with detailed tracking of model tool use, recursion, data flows and requests for full transparency and easier debugging.

Configuring Language Models

Spice supports a variety of LLMs (see Model Providers).

Core Features

• Custom Tools: Provide models with tools to interact with the Spice runtime. See Tools.
• System Prompts: Customize system prompts and override defaults for v1/chat/completion. See Parameter Overrides.
• Memory: Provide LLMs with memory persistence tools to store and retrieve information across conversations. See Memory.
• Vector Search: Perform advanced vector-based searches using embeddings. See Vector Search.
• Evals: Evaluate, track, compare, and improve language model performance for specific tasks. See Evals.
• Local Models: Load and serve models locally from various sources, including local filesystems and Hugging Face. See Local Models.

For API usage, refer to the API Documentation.

📄️ Tools

Learn how LLMs interact with the Spice runtime.

📄️ Memory

Learn how to provide LLMs with memory

📄️ Evals

Learn how Spice evaluates, tracks, compares, and improves language model performance for specific tasks

📄️ Parameter Overrides

Learn how to override default LLM hyperparameters in Spice.

📄️ Local Models

Learn how to load and serve large learning models.