Amazon S3 Vectors Engine
🎓 Learn how it works with the Amazon S3 Vectors with Spice engineering blog post.
Amazon S3 Vectors, announced in public preview at AWS Summit New York 2025, is a new S3 bucket type designed for storing and querying vector embeddings at scale. It supports billions of vectors with sub-second similarity queries, reducing costs by up to 90% compared to traditional vector databases by separating storage from compute. Spice AI integrates S3 Vectors as a vector index backend, managing embedding indexing, lifecycle, and queries for hybrid search experiences.
To use Amazon S3 Vectors as a Vector Engine, specify s3_vectors
as the engine
, and configure the associated location and AWS credentials.
datasets:
- from: spice.ai:dataset.with.embeddings
name: my_dataset
vectors:
enabled: true
engine: s3_vectors
params:
s3_vectors_bucket: my-s3-vector-bucket
columns:
- name: 'body'
embeddings:
- from: bedrock_titan
embeddings:
- name: bedrock_titan
# ... Define an embedding model to use.
Parameters​
Parameter | Description | Example Value |
---|---|---|
s3_vectors_arn | The S3 vectors index to use. Incompatible with s3_vectors_bucket and s3_vectors_index . | arn:aws:s3vectors:123456654321/bucket/a-bucket/index/index-of-important-embeddings |
s3_vectors_aws_access_key_id | The access key ID for the S3 vectors index | - |
s3_vectors_aws_region | The AWS region for the S3 vectors index. | us-east-1 |
s3_vectors_aws_secret_access_key | The secret access key for the S3 vectors index | - |
s3_vectors_aws_session_token | Session token for the S3 vectors index. | - |
s3_vectors_bucket | The S3 vectors bucket to use. If s3_vectors_index is not specified, an index will be created based on the underlying embedding column. Incompatible with s3_vectors_arn | a-bucket |
s3_vectors_index | The name of the s3 vectors index to use or create. Incompatible with s3_vectors_arn . | index-of-important-embeddings |
s3_vectors_index
ands3_vectors_arn
specify a single index for the dataset and therefore should not be used with a dataset containing more than one embedding column.- S3 Vectors uses approximate nearest neighbor (ANN) algorithms for performance, providing probabilistically closest results.
Overview​
Amazon S3 Vectors exposes two core operations: upsert vectors (assign a vector to a key with optional metadata) and vector similarity queries (find closest vectors by metrics like cosine or Euclidean distance). It stores vectors durably in S3 and executes queries on transient compute, avoiding always-on servers. This enables petabyte-scale storage at low cost, ideal for semantic search, recommendations, and RAG in AI applications.
Configuration​
Annotate dataset schemas to specify columns for embedding and models. Spice supports local or hosted models like Amazon Titan Embeddings. When data ingests, Spice generates embeddings and stores them in the configured S3 Vectors index. Spice handles index creation, updates, and synchronization with data changes.
Example with AWS Titan model:
datasets:
- from: oracle:"CUSTOMER_REVIEWS"
name: reviews
vectors:
enabled: true
engine: s3_vectors
params:
s3_vectors_bucket: my-s3-vector-bucket
columns:
- name: body
embeddings:
from: bedrock_titan
embeddings:
- from: bedrock:amazon.titan-embed-text-v2:0
name: bedrock_titan
params:
aws_region: us-east-2
dimensions: '256'
Querying​
Perform vector searches via HTTP API or SQL table-valued function vector_search(dataset, query)
.
HTTP example:
curl -X POST http://localhost:8090/v1/search \
-H "Content-Type: application/json" \
-d '{
"datasets": ["reviews"],
"text": "issues with same day shipping",
"additional_columns": ["rating", "customer_id"],
"where": "created_at >= now() - INTERVAL '7 days'",
"limit": 2
}'
SQL example:
SELECT review_id, rating, customer_id, body, score
FROM vector_search(reviews, 'issues with same day shipping')
WHERE created_at >= to_unixtime(now() - INTERVAL '7 days')
ORDER BY score DESC
LIMIT 2;
Results include matching snippets, additional fields, primary keys, scores, and dataset names.
Managing Embeddings​
Spice manages the vector lifecycle: embedding data on ingestion, upserting to S3 Vectors with primary keys as identifiers, and handling updates/deletions. Vectors can be pre-stored in S3 for scalability, avoiding memory limits in accelerators or slow just-in-time computations.
Query Execution​
Spice pushes similarity computations to S3 Vectors, retrieving top matches as (id, score) pairs. These form a temporary table joinable with the dataset for full records, reducing processing to candidates only.
Handling Filters​
Mark columns as filterable metadata to push filters into S3 Vectors queries:
columns:
- name: created_at
metadata:
vectors: filterable
This ensures results respect constraints like time ranges during similarity search.
Optimizations​
Store non-filterable columns as metadata to avoid joins:
columns:
- name: rating
metadata:
vectors: non-filterable
Queries can then retrieve all needed data directly from the index, improving latency for read-heavy workloads.
Advanced Features​
Spice supports hybrid search (vector + full-text via BM25, merged with RRF), multi-vector queries (weighting columns), and re-ranking (e.g., keyword first-pass then vector on candidates). Compose via SQL CTEs and joins.
Hybrid RRF example:
WITH
vector_results AS (
SELECT review_id, RANK() OVER (ORDER BY score DESC) AS vector_rank
FROM vector_search(reviews, 'issues with same day shipping')
),
text_results AS (
SELECT review_id, RANK() OVER (ORDER BY score DESC) AS text_rank
FROM text_search(reviews, 'issues with same day shipping')
)
SELECT
COALESCE(v.review_id, t.review_id) AS review_id,
(1.0 / (60 + COALESCE(v.vector_rank, 1000)) +
1.0 / (60 + COALESCE(t.text_rank, 1000))) AS fused_score
FROM vector_results v
FULL OUTER JOIN text_results t ON v.review_id = t.review_id
ORDER BY fused_score DESC
LIMIT 50;
Multi-column example (weighting title higher):
WITH
body_results AS (
SELECT review_id, score AS body_score
FROM vector_search(reviews, 'issues with same day shipping', col => 'body')
),
title_results AS (
SELECT review_id, score AS title_score
FROM vector_search(reviews, 'issues with same day shipping', col => 'title')
)
SELECT
COALESCE(body.review_id, title.review_id) AS review_id,
COALESCE(body_score, 0) + 2.0 * COALESCE(title_score, 0) AS combined_score
FROM body_results
FULL OUTER JOIN title_results ON body_results.review_id = title_results.review_id
ORDER BY combined_score DESC
LIMIT 5;
Cookbook​
- A cookbook recipe to configure a dataset with an S3 vectors engine in Spice. S3 Vectors engine