Skip to main content

One post tagged with "json"

JSON data handling and processing

View All Tags

Spice v1.6.1 (Sep 1, 2025)

ยท 3 min read
Jack Eadie
Token Plumber at Spice AI

Announcing the release of Spice v1.6.1! โšก

Spice 1.6.1 is a patch release that provides improved Kafka type inference and JSON flattening support, alongside several bug fixes.

What's New in v1.6.1โ€‹

Improved Kafka Type Inference: Improve Kafka type inference by configuring the number of Kafka messages sampled during schema inference. Increasing the sample size can improve the robustness and reliability of inferred schemas, especially in cases where data contains optional fields or varying structures.

Example spicepod.yml:

dataset:
- from: kafka:orders_events
name: orders
params:
schema_infer_max_records: 100 # Default 1.

For details, see the Kafka Data Connector Documentation.

Improved Kafka JSON Support: Enable nested JSON Kafka messages to be represented in flattened JSON format for the dataset schema.

Example spicepod.yml:

dataset:
- from: kafka:orders_events
name: orders
params:
flatten_json: true # default false

For example, the object:

{
"order_id": "a1f2c3d4-1111-2222-3333-444455556666",
"customer": {
"id": 101,
"name": "Alice",
"premium": true,
"contact": {
"email": "[email protected]",
"phone": "555-1234"
}
},
"discount": 5.0,
"shipped": false
}

With flatten_json: true the result is:

+------------------------+-----------+-------------+
| column_name | data_type | is_nullable |
+------------------------+-----------+-------------+
| order_id | Utf8 | YES |
| customer.id | Int64 | YES |
| customer.name | Utf8 | YES |
| customer.premium | Boolean | YES |
| customer.contact.email | Utf8 | YES |
| customer.contact.phone | Utf8 | YES |
| discount | Float64 | YES |
| shipped | Boolean | YES |
+------------------------+-----------+-------------+

With flatten_json: false or ommitted the result is:

+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| column_name | data_type | is_nullable |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| order_id | Utf8 | YES |
| customer | Struct([Field { name: "id", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "name", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "premium", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "contact", data_type: Struct([Field { name: "email", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "phone", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) | YES |
| discount | Float64 | YES |
| shipped | Boolean | YES |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+

For details, see the Kafka Data Connector Documentation.

Contributorsโ€‹

Breaking Changesโ€‹

No breaking changes.

Cookbook Updatesโ€‹

No new cookbook recipes added in this release.

The Spice Cookbook includes 77 recipes to help you get started with Spice quickly and easily.

Upgradingโ€‹

To upgrade to v1.6.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.6.1 image:

docker pull spiceai/spiceai:1.6.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

๐ŸŽ‰ Spice is now available in the AWS Marketplace!

What's Changedโ€‹

Changelogโ€‹

  • Fix metadata field issue by @Advayp in #6957
  • Update datafusion and datafusion-table-providers crates (#6985) by @Jeadie in #6985
  • Add flatten_json param support for Kafka connector (#6976) by @sgrebnov in #6976
  • Add schema_inference_sample_count param support for Kafka connector (#6969) by @sgrebnov in #6969
  • Add integration test for Kafka connector (#6965) by @sgrebnov in #6965
  • Skip dataset health check for IcebergTableProvider datasets by @phillipleblanc in #6995