One post tagged with "flightsql"

Announcing the release of Spice v1.11.1! 🛠️

v1.11.1 is a patch release improving Spice Cayenne accelerator reliability and performance, enhancing DynamoDB Streams and HTTP data connectors, and fixing issues in Federated Task History and FlightSQL.

What's New in v1.11.1​

Spice Cayenne Accelerator Improvements​

This release includes stability and performance fixes for the Spice Cayenne accelerator:

• Row-based Deletion Logic: Refactored row-based delete operations to use per-file deletion vectors with RoaringBitmap. Deletion scans now use Vortex-native streaming with filter pushdown and project only row indices, achieving zero data I/O for delete operations.
• Constraints & On Conflict: constraints and on_conflict configurations are now automatically inferred from federated table metadata, enabling datasets like DynamoDB to work without explicitly defining primary_key in the Spicepod.
• Partitioned Table Deletion: Fixed an issue where DELETE operations on partitioned Cayenne tables failed.
• Data Integrity: Fixed two issues with acceleration snapshot handling: protected snapshots are now included in conflict detection keyset scans (preventing duplicate key creation during append refresh), and snapshot cleanup no longer deletes protected snapshots.

Data Connector Improvements​

• DynamoDB Streams: Added automatic re-bootstrapping when the stream lag exceeds DynamoDB shard retention (24h). Configurable via the new lag_exceeds_shard_retention_behavior parameter with values error (default), ready_before_load, or ready_after_load.
• HTTP Connector: HTTP responses now include a response_status column (UInt16). 4xx responses (e.g., 404 Not Found) are treated as valid queryable data and cached normally. 5xx responses are retried with backoff, returned to the user, but excluded from the cache to prevent transient server errors from polluting cached results.

Other Improvements​

• Reliability: Added retries for SnapshotManager operations and general snapshot reliability improvements.
• Reliability: Fixed handling of timestamp precision mismatches in query result caching.
• Reliability: Fixed a double projection issue in federated task history queries that caused Schema error: project index out of bounds errors in cluster mode.
• Developer Experience: Added cookie middleware support to the FlightSQL data connector.

Contributors​

• @krinart
• @phillipleblanc
• @sgrebnov

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates. The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.11.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.11.1 image:

docker pull spiceai/spiceai:1.11.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.11.1

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Cayenne: row-based delete logic improvements by @sgrebnov in #9237
• Proper support for constraints/on_conflict in Cayenne Acceleration by @krinart in #9335
• Retries for SnapshotManager by @krinart in #9334
• fix(cayenne): Include protected snapshots in conflict detection keyset scan by @sgrebnov in #9176
• fix(cayenne): Fix data loss by preserving protected snapshots during cleanup by @sgrebnov in #9182
• Simplify retention filter expressions before pushdown by @sgrebnov in #9244
• Fix test_retention_complex_sql by @sgrebnov in #9270
• runtime: avoid double projection in federated task history by @phillipleblanc in #9326
• feat(http): Return all HTTP responses as data, skip caching 5xx by @sgrebnov in #9313
• Snapshots Improvements by @krinart in #9318
• fix(caching): Handle timestamp precision mismatch and add more tests by @sgrebnov in #9315
• DynamoDB Streams Table Rebootstrapping by @krinart in #9305
• Fix Cayenne partitioned table deletion support by @sgrebnov in #9267
• FlightSQL: add cookie middleware support by @phillipleblanc in #9282
• Apply SchemaCastScanExec before applying changes in process_upsert_batch by @krinart in #9297