One post tagged with "parquet"

Announcing the release of Spice v1.0.5 🧊

Spice v1.0.5 expands Iceberg support with the introduction of the Iceberg Data Connector, in addition to the existing Iceberg Catalog Connector. This new connector enables direct dataset creation and configuration for specific Iceberg objects, enabling federated and accelerated SQL queries on Apache Iceberg tables.

Performance improvements include object-store optimized Parquet pruning in append mode, where object-store metadata is now leveraged alongside Hive partitioning to optimize file pruning. This results in faster and more efficient queries.

DuckDB has been upgraded to v1.2.0, along with additional stability improvements, including improved graceful shutdown and the ability to configure the DuckDB memory limit.

Additional updates include support for the Arrow Map type.

Highlights in v1.0.5​

• New Iceberg Data Connector: Enables direct dataset creation and querying of Iceberg tables.

  Example usage in spicepod.yaml:

  datasets:
    - from: iceberg:https://iceberg-catalog-host.com/v1/namespaces/my_namespace/tables/my_table
      name: my_table
      params:
        # Same as Iceberg Catalog Connector
      acceleration:
        enabled: true
  

  For detailed setup instructions, authentication options, and configuration parameters, refer to the Iceberg Data Connector documentation.

• Improved Parquet pruning in append mode: Uses object-store metadata for more efficient file pruning.

• DuckDB upgrade to v1.2.0 with improved graceful shutdown: Read the DuckDB v1.2.0 announcement for details, including breaking changes for map and list_reduce. Graceful shutdown of DuckDB has been improved for better stability across restarts.

• Configurable DuckDB memory limit: Use the duckdb_memory_limit parameter to set the DuckDB acceleration memory limit:

  - from: spice.ai:path.to.my_dataset
     name: my_dataset
     acceleration:
       params: 
         duckdb_memory_limit: '2GB'
       enabled: true
       engine: duckdb
       mode: file
  

Contributors​

• @peasee
• @phillipleblanc
• @sgrebnov
• @lukekim

Breaking Changes​

• DuckDB v1.2.0 has breaking changes.

Upgrading​

To upgrade to v1.0.5, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.0.5 image:

docker pull spiceai/spiceai:1.0.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

• duckdb-rs: Upgraded from 1.1.1 to 1.2.0

Changelog​

• fix: Update OpenAI model health check by @peasee in #4849
• fix: Allow metrics endpoint setting in CLI by @peasee in #4939
• DuckDB acceleration: fix Decimal with zero scale support by @sgrebnov in #4922
• Introduce runtime shutdown state by @sgrebnov in #4917
• Add support for Flight and HTTP endpoints configuration to Spice CLI (run and sql) by @sgrebnov and @lukekim in #4913
• Fix Datafusion resources deallocation during shutdown by @sgrebnov in #4912
• DuckDB: fix error handling during record batch insertion by @sgrebnov in #4894
• DuckDB: add support for Map Arrow type for DuckDB acceleration by @sgrebnov in #4887
• Upgrade to DuckDB v1.2.0 by @sgrebnov in #4842
• Gracefully shutdown the runtime and deallocate static resources by @sgrebnov in #4879
• Implement an Iceberg Data Connector by @phillipleblanc in #4941
• Don't trace canceled dataset refresh during runtime termination by @sgrebnov in #4958
• Use metadata column last_modified when specified as a time_column by @phillipleblanc in #4970
• Add duckdb_memory_limit param support for DuckDB acceleration by @sgrebnov in #4971
• Add Iceberg dataset integration test by @phillipleblanc in #4950

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.0.4...v1.0.5