0.83.0

What's Changed

• break: new layout tree vtable by @joseph-isaacs in #8910
• Introduce array_slots macro for type-safe array slot access by @joseph-isaacs in #8912
• fix(cuda): synchronize Arrow exports with buffer streams by @0ax1 in #8952

Full Changelog: 0.82.0...0.83.0

0.82.0

What's Changed

• Gate vortex benchmark under label by @myrrc in #8911
• New C++ API examples by @myrrc in #8755
• Increse duckdb retries when operating on parquet s3 by @robert3005 in #8903
• Bypass unknown aggregates in ZonedLayout by @connortsui20 in #8904
• BufferIterator can be Send + Sync by @robert3005 in #8873
• duckdb: forbid reading files with duplicate struct field names by @myrrc in #8917
• Decimal Add/Sub kernels by @mhk197 in #8724
• duckdb: fix extract path with multiple versions by @myrrc in #8926
• Read the data child directly when the zone map is empty by @connortsui20 in #8905
• Fix Kleene boolean dictionary pushdown by @connortsui20 in #8929
• perf(cuda): skip patch-free bitpacked synchronization by @0ax1 in #8925
• feat(cuda): expose row block and scan batch size controls by @0ax1 in #8923
• Exclude Markdown from ruff format by @robert3005 in #8940
• Add UnionArray compute functions by @connortsui20 in #8884
• perf(array): hoist decimal same-scale cast plan by @ClSlaid in #8932
• perf(cuda): decode FSST strings with GPU-built Arrow offsets by @0ax1 in #8921
• duckdb: pushdown expressions with LIMIT by @myrrc in #8943
• feat[compressor]: require schemes to declare their produced encodings by @joseph-isaacs in #8944
• Expose LayoutStrategy::buffered_bytes through the jni boundary by @robert3005 in #8909
• vortex-jni includes vortex-geo, jni arrow import/export uses session methods by @robert3005 in #8914
• feat[cuda]: add direct I/O for local file scans by @onursatici in #8898

New Contributors

• @ClSlaid made their first contribution in #8932

Full Changelog: 0.81.0...0.82.0

0.81.0

What's Changed

• Add file-level user metadata by @mprammer in #8740
• refactor[array]: serialization context filter by ids (not whole registry) by @joseph-isaacs in #8897
• StructReader::partition_expr doesn't hold lock if the expression is being newly partitioned by @robert3005 in #8892
• Add UnionArray by @connortsui20 in #8834
• Don't emit STB_UNIQUE symbols in GCC for vortex-duckdb by @myrrc in #8913

Full Changelog: 0.80.0...0.81.0

0.80.0

What's Changed

• Assortment of VortexSource improvements and fixes by @AdamGS in #8718
• fix(spark): reject MapType instead of emitting a childless Arrow List by @jackylee-ch in #8811
• build(java): bump Spark 3 from 3.5.8 to 3.5.9 by @jackylee-ch in #8812
• build(java): bump Spark 4 from 4.1.1 to 4.1.2 by @jackylee-ch in #8813
• Skip building binaries in when there are no tests by @AdamGS in #8820
• LayoutEncodingAdapter::build doesn't need to panic, it can error by @robert3005 in #8824
• Fix nested scalar hashing by @connortsui20 in #8804
• Add documentation links to benchmark suite PR comments by @joseph-isaacs in #8815
• feat(vortex-onpair): update to the onpair 0.1 API by @gargiulofrancesco in #8675
• feat(vortex-array): union_child_validities expr helper by @HarukiMoriarty in #8829
• Add PiecewiseSequence index array by @danking in #8799
• Avoid redundant allocations in struct layout reader by @joseph-isaacs in #8851
• Validate PostscriptFragment offsets when parsing the footer by @robert3005 in #8818
• docs: introduce Vortex editions for users of the file format by @joseph-isaacs in #8855
• Use PiecewiseSequence indices for FSL take by @danking in #8800
• feat(vortex-geo): null-propagating scalar functions by @HarukiMoriarty in #8803
• feat: exploit PiecewiseSequence for faster take by @danking in #8802
• Fix remaining flaky CodSpeed microbenchmarks by @connortsui20 in #8861
• Remove unused ArrayRef::is_arrow by @connortsui20 in #8868
• Specialize VarBin PiecewiseSequence takes by @danking in #8836
• Validate untrusted footer metadata on open instead of panicking by @connortsui20 in #8867
• feat(onpair): promote OnPair to a stable read-only encoding by @gargiulofrancesco in #8872
• break[array]: Remove unimplemented child() method from VTable implementations by @joseph-isaacs in #8870
• Fix CUDA ListView dictionary export regression test by @onursatici in #8876
• Route CUDA pruning io through host memory by @onursatici in #8862
• Remove mutable state from Vortex open options by @onursatici in #8879
• bool array slice to support device buffers by @onursatici in #8875
• Use cursor copies for PiecewiseSequence slices by @danking in #8832
• feat[edition]: session-registered editions defitions by @joseph-isaacs in #8871
• Lazily untranspose DeltaArray validity and depend on fastlanes bit transpose code by @robert3005 in #8725
• Update dependency pyarrow to v23 [SECURITY] by @renovate[bot] in #8795
• Expose FFI runtime worker pool control by @onursatici in #8789
• Add UnionValue and UnionScalar by @connortsui20 in #8838
• Use PiecewiseSequence for List take elements by @danking in #8833
• Add PiecewiseSequence take microbenchmarks by @danking in #8830
• CONTRIBUTING.md: Update committers and maintainers by @brancz in #8880
• Prevent non empty unions and clarify null docs by @connortsui20 in #8882
• Revert "Lazily untranspose DeltaArray validity and depend on fastlanes bit transpose code" by @joseph-isaacs in #8883
• Masking List-typed array preserves zctl by @mhk197 in #8889
• Add take benchmarks for FSL(Chunked) and Chunked(FSL) arrays by @robert3005 in #8887
• feat(vortex-geo): per-row bounding-box scalar function (vortex.geo.envelope) by @HarukiMoriarty in #8831
• non_nan_check doesn't try to cast arbitrary scalars to primitive by @robert3005 in #8891
• feat[layout]: extract a LayoutStrategyEncodingValidator from the FlatLayoutStrategy by @joseph-isaacs in #8885
• duckdb 1.5.5 by @myrrc in #8895

New Contributors

• @gargiulofrancesco made their first contribution in #8675

Full Changelog: 0.79.0...0.80.0

0.79.0

What's Changed

• Remove arrow usage from VarBin::compare, VarBinView::filter and BoolArray::from_iter by @robert3005 in #8695
• Add list_sum expression by @mhk197 in #8676
• cuda: build.rs add trigger by @mag1c1an1 in #8708
• vortex-file: Spawn tasks via runtime instead of directly via tokio by @brancz in #8711
• Add UnionVariants type and embed it in DType::Union by @connortsui20 in #8703
• Add golden-corpus determinism tests for the default compressor by @connortsui20 in #8697
• Ignore quick-xml rustsec since we cannot update it ourselves by @robert3005 in #8712
• Ignore one more quick-xml rustsec advisory by @robert3005 in #8716
• Remove stored Union Nullability by @connortsui20 in #8714
• Optimize dependencies using cargo-shear by @AdamGS in #8715
• Implement LIKE in vortex instead of falling back to arrow by @robert3005 in #8709
• Update actions/setup-java digest to 0f481fc by @renovate[bot] in #8729
• Update actions/setup-python digest to ece7cb0 by @renovate[bot] in #8730
• Update actions/stale digest to 1e223db by @renovate[bot] in #8731
• Benchmarks for fixed-width array filtering by @robert3005 in #8727
• Add docs on how to use Spark vortex datasource by @robert3005 in #8728
• Fix integer overflow in BitBufferMut::truncate and append_n by @ksj1230 in #8474
• Reduce Java and vortex-jni build sizes by @AdamGS in #8734
• Update dependency ray to v2.56.0 [SECURITY] by @renovate[bot] in #8732
• fix(layout/dict): probe with the configured compressor instead of a hardcoded default by @tomsanbear in #8406
• remove old c++ api by @myrrc in #8736
• Split arrow dependency from vortex-array by @robert3005 in #8717
• New C++ api docs by @myrrc in #8737
• vx_expression_clone, remove vx_file_write_array by @myrrc in #8735
• ST_Intersects pushdown: vortex.geo.intersects kernel, DuckDB lowering, spatial function overrides by @HarukiMoriarty in #8704
• Reject negative Union type IDs by @connortsui20 in #8738
• feat(vortex-geo): add native geoarrow.box bounding-box type by @HarukiMoriarty in #8746
• Fix flaky CodSpeed microbenchmarks by @connortsui20 in #8742
• Reorganize the compressor by @connortsui20 in #8745
• Update dependency setuptools to v83 [SECURITY] by @renovate[bot] in #8752
• New C++ API by @myrrc in #8741
• fix(onpair): pin OnPair dictionary-training seed for determinism by @claude[bot] in #8756
• Better error message for unsupported Arrow vector type in Spark reader by @jackylee-ch in #8759
• duckdb 1.5.4 by @myrrc in #8761
• forbid todo!/unimplemented! in vortex-duckdb by @myrrc in #8762
• fix(spark): warn that overwrite destructively deletes existing files by @jackylee-ch in #8767
• test(spark): characterize ArrowUtils Arrow-to-Spark type mapping by @jackylee-ch in #8770
• Support reporting statistics and currently written byte size from writers in jni by @robert3005 in #8726
• feat(vortex-geo): geometry Bbox zone-map statistic + distance-filter pruning by @HarukiMoriarty in #8646
• VortexArrowColumnVector supports conversion from all arrow types that Spark can support by @robert3005 in #8771
• feat(vortex-geo): prune ST_Intersects filters via the AABB zone statistics by @HarukiMoriarty in #8775
• fix(cuda): preserve dictionary value nulls by @0ax1 in #8774
• ci: cut wasted minutes from measurement_id and WASM jobs by @a10y in #8776
• ListArray::reset_offsets doesn't need to revalidate array after reset by @robert3005 in #8781
• New C++ API tests by @myrrc in #8751
• test(spark): characterize PartitionPathUtils partition discovery and materialization by @jackylee-ch in #8783
• Own SQL benchmark suite by @myrrc in #8719
• test(spark): characterize SparkToArrowSchema Spark-to-Arrow conversion by @jackylee-ch in #8782
• Expose CUDA-compatible file sink through FFI by @onursatici in #8784
• Expose pooled CUDA file scans through FFI by @onursatici in #8788
• feat(jni): bridge caller-provided Java I/O into VortexReadAt/VortexWrite by @robert3005 in #8748
• feat(cuda): export FSST directly to Arrow varbin by @0ax1 in #8787
• feat[buffer]: specialized collect_bool per arch by @joseph-isaacs in #8749
• feat: arch intrinsic feature discovery pattern by @joseph-isaacs in #8750
• Use u8 Union type IDs by @connortsui20 in #8790
• Add outer nullability to Union by @connortsui20 in #8798
• ListLayout baseline by @mhk197 in #8706
• Remove the old v2 benchmarks website by @connortsui20 in #8805
• apply alp patches in parallel in cuda by @onursatici in #8793

New Contributors

• @mag1c1an1 made their first contribution in #8708
• @ksj1230 made their first contribution in #8474
• @claude[bot] made their first contribution in #8756
• @jackylee-ch made their first contribution in #8759

Full Changelog: 0.78.0...0.79.0

0.78.0

Changes

✨ Features

• Implement compare in vortex instead of falling back to arrow (#8661) @robert3005

🚀 Performance

• Ungrouped aggregate function pushdown for duckdb (#8645) @myrrc

🐛 Bug Fixes

• fix: accept FixedSizeList dtype when deserializing ListValue scalars (#8696) @brancz
• Audit panic handling in detached tasks (#8677) @AdamGS

🧰 Maintenance

• Split TableStrategy dispatcher from StructStrategy (#8638) @mhk197

0.77.0

Changes

NOTE: 0.76.0 was yanked, because of a (hopefully very rare) latent correctness bug, that became much easier to hit in 0.76.0. There was a long-standing incorrect behavior in the TableStrategy writer, wherein it would not correctly propagate panics from the parallel writer subtasks. The fix was in #8672.

If you passed a stream of well-formed valid arrays straight to the builtin vortex writers, you are definitely unaffected.

If any of the following are true, you may have files that were silently truncated, in part (all versions) or in whole (0.76.0):

• you had a stream of well-formed arrays, AND called map on that ArrayStream with a function that panicked on invocation, then passed the resulting stream to a builtin vortex writer
• you passed a stream of malformed arrays that could have triggered a panic or assertion failure (e.g., you constructed invalid vortex arrays using unsafe/new_unchecked; to our knowledge you can't hit this path with the safe/regular constructors)

If you believe you may have been affected by this issue, please reach out in slack: https://vortex.dev/slack

⚠️ Breaks

• FFI: replace vx_string/vx_binary with vx_view (#8668) @myrrc
• Replace ToCanonical trait usages with execute (#8609) @robert3005
• Remove borrowed types from FFI; fixed_size_list constructor accepts u32 (#8658) @myrrc
• Remove variants module (#8643) @robert3005
• Disallow the legacy global session and allowlist existing call sites (#8637) @robert3005

✨ Features

• feat(geo): add native LineString, MultiPoint, MultiLineString types (#8679) @HarukiMoriarty
• Cast pushdown for duckdb (#8620) @myrrc
• mean() for decimals (#8649) @myrrc
• FFI: canonicalize array, data pointer for primitives (#8665) @myrrc
• VortexSession register is public and allow_unknown takes reference (#8647) @robert3005
• feat: support ST_DWithin pushdown in vortex (#8625) @HarukiMoriarty
• feat(vortex-bench): support vortex native geo types into SpatialBench, and wire into benchmark orchestrator (#8623) @HarukiMoriarty
• feat(vortex-geo): add MultiPolygon extension type + WKB export for native geometry (#8608) @HarukiMoriarty
• feat(vortex-bench): wire SpatialBench into the bench orchestrator (#8607) @HarukiMoriarty

🚀 Performance

• Optimize bit iteration and validity checking with SIMD popcount (#8636) @robert3005
• perf: branchless mask-select for listview zip (#8264) @joseph-isaacs

🐛 Bug Fixes

• fix: TableStrategy currently hides panic in spawned column writer tasks. (#8672) @AdamGS
• Grow list builders as we append elements to it (#8673) @robert3005
• Fix vortex-jni local path handling, centralise logic in vortex-file (#8650) @robert3005

🧰 Maintenance

13 changes

• Consolidate benchmark SQL queries into vortex-bench/sqldirectory (#8684) @connortsui20
• Direct issues to templates instead of GitHub Discussions (#8685) @connortsui20
• Better error message in case of unknown encoding id (#8681) @robert3005
• Hardcode constant detection into the cascading compressor (#8667) @connortsui20
• Add policy-bot configuration file (#8659) @robert3005
• Lock file maintenance (#8654) @renovate[bot]
• Add bit iteration benchmarks (#8656) @robert3005
• Lock file maintenance (#8655) @renovate[bot]
• Update Rust crate cxx to v1.0.195 [SECURITY] (#8653) @renovate[bot]
• bench: warm ArrayKernels before the bench loop (#8479) @joseph-isaacs
• Remove execute_boolean fallback to arrow (#8644) @robert3005
• Decouple optional features from pulling vortex-file features for top crate (#8642) @AdamGS
• feat: add CUDA cuDF convenience API (#8624) @0ax1

0.76.0

Changes

⚠️ Breaks

• Remove ArrayAccessor (#8603) @robert3005
• Fix bounded max partial state (#8515) @gatesn
• Use aggregate descriptors for zoned stats (#7938) @gatesn
• Remove duckdbfs (#8468) @myrrc
• Move execute-parent kernels into session registry (#8482) @gatesn
• break: add Validity::definitely_all_null() method (#8447) @joseph-isaacs
• Make Mask execution strict about nullable booleans (#8121) @joseph-isaacs
• Support reporting statistics in spark datasource (#8057) @robert3005

✨ Features

• Add swimlane view to vortex-web that visualizes physical layout of columns in files (#8634) @robert3005
• feat(vortex-bench): infra of SpatialBench on DuckDB, plain binary (#8598) @HarukiMoriarty
• Expose array buffer replacement (#8606) @gatesn
• bool -> Primitive casts (#8621) @myrrc
• feat: add metadata bridge for Python CUDA export (#8604) @0ax1
• Add list_length scalar function (#8495) @mhk197
• feat(file): make Footer::new public (#8602) @tobias-fire
• feat(array): push struct validity into children (#8589) @miniex
• Support session-time source configuration in DataFusion, and clear up precedence between config sources (#8575) @AdamGS
• feat(python): package CUDA as an optional extension (#8510) @0ax1
• Decimal casting (#8562) @gatesn
• Add JSON to Parquet Variant conversion with shredding (#8391) @AdamGS
• Add extension storage scalar function (#8540) @gatesn
• Numerical aggregate functions have an option to skip or include nans in calculation, skip by default (#8457) @robert3005
• feat(vortex-geo): ST_Distance via the geo crate + Polygon type (#8497) @HarukiMoriarty
• feat[gpu]: arrow device array stream support (#8483) @0ax1
• feat(vortex-datafusion): struct scalar conversion + extension-over-struct scan (#8453) @HarukiMoriarty
• feat: DeltaScheme adjustable compression ratio threshold (#8461) @joseph-isaacs
• Add Delta scheme to the integer compressor (unstable_encodings) (#8212) @joseph-isaacs
• Push down some expressions to Dict layout reader's cached values (#8341) @myrrc
• --patterns flag for random-access-bench (#8446) @myrrc
• ffi: create data source from memory buffer (#8426) @myrrc
• Constant comparison and byte_length OnPair kernels (#8371) @myrrc
• REGEX support for sqllogictests (#8392) @myrrc
• feat(vortex-geo): Arrow import/export for the native Point type (#8374) @HarukiMoriarty

🚀 Performance

• perf(datafusion): push down list_length expression (#8600) @mhk197
• perf: cache static identifiers to avoid per-call interner locks (#8614) @miniex
• perf(duckdb): push down list length expressions (#8544) @mhk197
• Spawn arrow conversion in jni bindings (#8595) @robert3005
• Compute zone layout stats concurrently when writing (#8594) @robert3005
• Chunked dict values pullup (#8577) @gatesn
• Tune varbinview compaction for Arrow export (#8585) @gatesn
• Report DuckDB max cardinality for exact scans (#8582) @gatesn
• perf(scan): intra-file decode parallelism — sub-split large chunk spans (#8400) @lukekim
• perf(alp): avoid per-combination allocations when searching exponents (#8565) @miniex
• Optimize interleave boolean gather (#8350) @joseph-isaacs
• RLEExporter for duckdb (#8539) @myrrc
• ReusableDict for RunEnd duckdb exporter (#8538) @myrrc
• perf: align buffer to Alignment::DEFAULT_ALIGNMENT = Alignment::new(256) (#8490) @joseph-isaacs
• DuckDB chunk exporter (#8520) @gatesn
• Faster take for runend array (#8228) @robert3005
• Inline Kleene boolean kernels (#8478) @gatesn
• Scalar functions pushdown to Duckdb (#8303) @myrrc
• perf: don't use fill_null for executing an array into a mask (#8466) @joseph-isaacs
• Optimize buffer ops (#8322) @AdamGS
• perf[gpu]: export arrow device validity on the gpu (#8440) @0ax1
• perf[buffer]: iteration for fallible operations with validity (#8120) @joseph-isaacs
• Faster str interning (#8459) @AdamGS
• perf[gpu]: generate FixedSizeList offsets on device (#8458) @0ax1
• Arithmetic Operations (#8398) @gatesn
• perf(buffer): read byte-aligned bit operands directly as words (#8436) @miniex
• perf: non-blocking Arrow Device array release (#8390) @0ax1

🐛 Bug Fixes

21 changes

• Convert extension type scalars to Value correctly (#8635) @myrrc
• fix(vortex-bench): map gs:// scheme to gcs storage label (#8630) @brancz
• fix: CUDA bitpacked sliced output allocation (#8622) @0ax1
• Fix wasm32 build by gating MultiFileSession on non-wasm targets (#8612) @robert3005
• fix: support 64K alignment (#8599) @a10y
• Guard calling is_nan on scalar value on dtype being a float (#8593) @robert3005
• vortex-datafusion: Pipe session through to converter (#8591) @brancz
• Save children dtypes/length in ZstdBuffersMetadata (#8572) @myrrc
• fix(array): don't panic on unsupported arrow types (#8564) @miniex
• fix(layout): don't panic collecting an empty stream (#8472) @miniex
• Correctly calculate FSST compressed output size (#8551) @robert3005
• refactor[fsst]: take ArrayRef in fsst_compress, decide offset width upfront (#7900) @mprammer
• Move optimizer kernels into a dedicated KernelSession (#8511) @gatesn
• Fix is_zero() for FixedSizeList and Struct scalar types (#8500) @connortsui20
• Handle null lanes in primitive take (#8505) @gatesn
• Preserve dict referenced values metadata (#8499) @gatesn
• fix: preserve operand width in DecimalValue checked arithmetic (#7380) @abnobdoss
• fix Fuzz constant OOM for CUDA (#8481) @a10y
• vortex-tui: query tab should work for non-*.vortex files (#8473) @a10y
• Ignore PyO3 rustsec (#8381) @connortsui20
• fix: vx_data_source_new_buffer sem-merge conflict (#8437) @joseph-isaacs

📖 Documentation

• Expanding conceptual docs (and some other minor docs) (#8552) @AdamGS
• docs: update doc links (#8375) @joseph-isaacs

🧰 Maintenance

49 changes

• Back VortexSession with ArcSwap for mutable but less errorprone session management (#8631) @robert3005
• chore: forbid the locking Id constructors with a clippy lint (#8617) @miniex
• Patches have correct dtype by construction instead of normalised during array construction (#8626) @robert3005
• chore: Python CUDA bridge: CI and buffer handoff ABI (#8618) @0ax1
• Lock file maintenance (#8616) @renovate[bot]
• Polish treemap view in vortex explorer (#8613) @robert3005
• Migrate WASM Arrow conversion to use VortexSession (#8619) @robert3005
• Lock file maintenance (#8615) @renovate[bot]
• Improve SLT test harness (#8567) @AdamGS
• Add benchmarks for decimal casting (#8569) @robert3005
• Use AllNonDistinct in assert_arrays_eq and implement it for variant types (#8546) @robert3005
• Refactor DictEncoder to not use ArrayAccessors (#7759) @robert3005
• Clear 3 orphan json files from root of repo (#8558) @AdamGS
• Remove LEGACY_SESSION from benchmarks and back compat tests (#8554) @robert3005
• Bump codspeed to 5.0 (#8553) @robert3005
• Remove usages of LEGACY_SESSION from tests (#8547) @robert3005
• assert_arrays_eq and assert_nth_scalar require ExecutionCtx (#8509) @robert3005
• Update Rust crate memmap2 to v0.9.11 [SECURITY] (#8545) @renovate[bot]
• Update lance benchmark dependencies (major) (#8525) @renovate[bot]
• Update actions/setup-java digest to ad2b381 (#8521) @renovate[bot]
• Update anthropics/claude-code-action digest to 2fee155 (#8522) @renovate[bot]
• Update taiki-e/install-action digest to 9e1e580 (#8523) @renovate[bot]
• Update actions/checkout action to v7 (#8524) @renovate[bot]
• Lock file maintenance JS lock file maintenance (#8536) @renovate[bot]
• Lock file maintenance Rust lock file maintenance (#8537) @renovate[bot]
• Make sure both pushdown features are always enabled for DF benchmarks (#8507) @AdamGS
• Misc upgrades to VortexSource (#8501) @AdamGS
• Generalize SIMD take operations to Copy (#8496) @connortsui20
• Migrate to new stats falsification rules (#8345) @gatesn
• codegen only used functions in duckdb C API (#8491) @myrrc
• Change ALIGNMENT_TO_HOST_COPY to Alignment::HOST_COPY (#8488) @AdamGS
• Touch up some of the interfaces and docs in vortex-datafusion (#8485) @AdamGS
• Add benchmarks for take on runend array (#8469) @robert3005
• Include vortex-compute in codspeed benchmarks (#8477) @robert3005
• Add Kleene boolean benchmarks (#8476) @gatesn
• Immutable session backed by HashMap (#8454) @AdamGS
• test[gpu]: expand cuDF CUDA e2e coverage (#8463) @0ax1
• Upgrade pyO3 to 0.29.0 (#8462) @robert3005
• Update dependency starlette to v1.3.1 [SECURITY] (#8435) @renovate[bot]
• fmt: normalize_comments = true (#8428) @joseph-isaacs
• Update taiki-e/install-action digest to 7a79fe8 (#8409) @renovate[bot]
• Update gradle/actions digest to 3f131e8 (#8408) @renovate[bot]
• Update anthropics/claude-code-action digest to d5726de (#8407) @renovate[bot]
• Update dependency duckdb to v1.4.2 [SECURITY] (#8431) @renovate[bot]
• Update dependency pip to v26.1.2 [SECURITY] (#8432) @renovate[bot]
• Align data buffers in fastlanes benchmarks (#8393) @AdamGS
• Update plugin com.palantir.java-format to v2.93.0 (#8416) @renovate[bot]
• More arithmetic benchmarks (#8402) @gatesn
• Add binary operator benchmarks (#8396) @gatesn

0.75.0

Changes

⚠️ Breaks

• aggregate fns to have grouped aggregate kernels for sum and count (#8314) @onursatici
• Remove len parameter from ScalarFnArray::try_new (#8378) @connortsui20
• Remove all TurboQuant stuff (#8377) @connortsui20
• Fix Validity::mask_eq semantics for mixed variants (#8334) @joseph-isaacs
• Rename Validity::no_nulls to definitely_no_nulls and add execute_no_nulls (#8333) @joseph-isaacs
• Pass execution context through list view rebuild (#8274) @connortsui20
• Initialize LazyBitBufferBuilder from Mask if available, when appending masks pass them by reference (#8221) @robert3005
• layout reader ctx (#8037) @onursatici

🚧 Deprecation

• Refactor validity checks to require explicit ExecutionCtx (#8273) @joseph-isaacs
• Deprecate vortex-array public APIs that use the hidden LEGACY_SESSION (#8269) @joseph-isaacs

✨ Features

• feat(vortex-geo): native Point extension type and GeoDistance scalar function (#8372) @HarukiMoriarty
• Add arrow import/export for vortex-json (#8339) @AdamGS
• WKB import/export VTables to Arrow (#8325) @a10y
• feat[gpu]: export as Arrow device array binary (#8320) @0ax1
• feat[gpu]: sliced validity in Arrow device export (#8318) @0ax1
• add vortex-geo crate and WKB extension type (#7722) @a10y
• Expose Merge expression in vortex-jni (#8319) @robert3005
• feat(vortex-array): add Interleave array encoding (#8277) @joseph-isaacs
• feat: add cuDF C FFI (#8236) @0ax1
• feat: export dictionary arrow device arrays (#8251) @0ax1
• Add zstd compression schemes for binary data (#8304) @AdamGS
• Upgrade DataFusion to 54 (#8044) @AdamGS
• byte_length() expression (#8298) @myrrc
• feat(vortex-row): row-oriented byte encoder (size + encode passes) (#8253) @joseph-isaacs
• Support pack expressions through JNI (#8266) @AdamGS
• feat: very wide column benchmark (#8252) @joseph-isaacs
• Support ParquetVariant through JNI (#8129) @AdamGS
• Introduce a new JSON extension type in vortex-json (#8215) @AdamGS
• feat[gpu]: arrow device array decimal export (#8155) @0ax1
• feat[gpu]: arrow device array list view export (#8219) @0ax1
• Add RoaringBitmap support to vortex-jni bindings (#8220) @robert3005
• Implement ZipKernel for ListViewArray (#8218) @robert3005
• move towards BitBuffer/{Mut}View over owned bit buffers (#8208) @joseph-isaacs

🚀 Performance

• perf: branchless primitive zip kernel (#8270) @joseph-isaacs
• perf[gpu]: coalesce streaming kernels (#8357) @0ax1
• perf[mask]: check AllTrue/AllFalse in eq (#8354) @joseph-isaacs
• BitBuffer binary operations support in place updates to lhs (#8162) @robert3005
• Optimize slice for dict and minor changes in other arrays (#8321) @AdamGS
• perf: remove unneeded scalar_at and valid_at and use arrays instead (#8307) @joseph-isaacs
• Allow caching LayoutReader in VortexFile (#8244) @myrrc
• perf: branchless boolean zip kernel (#8275) @joseph-isaacs
• Initial impl for sort pushdown in DataFusion FileSource implementation (#8235) @AdamGS
• Optimize dict array validation to use true_count() (#8263) @joseph-isaacs
• Add #[inline] hints to hot path functions (#8260) @joseph-isaacs
• perf(fastlanes): fuse bit-packed compare into a transposed mask + untranspose (#8239) @joseph-isaacs
• case_when with is_null -> fill_null (#8245) @onursatici
• perf: optimize varbinview construction (#8146) @joseph-isaacs
• ViewedLayoutChildren child layout cache (#8234) @myrrc
• Don't recalculate chunk offsets for ChunkedReader (#8231) @myrrc
• Share FSST symbol table state between executed variants of arrays. (#8222) @AdamGS

🐛 Bug Fixes

13 changes

• fix: keep Arrow device schemas aligned with exports (#8360) @0ax1
• aggregate_fn: Return NaN instead of null for mean of all-null input (#8365) @dimitarvdimitrov
• fix[file]: read the one-row pruning result in can_prune instead of re… (#8369) @tomsanbear
• Fix generation of stat pruning expression for unsupported dtypes (#8326) @robert3005
• fix[gpu]: decode extension arrays on the GPU (#8353) @0ax1
• fix issue with FileSystem path handling (#8248) @robert3005
• Wrap object stores in python with Compat layer (#8316) @robert3005
• Fix semantic conflict in vortex-array (#8312) @AdamGS
• fix: [profile.bench] codegen-units = 16 (#8257) @joseph-isaacs
• fix grouped sum index for listview array (#8255) @onursatici
• split registration to respect nested projection and filters (#8213) @onursatici
• Fix zoned min/max over fixed-size-list stats (#8243) @gatesn
• fix session deadlock (#8217) @onursatici

📖 Documentation

• Validate all rustdoc builds (#8386) @AdamGS

🧰 Maintenance

40 changes

• chore: clean up Arrow device export (#8359) @0ax1
• chore: polish cuDF C FFI (#8358) @0ax1
• Remove the unused website and clean other dependencies (#8362) @AdamGS
• Minor touchups for vortex-datafusion (#8356) @AdamGS
• Add docs for StatsRewriteRule (#8355) @AdamGS
• Disable wide-columns benchmark to stop benchmarks from timing out (#8331) @AdamGS
• Optimize fsst_compress_offsets_overflow_i32 by 2X locally (#8324) @AdamGS
• Add slicing benchmarks for vortex-buffer (#8323) @AdamGS
• test[gpu]: more e2e tests (#8317) @0ax1
• Publish Parquet-Variant encoding (#8313) @AdamGS
• Fix javadoc warnings and run lints in ci (#8309) @robert3005
• Update all patch updates (#8286) @renovate[bot]
• Update Rust crate tabled to 0.21.0 (#8184) @renovate[bot]
• Update logback monorepo to v1.5.34 (#8287) @renovate[bot]
• Update taiki-e/install-action digest to 0631aa6 (#8285) @renovate[bot]
• Update actions/checkout digest to df4cb1c (#8281) @renovate[bot]
• Update anthropics/claude-code-action digest to fbda2eb (#8282) @renovate[bot]
• Lock file maintenance Rust lock file maintenance (#8296) @renovate[bot]
• Update aws-actions/configure-aws-credentials digest to e7f100c (#8283) @renovate[bot]
• Update netty monorepo to v4.2.15.Final (#8288) @renovate[bot]
• Update storybook monorepo to v10.4.2 (#8290) @renovate[bot]
• Update react monorepo (#8289) @renovate[bot]
• Update tokio-prost monorepo to v0.14.4 (#8291) @renovate[bot]
• Update typescript-eslint monorepo to v8.60.1 (#8292) @renovate[bot]
• Update tanstack-virtual monorepo to v3.14.2 (#8293) @renovate[bot]
• Lock file maintenance JS lock file maintenance (#8295) @renovate[bot]
• Update codecov/codecov-action action to v7 (#8294) @renovate[bot]
• Update codecov/codecov-action digest to fb8b358 (#8284) @renovate[bot]
• Ignore RUSTSEC-2026-0173 (#8300) @AdamGS
• Rename ArrayEq/Hash Precision to EqMode (#8214) @AdamGS
• use octet_length() instead of len() in clickbench (#8267) @myrrc
• Mask bool iterator (#8258) @onursatici
• Update dependency starlette to v1 [SECURITY] (#8254) @renovate[bot]
• bench: sweep bit-packed compare across all int types and bit widths (#8238) @joseph-isaacs
• Make from_arrow not panic (#8242) @connortsui20
• clean: drop unused API lock-file (#8241) @0ax1
• CachedIds for functions (#8240) @myrrc
• vortex-io: Add explicit span on spawn_io (#8232) @brancz
• Add select_unpredictable hint to search_sorted (#8225) @AdamGS
• Add ArcSwapMap and use it throughout our session registries (#8072) @robert3005

0.74.0

Changes

⚠️ Breaks

• ffi: remove vx_file scan (#8152) @myrrc

✨ Features

• Expose row_idx in vortex-jni bindings (#8211) @robert3005
• Support for importing Arrow extension arrays into core vortex arrays and ParquetVariant support. (#8125) @AdamGS
• feat: add experimental onpair string encoding (#8144) @joseph-isaacs
• Teach zoned pruning to lower StatFn (#7937) @gatesn
• TurboQuant: Add distortion benchmark (#8070) @connortsui20
• Add UUID literal expression to vortex-jni (#8154) @robert3005
• Introduce basic binary compressor schemes and refactor some other bits (#8153) @AdamGS

🚀 Performance

• perf: trim zctl listview instead of full rebuild (#8164) @mhk197
• Use Vec instead of BTreeSet for splits (#8194) @myrrc
• Skip empty chunks when aggregating over chunked arrays (#8168) @robert3005
• perf: fast FixedSizeListArray canonicalization for chunked arrays (#8161) @joseph-isaacs
• Always use FileStatsLayoutReader if available for VortexFile reader (#8147) @connortsui20
• Use selection in repeated scans (#8137) @myrrc
• Reduce monomorphization by dispatching over unsigned integer widths (#8110) @joseph-isaacs

🐛 Bug Fixes

• fix(fuzz): VortexError in file_io (#8189) (#8201) @vortex-claude[bot]
• fix: onpair use crates verison (#8200) @joseph-isaacs
• Fix unsound Delta unsigned-widening cast (#8195) @joseph-isaacs
• fix[fastlanes]: remove unsound delta cast? (#8192) @joseph-isaacs
• Always try compressing extension array's storage directly (#8157) @AdamGS
• fix: code coverage ci task (#8142) @0ax1
• fix: speed up coverage tests and LLVM tools path (#8136) @0ax1
• Pin cargo-edit to version that still supports rust 1.91.0 (#8141) @robert3005

📖 Documentation

• docs: link to Vortex Explorer (#8132) @danking

🧰 Maintenance

22 changes

• Refactor: Split monolithic scheme files into modular submodules (#8207) @joseph-isaacs
• test: expand utf8view and binaryview cuda export coverage (#8159) @0ax1
• Update all patch updates (#8174) @renovate[bot]
• don't account array deallocation in random access bench (#8160) @myrrc
• Update aws-actions/configure-aws-credentials digest to 99214aa (#8170) @renovate[bot]
• Update docker/setup-qemu-action digest to 0611638 (#8171) @renovate[bot]
• Update EmbarkStudios/cargo-deny-action digest to bb137d7 (#8172) @renovate[bot]
• Update taiki-e/install-action digest to e49978b (#8173) @renovate[bot]
• Update Rust crate opentelemetry_sdk to v0.32.1 (#8178) @renovate[bot]
• Update dependency concurrently to v10 (#8186) @renovate[bot]
• Update Rust crate serial_test to v3.5.0 (#8183) @renovate[bot]
• Update plugin com.diffplug.spotless to v8.6.0 (#8182) @renovate[bot]
• Update dependency typescript-eslint to v8.60.0 (#8181) @renovate[bot]
• Update dependency lucide-react to v1.17.0 (#8180) @renovate[bot]
• Update crate-ci/typos action to v1.47.0 (#8179) @renovate[bot]
• Update dependency eslint to v10.4.1 (#8177) @renovate[bot]
• Update dependency ch.qos.logback:logback-classic to v1.5.33 (#8176) @renovate[bot]
• Update dependency @tanstack/react-virtual to v3.13.26 (#8175) @renovate[bot]
• test: expand cuDF e2e primitive coverage (#8149) @0ax1
• Fix file size baseline detection (#8151) @AdamGS
• test[gpu]: expand primitive Arrow Device export coverage (#8134) @0ax1
• test: nullable cuDF e2e tests (#8128) @0ax1