Skip to content

Conversation

@vustef
Copy link

@vustef vustef commented Nov 17, 2025

Implements support for Arrow's RunEndEncoded (REE) layout as specified in the Arrow format specification. REE is
a run-length encoding variant that efficiently stores arrays with repeated values using two child arrays:
run_ends (indices where runs terminate) and values (the actual run values).

Implementation

  • Core type: Added Arrow.RunEndEncoded{T,R,A} struct with O(log n) binary search indexing
  • Type system: Registered RunEndEncodedKind in ArrowTypes module
  • Serialization: Implemented arrowvector() and makenodesbuffers!() for writing REE arrays
  • Deserialization: Added build() function and juliaeltype() for reading REE arrays from Arrow IPC format
  • Interoperability: Validated against PyArrow-generated test files (included as fixtures)

Testing

  • Cross-language validation using PyArrow 20.0.0-generated test files
  • Round-trip tests for various data types (integers, floats, strings, booleans, with nulls)
  • Edge cases: single runs, alternating values, long runs

Closes #476

function Base.iterate(r::RunEndEncoded{T}) where {T}
isempty(r) && return nothing
# State: (current_physical_index, current_logical_index, run_end)
run_idx = 1
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unnecessary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature request] support run-end encoded layout

1 participant