[DISCUSSION] Usage of arrow-rs in other areas

[alamb]

In any case, at least at my company we probably have a few PiB of data written with this or an even earlier version.

BTW this is so cool to hear

Really want to make sure I don't spam this PR with too much sideinfo, but I just wanted to use this opportunity to share that I (+ a coworker) will give the talk "Scaling Data Processing for Training Workloads at DeepL Research with Rust" at this year's PyCon DE / PyData in Darmstadt (Germany), where we go a bit into detail about this!

Working with arrow-rs (+ PyO3 as the Python binding layer) has been an absolute blast so far for coming up with a highly optimized and efficient deep learning data ingress pipeline.

Especially compared to pyarrow, we've rarely or never seen any issues concerning surprisingly high resource usages, memory leaks or randomly not supported features (I'm somewhat sure selectively decoding specific rows by row index to reduce memory usage during sparse decoding isn't possible in a non-clunky way with pyarrow, and with arrow-rs's RowSelection this was trivially easy, even as a feature exposed to Python). Happy to stay connected on this topic.

Originally posted by @jonded94 in #9374 (comment)

[alamb]

I opened this discussion to figure out how to highlight this work (and maybe similar experiences)

@jonded94 do you happen to have any slides / recording you can share?

[jonasdedden]

@jonded94 do you happen to have any slides / recording you can share?

The talk will be in April, and so far we didn't give this talk at all publicly, it has been purely internal so far. Also given that the public variant of this will be a PyCon talk, means this will mainly be about how to do Python<->Rust interop and what works great in this ecosystem (and better than the C++<->Python one). There are not too many slides covering arrow-rs-specific things, just benchmarks of our end-to-end library against pyarrow. I don't we will have time to go into many arrow-rs details unfortunately.

Let me show a few slides that are probably fine to share (besides meme-heavyness..) and are potentially the most interesting for you:

Note that we ideally either want to make our library public, or contribute things back to arro3, if possible. This just unfortunately hasn't been on our agenda at all and we haven't found much time for doing so 😢

[alamb]

Thank you so much @jonded94 - I personally love meme heavy slides :)

FYI @kylebarron as your smiling face appears on these slides too

[kylebarron]

@jonded94 Cool! Is disco-parquet open source?

[paddyhoran]

I think this is really cool and interesting, especially the comparison to pyarrow. I would just note that it is one community, the arrow community, I would be careful to focus on highlighting positives (of arrow-rs) vs highlighting negatives (of pyarrow).

Look forward to checking this out :)

[jonasdedden]

@jonded94 Cool! Is disco-parquet open source?

Unfortunately not, as I stated here already:

Note that we ideally either want to make our library public, or contribute things back to arro3, if possible. This just unfortunately hasn't been on our agenda at all and we haven't found much time for doing so 😢

It's unclear to me when/whether we will get necessary capacity to do this. In any case, our internal library actually has a slightly different philosophy than arro3 (mostly driven out of deadlines, not technical reasons): Whilst arro3 appears to aim for maximising modularity and minimalness for providing a fully pyarrow-less and pure Rust-based experience, our library is a bit more monolithic and at least right now strictly dependent on pyarrow (because it uses the arrow-pyarrow subcrate for the Arrow<->PyO3 interaction).
Its strength right now lies more in the field of interacting with parquet files and datasets for random sampling workflows [1] as performant and resource efficient as possible, all through an hopefully easy to use Python interface. But this shouldn't be too hard to include in arro3 too, as I think there was a PR open for some while that adds ParquetFile and ParquetDataset classes. One easily could add methods that read a single row group with some additional RowSelection filter and such, and datasets would require some async magic for performant multi-file metadata lookups on latency-heavy filesystems, and after that there isn't too much additional magic left that our library provides right now.

[1] (i.e. continously decode just a few rows out of random row groups across dozens of different parquet datasets in parallel)

I would just note that it is one community, the arrow community, I would be careful to focus on highlighting positives (of arrow-rs) vs highlighting negatives (of pyarrow).

Totally! We want to keep the pyarrow comparisons as minimal as absolutely possible, as we don't want to point fingers at friendly communities and honestly we also haven't been too thorough at properly reporting our issues (we've opened a bunch of Github issues, but the C++/Cython codebase is just too complex for us to help debugging in any productive way). It's just that, at least for our niche use case, we have been more productive reaching absolute lowest resource consumptions with just writing a small wrapper library around arrow-rs, instead of trying to tune and understand how pyarrow's memory pools and general parquet file & dataset classes behave, as with arrow-rs you seem to have some stronger control over this.

[jonasdedden]

Hey, just wanted to share that I just came back from PyConDE 2026, and our talk was very well received 🥳 I was approached by people telling me they love to hear about Rust, especially for Python interop ❤️ We sadly were the only talk about Rust at this year's PyCon..

Eventually, there will be recordings (potentially even on YouTube); until then, you can have a look here for some high-level description of the talk and slides are available here.

We also included a slide with a comparison of pyarrow, arro3 and disco-parquet for object store reads. As our library is heavily optimized for minimal memory load [1] and we're entirely single-threaded, we are strongly beaten by pyarrow on throughput. But our memory load is far less than both pyarrow and arro3, but in arro3's case, I think this is only because it actually eagerly collects an entire file (?) from object store into RAM and converts it into a Table before serving rows from it: https://github.com/kylebarron/arro3/blob/4cf69f475bba07a6eec098b8351057ea15be0c62/arro3-io/src/parquet.rs#L77

As a comparison, we just use OpenDAL and the existing Parquet<->OpenDAL integration to have a little bit of a better memory footprint by iterating through files the same way we do on normal block storage. This should be doable in arro3 too.

[1] this comes out of a requirement of our distributed largest scale model trainings, where per dataset we need to use as little memory as possible, to not let GPU workers go OOM

[kylebarron]

in arro3's case, I think this is only because it actually eagerly collects an entire file (?) from object store into RAM and converts it into a Table before serving rows from it: kylebarron/arro3@4cf69f4/arro3-io/src/parquet.rs#L77

Yes, I think that's accurate. I spent the most time on the core Arrow classes in arro3-core and never spent that much time on trying to make Parquet loading efficient. And that was before I learned (in https://github.com/developmentseed/obstore) how to expose a Rust async stream to Python as an async iterator. We could refactor the arro3-io reader to expose an async stream of Arrow record batches without too much difficulty.

I did also prototype an async Parquet API similar to pyarrow's kylebarron/arro3#313, but I never stabilized it enough to merge it.

If either of these interest you, happy to discuss more on arro3 issues.