Can Velox LazyVectors Do Async IO ?

[adongfan]

Hi Guys,

I noticed the introduction of LazyVector in Velox, along with its related code implementation. The design concept of LazyVector is excellent; it employs lazy loading to only fetch the necessary rows of data as needed during expression evaluation, thereby minimizing I/O as much as possible.

However, I find it somewhat odd that, the loading process for LazyVector is synchronous, whereas data loading by the connector is generally performed asynchronously from storage. Moreover, since expression execution occurs on threads rather than coroutines, this synchronous calling can continuously occupy thread resources when encountering I/O wait.

Is this current approach viable? Or are there any other techniques or considerations to take into account when using it?

[adongfan]

@mbasmanova would you please have a look at this question ? many thanks

[mbasmanova]

CC: @Yuhta @pedroerp @oerling

[oerling]

Decoding the data is done when loadingthe lazy. This is not IO. If loading the lazy is rare, synchronous IO is what you want. If this is loaded in one of a miillion Splits, then there's no poit reading, except for the one split. On te other hand, if the lazy is loaded frequently, its stream (the piece of file it referes to) is frequently loaded and coalesced into the read plan. So the data is read with the rest but it will not get touched (i.e. decompressed and decoded) until needed.