Conversation
|
|
||
| - The Scan API is not itself a full relational query engine. | ||
| - `LayoutReader` should not grow unknown-cardinality operator semantics. | ||
| - Vortex should not require a specific Rust async runtime such as Tokio. |
There was a problem hiding this comment.
It's weird that we would go through all of this and still assume Tokio but I haven't read all of it yet
There was a problem hiding this comment.
We already don't assume tokio, it just continues to be an explicit goal
There was a problem hiding this comment.
The double negation here implies the opposite? You want the goal to be that the runtime doesn't assume tokio? maybe I am reading too much into random ai generated strings
|
|
||
| - the host may provide a CPU scheduler | ||
| - Vortex may use it for bounded split-local CPU work | ||
| - Vortex must not assume ownership of the whole query runtime |
There was a problem hiding this comment.
What does this statement mean in practice? I think there's intent behind it but I fail to understand what this means in practice?
There was a problem hiding this comment.
I'd guess in particular in terms of use of resources, e.g. spawning threads but also unix process ownership e.g. Vortex should never crash the host. @gatesn correct me if you had sth else in mind.
There was a problem hiding this comment.
Vortex should never crash the host.
Error handling might deserve a small section in this PR. I briefly talked about this with @myrrc but I think we'll need a panic handler (the host maybe can configure) to prevent that we never crash a host.
| - split lookahead policy | ||
| - efficient materialization of output batches | ||
|
|
||
| ### What `Partitioning` Means |
There was a problem hiding this comment.
Words are hard, partitioning usually means some arrangement of data which this is not about. But maybe this is Partitioning and the other thing is Arrangament
|
|
||
| Correctness is more important than maximal pushdown. | ||
|
|
||
| ## Ordering, Limits, and Future Dynamic Filters |
There was a problem hiding this comment.
You should mention Partitioning here (or a I redefined it Arrangement). It's a super set of ordering
0ax1
left a comment
0ax1
left a comment
There was a problem hiding this comment.
Just a thought, maybe worthwhile clauding some ascii diagrams to illustrate some of the aspects.
3 participants
This RFC looks at how we can expose deeper integration with query engine internals like scheduling, threading models, buffer pools, and so on