Which presets would you like to see in a CMakePresets.json

[assignUser]

I would like to make running certain configurations of Velox easier, e.g. to replicate CI builds without having to do a bunch of manual flag setting and to clean up and simplify the Makefile. You can read about the technical aspect here

Some configurations/presets are obvious:

• minimal (with dwio)
• maximal aka adapters in CI
• Pyvelox
• GPU
  • should libcudf be separate or combined with wave? @bdice
• CI configs
  • adapters (see maximal)
  • benchmark
  • fuzzer (with python legacy)

In addition these could all be done as a permutation with:

• release/debug
• testing on/off
• shared on/off (off -> still mono as it's now the default)

Then I think there should also be a 'default' which imo should be roughly what you get now when you run cmake without any additional flags set but in recent times a lot of stuff was added to that config and we should discuss what this 'default' should encompass.

I think it should be what we think of as the core functionality of Velox but I am not sure what exactly that means?

• exec
• aggregates
• functions (presto, spark, iceberg(?))
• dwio (all formats? just parquet?)
• simple/udf
• ?

Things I would lean towards not adding (just feelings in regards to dependencies, no specific arguments):

• geos
• faiss
• remote functions
• tpch/tpcds connectors
• runner
• ...?

Cloud adapters should obviously not be part of default.

Are there any other presets you would like to see? Or do you have thoughts about the ones listed above, please discuss!

[MBkkt]

cmake presets are good. But from my experience if they are too difficult (theres a lot of names, inheritance, etc) people tends to don't use it.
Build configs are quite useless (only configure really helps).

In addition these could all be done as a permutation with:

For an example I see when people rewrote CI to use cmake presets, after this common devs avoided presets because they are became too difficult to change value/understand what options are used/etc

Maybe the good options is have few simple build + "examples" configs.
Have something separate for CI?
And everything else people can write in their own user presets json file?

WDYT?

[MBkkt]

So in short, I think people need to simple way to specify multiple cmake parameters, sometime change few of them.

It can be console history, simple cmake presets, simple makefile, some bash scripts

At the same time, CI needs different and I think these intersects not ideally

[assignUser]

simple way to specify multiple cmake parameters,

Right, that's what presets are, I agree that they should be written in a way that is relatively simple to follow. For example the Arrow ones are a bit to complex for most people to modify with all their inheritance. But they should still be simple enough to use. And if you or others don't like them for some reason, you can just... not use them?

Have something separate for CI?

Why? presets is exactly for this? I don't want to have an extra script to replicate CI build if I can make use of presets which will always be in sync with what happens in CI.

And everything else people can write in their own user presets json file?

They can always do that if the few presets we add in CMakePresets.json aren't to their liking.

[MBkkt]

Ofc they can, you asked about opinion, I wrote it.

The main point if you want users use presets it probably should be easy to do following:

1. understand what preset to run
2. change/add few cmake parameters to existing preset

2

In ideal world cmake presets should allow to do something like

cmake --preset <name> -DMY_PARAMETER=ON

But they are not.

Have something separate for CI?

I meant that maybe it makes sense to separate CI and user configure command.
It can be preset for CI and makefile for user, or opposite.

[assignUser]

you asked about opinion

You are right, sorry ^^

In ideal world cmake presets should allow to do something like

Good news, that does work! Any explicitly set variable via -D... will override the preset:

❯ cmake --preset default                                                                                  
Preset CMake variables:

  foo="bar"

-- The C compiler identification is GNU 15.2.1
[...]
-- foo: bar

❯ cmake --preset default -Dfoo=foo
-- The C compiler identification is GNU 15.2.1
[...]
-- foo: foo

It can be preset for CI and makefile for user, or opposite

I see, I would be fine with that actually, a lot of people seem to prefer the make file 🤷 but for CI and replicating the env setup it would be nice to have presets. (I personally also dislike big organically grown Makefiles like we have here, they abstract a lot of cmake functionality away (like building just the one target you are working on, instead of everything.)

[MBkkt]

Good news, that does work! Any explicitly set variable via -D... will override the preset:

Strange, I thought I tried this and it didn't work. Anyway I glad to know it works!
I think something like default/debug/user/generic preset will be nice to have in such case!

[bdice]

Responding to the tag on GPU: it probably makes sense to separate cuDF and Wave, since cuDF requires additional dependencies that Wave does not require.

Generally I do not know much about presets or how to use them. I am happy with the current setup of passing -DVELOX_ENABLE_CUDF=ON.

[zoltan]

I'm just chiming in to say that I agree and I also think cuDF and Wave should be separate.

[pedroerp]

Regarding the default preset: my take is that it should contain everything needed to enable you to run a full query plan, without adding much beyond that. So that would include include exec, dwio (Parquet and DWRF), and Hive connector.

There is a question of which dialect we assume is default. For now we end up using Presto because it was added first and probably has better coverage and stability. I feel it should be either Presto or Spark, and having the other as optional. Fuzzer, testing infrastructure, benchmarks, cloud storage adapters, PyVelox, GPU, and such should all be optional IMHO.

[assignUser]

Sounds good, I'll put something together.