Schema Evolution Support in Velox

[Yuhta]

Currently the schema evolution support in Velox is very limited: it only supports positional matching of columns and subfields based on the usage of data warehouse inside Meta, and only implemented for DWRF reader. We will need to extend the functionality to support more use cases:

1. Match by position
2. Match by field name (e.g. Gluten)
3. Match by unique permanent ID (e.g. Iceberg)

Thus here I start a discussion to collect the information first and will put out designs to be discuss.

@rui-mo Can you share some detail about how schema evolution is supposed to be handled in Gluten/Spark?

[rui-mo]

@Yuhta Below are some details.
Spark now supports both matching by ID (since v3.3.0) and by name. As quoted from Spark,

when the requested schema has field IDs, Parquet readers will first use the field ID to determine which Parquet columns to read if the field ID exists in Spark schema, before falling back to match using column names.

The config spark.sql.parquet.fieldId.read.enabled is false by default, so matching by name is still the default behavior.

Links:
To support matching by ID: apache/spark@b5eae59, and please check ParquetRowConverter.
Previously, only matching by name: apache/spark@215609d

[rui-mo]

Yes, it is in the SchemaElement under file metadata, and please check:
https://github.com/apache/parquet-format/blob/300b018e0beffb4c335b4a0d7763d9edc1f3cd06/src/main/thrift/parquet.thrift#L550.

[zhli1142015]

Hello @Yuhta and @rui-mo
May I know whether there is any plan to support matching columns by Parquet field_id in velox?
Thanks.

[rui-mo]

@zhli1142015 Since Spark disables this by default and no Gluten users have requested it, we currently have no plans to support Parquet field_id matching. Have you received any customer requirements for this?

[PingLiuPing]

@zhli1142015 I have plan to fix this in reader. I'm working on writer now to write field_id first.

[PingLiuPing]

Opened an issue #14833