PXF suggest adding a roadmap

[MisterRaindrop]

I suggest adding a roadmap to better plan the functionality of PXF. What are some better suggestions?
@tuhaihe @ostinru

[tuhaihe]

Thanks for raising this — I agree that having a roadmap would be very helpful.

While I’m not deeply familiar with the detailed technical evolution of PXF itself, I’d like to share a few suggestions from a community, marketing, and branding perspective, especially considering its role within Apache Cloudberry:

1. Documentation refresh and branding alignment
   Update existing documentation to replace legacy Greenplum-related branding with Apache Cloudberry–aligned terminology where appropriate. This helps avoid confusion for users and reinforces the current project identity.

2. Documentation integration with the Cloudberry website
   Ensure PXF documentation is properly integrated or cross-referenced with the Cloudberry official website, so users can easily discover, understand, and adopt PXF as part of the broader ecosystem.

3. Release alignment and packaging support
   Align the PXF release cadence with Cloudberry’s release cycle where possible, and provide standard distribution formats such as .deb, .rpm, and ASF-compliant source release artifacts. This improves usability, consistency, and adoption across different environments.

These steps can help improve clarity, adoption, and overall ecosystem cohesion. Happy to hear others’ thoughts as well.

[ostinru]

From our side - we haven't planned adding major features to PXF. My scope is mostly maintenance:

• add Java 17 support
• continue updating dependencies to close security warning that github generates. Most of issues almost impossible to exploit, however our policies recommend to fix it anyway.

We can consider:

• upgrading hadoop-related libraries to v3.x.
• Also, I was thinking about using TestContainers to start singlecluster/minio containers (basically replace run-test.sh with pure java). It is big change, and it doesn't have impact on end-users.

[MisterRaindrop]

some suggestions to enhance the functionality of PXF.

1. PXF parallel data loading
2. lake house example iceberg/hudi support ?
3. PXF pushdown capabilities

[ostinru]

lake house example iceberg/hudi support ?

It would be great! I'd like to brainstorm on this in separate thread. However I am not sure that we will have people allocated full time on this.

PXF pushdown capabilities

I have seen new pushdown-s in arenadata/pxf repository. I think that we can cherry-pick some of them. However our (internal) policy requires 100% understanding of what is going to be merged, so it often takes time comparable with time for implementing from scratch =(

[MisterRaindrop]

lake house example iceberg/hudi support ?

It would be great! I'd like to brainstorm on this in separate thread. However I am not sure that we will have people allocated full time on this.

PXF pushdown capabilities

I have seen new pushdown-s in arenadata/pxf repository. I think that we can cherry-pick some of them. However our (internal) policy requires 100% understanding of what is going to be merged, so it often takes time comparable with time for implementing from scratch =(

Lakehouse can be gradually advanced for now. I understand that we can start by trying to achieve the most basic readability. The entire lakehouse involves a wide range of aspects, and we may need design documents to discuss it. If we only aim for a simple implementation of functionality, it should be relatively easy.

For pushdown, I can first select a batch of submissions and then make targeted modifications.

[MisterRaindrop]

I think you can open an lake house issue so we can discuss and design it accordingly. We can start by implementing the most basic version. @ostinru

[tuhaihe]

Hi @MisterRaindrop could you please send one email to the Dev mailing list to get more feedback?

[ostinru]

Also IMPORT FOREIGN SCHEMA is on my wishlist (haven't tested it with pxf_fdw).

[MisterRaindrop]

lake house example iceberg/hudi support ?

It would be great! I'd like to brainstorm on this in separate thread. However I am not sure that we will have people allocated full time on this.

PXF pushdown capabilities

I have seen new pushdown-s in arenadata/pxf repository. I think that we can cherry-pick some of them. However our (internal) policy requires 100% understanding of what is going to be merged, so it often takes time comparable with time for implementing from scratch =(

arenadata/pxf Which project is this? I didn't find it.

[ostinru]

Arenadata is a russian company that (as far as I know) originally provided support for on-premise Greenplum installations and successfully merged a bunch of patches to mainline Greenplum. Then they extended their offerings with hadoop/clickhouse/kafka/etc in a single one Data Platform.
Currently they have forked Greenplum 7 and named it as Greengage 7. Regarding PXF... they have two repositories
https://github.com/arenadata/pxf (GP6)
https://github.com/GreengageDB/pxf (GP7)

PS: After Greenplum went proprietary, Yandex Cloud and Arenadata failed to cooperate and work on single repository. Yandex Cloud switched to Cloudberry.

[tuhaihe]

Thanks for the background.

Yes, I’m aware of the team and the Greengage DB project they initiated. They were indeed an important contributor group in the Greenplum ecosystem. From my understanding, however, their project is still mainly maintained by their internal team, and it hasn’t fully moved away from the single-vendor driven model.

I’ve had some communication with them before, but as far as I know they currently don’t have plans to join the Cloudberry community.

That said, some of their ideas and implementations could still be valuable references for us. We can definitely look at those and see what might make sense to adopt or evolve within cloudberry-pxf.