[Bug] PXF current CI repair progress

[MisterRaindrop]

Cloudberry Database and PXF version

This is the current breakdown of the PXF testing tasks. You can claim your own tasks.

https://docs.google.com/spreadsheets/d/1AXn5xVy0nVnxHdRwgTP7H6vwrXFPrGGKPy-bMLD6SA0/edit?usp=sharing

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.

[ostinru]

@liuxiaoyu , do you have any progress? Do you need any help from community?

[MisterRaindrop]

@ostinru
We do need some help. The entire PXF test case is very large. We have completed two-thirds of the automation. The remaining tests are not easy to handle. Tomorrow I will find help to break down the tasks and get more people from the community involved.
@tuhaihe I might need help, breaking down tasks

[tuhaihe]

Sure, please let me know what I can help.

…

------ Best, Dianjin Wang

[MisterRaindrop]

@ostinru I have already broken down the tasks. You can claim the PXF testing tasks. Please apply for access to this document.

We have now submitted the merged code to the merge-with-upstream branch. We are now submitting a PR for our fixed CI code #19.

[ostinru]

We have disabled load test for open-gpdb/pxf - it seems that there are not enough disk space in free github actions runners.

@liuxiaoyu, from list of tests with no assignee, I can get pxf*ExtensionVersion*. At least, I will get more familiar with your version of automation workflow.

[MisterRaindrop]

@ostinru Can you apply for the document on the Google Sheets above? I will give you permission to edit the document.

[MisterRaindrop]

@ostinru The previous my test PR #19 has been submitted. Now you can directly submit a PR, rather than creating a new PR on top of an existing one. We can now start iterating quickly.

[ostinru]

Leftovers:

• security tests - it seems that all tests from security also in other test-suites we are running.
• multi-cluster tests - @MisterRaindrop, do you think that we need it at this stage? If no, it seems that merge-with-upstream is ready to be merged to the master.

[MisterRaindrop]

I think multi-cluster tests are not necessary tests.
What do you think? @my-ship-it @tuhaihe

[tuhaihe]

I think multi-cluster tests are not necessary tests. What do you think? @my-ship-it @tuhaihe

Hi, I think the CI looks good now. For the multi-cluster tests, we can make it in the future IMO. cc @MisterRaindrop @ostinru

[tuhaihe]

Hi @MisterRaindrop, could you send a summary email to the dev mailing list (dev@cloudberry.apache.org) about PXF updates? We need more developers to be aware of this milestone.

BTW, I see that the merge-with-upstream branch has some commits behind the main branch: merge-with-upstream...main. Could you help confirm whether there is a need to cherry-pick them into the work branch?

[ostinru]

I just figured out that pxf_fdw extension should be tested with separate automation tests run (make USE_FDW=true)[1]. I have run it, and test results are not as good[2] as other tests:

Tests run: 381, Failures: 49, Errors: 0, Skipped: 203

I thin that we can fix it. However, firstly we should decide on where we keep pxf_fdw extension sources:

• in cloudberry-pxf/fdw - this version failed to compile against cloudberry-pxf[4].
  And we don't compile and install it when run make install due to [3].
• in cloudberry/gpcontrib/pxf_fdw - this version works well, however uses deprecated (v15) API.

These versions diverged slightly, hovever it seems to be easy to backport changes to either of them.

PS: I don't think that this should prevent us from merging merge-with-upstream into master because we haven't decided which way pxf or pxf_fdw we recommend to users[5].

[1] 2405174
[2] ostinru#12
[3] https://github.com/apache/cloudberry-pxf/blob/merge-with-upstream/Makefile#L10
[4] https://gist.github.com/ostinru/768e4fc07f5d36e15d403e262b00499a
[5] apache/cloudberry#983 (comment)

[tuhaihe]

Let's first update the pxf_fdw codes in cloudberry/gpcontrib, then synchronize those changes with cloudberry-pxf/fdw? This way, both repositories will stay aligned.

Then, we have a separate automation test for pxf fdw in this repo? Just my thoughts, for your information.

[MisterRaindrop]

I previously sent an email for discussion but didn't get a specific outcome.
https://lists.apache.org/list?dev@cloudberry.apache.org:2025-10

I have a few questions about PXF:

1. I'd like to better understand how PXF is integrated into Cloudberry ?
2. Can the installation of PXF be simplified further? For example, could everything be set up and running by just installing an RPM package?
3. In the future, will PXF only be used for Cloudberry, or will the GP series also be considered?

@ostinru @tuhaihe @my-ship-it