Whether use the JVM version of libhdfs to completely replace the C++ version of libhdfs3？

[JkSelf]

Currently, Velox utilizes the c++ version libhdfs3 to support the HDFS system. However, when a customer's Hadoop environment has Kerberos authentication and viewfs support enabled, Velox encounters errors with HDFS connection failures. Therefore, we plan to follow Arrow's approach and dynamically load the Hadoop and JVM libraries set up in the system during the runtime of Velox to invoke the JVM version implementation of HDFS. This would allow us to leverage more features from the Hadoop community. Given this requirement, whether use the JVM version of libhdfs to completely replace the C++ version of libhdfs3？

[assignUser]

cc: @pedroerp @majetideepak @FelixYBW

[assignUser]

I also want to note that Hawq (which libhdfs3 is part of) is on it's way to being archived and the library hasn't had any commits in 2+ years with very little activity before that. So it is also a security consideration to keep this dependency around (this is the reason arrow does no longer use it).

[assignUser]

@zhanglistar just pointed out that there is a maintained version of this that also has new features: https://github.com/ClickHouse/libhdfs3

[kecookier]

cc @chenxu14

[chenxu14]

There are some Critical bugs with libhdfs that have not been resolved yet
such as https://issues.apache.org/jira/browse/HDFS-12628

[assignUser]

The last comment (from 2019) seems to imply the problem is fixed? Can you replicate this issue?

[kecookier]

However, when a customer's Hadoop environment has Kerberos authentication and viewfs support enabled, Velox encounters errors with HDFS connection failures.

@JkSelf Could you provide some detailed information about this case?

[JkSelf]

@kecookier Velox will encounter a connection exception at here because the C++ version of libhdfs3 lacks support for Kerberos authentication.

[FelixYBW]

@JkSelf Could you provide some detailed information about this case?

It's the feature gap between libhdfs3 and jvm libhdfs. Couple of features are not supported in libhdfs3, as well as some pending bugs not fixed. jvm libhdfs is more mature but its performance is bit lower than libhdfs3.

@JkSelf can you do a quick performance test using TPCH Q6? Make sure to disable short-circuit

[kecookier]

Thanks

[zhanglistar]

While, I have the three option, we should add Kerberos authentication and viewfs support in libhdfs3 and still get accessing HDFS using native code instead of JVM to get better performance.

[assignUser]

we should add Kerberos authentication and viewfs support in libhdfs3

The hawq project is on it's way to being archived. It seems unlikely that any new features will be added.

[zhanglistar]

we should add Kerberos authentication and viewfs support in libhdfs3

The hawq project is on it's way to being archived. It seems unlikely that any new features will be added.

Yes, so we contribute code to this repo: https://github.com/ClickHouse/libhdfs3
And it supports EC now!

[assignUser]

Having a maintained version is a great improvement. Do you know if they added support for kerberos as required for @JkSelf usecase?

[jamondigimon]

Posting the relevant issue on libhdfs3 here: this . TLDR, it was mentioned that the Kerberos does not work for application <-> Hadoop KMS communication.

[wypb]

Is it possible to use libhdfs3 maintained by ClickHouse? It seems to be still maintained, and we use this internally.

[zhanglistar]

ned, and we use this internally.

Yes, https://github.com/ClickHouse/libhdfs3 this repos is still under maintained, and we contribute code to this repo, lots of features are added, such as EC supported.

[FelixYBW]

@JkSelf can you have a list of the features missing in libhdfs3? And if they are added to clickhouse/libhdfs3? If they are, looks clickhouse/libhdfs3 is a better choice.

[zhouyuan]

@zhanglistar @wypb In gluten we did tried to use CK/libhdfs3 before, we also added the delegation token support(oap-project/libhdfs3@9f234ed)
However, the gap on Kerberos support in CK/libhdfs3 is still there, e.g., Kerberos + hadoop.rpc.protection=privacy does not work. the issue exists in CK/byconity and marked as not supported
ClickHouse/ClickHouse#22236
ClickHouse/ClickHouse#51080
ByConity/ByConity#622

The viewfs support is also a gap based on what I learned

thanks
-yuan

[JkSelf]

@majetideepak @assignUser @zhanglistar @FelixYBW

We conducted performance tests on the Q6 query using a 2TB TPC-H dataset, comparing the results with HDFS Short Circuit enabled and disabled. According to the data we collected, with HDFS Short Circuit enabled, libhdfs3 performs approximately 1.08 x faster than libhdfs. However, with HDFS Short Circuit disabled, the performance of the two is quite similar.

Note: The observed 8% performance degradation occurs under extreme conditions. In real-world production environments, remote HDFS is commonly used, so the performance is same with HDFS Short Circuit disabled, which do not impact the overall performance. We conducted performance tests on 103 queries of the 2 TB TPC-DS in the same environment and found that the performance when using libhdfs and libhdfs3 is comparable, with no performance degradation.

Below are the machine models used for the tests:

Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU(s):                             112
On-line CPU(s) list:                0-111
Thread(s) per core:                 2
Core(s) per socket:                 28
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              85
Model name:                         Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
Stepping:                           7
CPU MHz:                            1000.004
CPU max MHz:                        4000.0000
CPU min MHz:                        1000.0000
BogoMIPS:                           5400.00
Virtualization:                     VT-x
L1d cache:                          1.8 MiB
L1i cache:                          1.8 MiB
L2 cache:                           56 MiB
L3 cache:                           77 MiB
NUMA node0 CPU(s):                  0-27,56-83
NUMA node1 CPU(s):                  28-55,84-111
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit:        KVM: Vulnerable
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:             Mitigation; Enhanced IBRS
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; TSX disabled
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall n
                                    x pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dte
                                    s64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer ae
                                    s xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enha
                                    nced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx5
                                    12dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm
                                    _mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d
                                    arch_capabilities

[assignUser]

Thanks for running the numbers! As I am not a user of this functionality I am not going to vote on it either way.

[majetideepak]

Based on these results, is it okay to deprecate current hawq/libhdfs3 usage in Velox and use the Java based libhdfs library?

[majetideepak]

Since Velox links against the libhdfs dynamic library, can users still use hawq/libhdfs3 by wrapping it around the libhdfs API if needed?

[FelixYBW]

library, can users still use hawq/libhdfs3 by wrapping it around the libhdf

I'd think so

[majetideepak]

Based on the experimental results and the vote, let's remove hawq/libhdfs3 and use java libhdfs.

[FelixYBW]

Based on the experimental results and the vote, let's remove hawq/libhdfs3 and use java libhdfs.

@JkSelf can you submit the PR?

[JkSelf]

@majetideepak @FelixYBW Sure. I will submit PR to support this later.

[chenxu14]

Sorry to see this too late.
I'm a user of both LIBHDFS and LIBHDFS3
In our own product env(fork from HADOOP-2.7.1), LIBHDFS often encountering some tricky problems such as MemoryLeak and JVM CRASH, and LIBHDFS3(fork from CLICKHOUSE) has performance a lot.
Maybe these issues have already been fixed in the new hadoop version，but should we consider that there are some users still using the old version?

[assignUser]

HADOOP-2.7.1

Just for reference, that version was released 9 years ago, so improvements in libhdfs are to be expected.
If you are stuck on that version for your product, I think you should be able to continue using libhdfs3 there and run velox against a newer version of libhdfs, right?

[beliefer]

In our environment, LIBHDFS3 is better than LIBHDFS.
The performance of Gluten 1.2 (LIBHDFS3) with TPCDS SF1000 benchmark is twice that of Gluten 1.4 (LIBHDFS)

[FelixYBW]

Currently Gluten still support both version. It's not completely replaced.

[ayyatsenko]

For us, libhdfs3 is practically the only one option. We use ClickHouse version, with some changes for Kerberos ( with user impersonation ). For encrypted data transfer (hadoop.rpc.protection=privacy) we restored a reverted patch
ClickHouse/libhdfs3@62e7cde
(we were unable to find out why it was reverted)
We changed HdfsReadFile class in Velox to use hdfsPread() instead of hdfsRead() due to a bug in libhdfs3, which is critical for us.
In hdfsRead() internals, a RemoteBlockReader initialized with length greater than we requested, so the datanode transmit more data and overflow socket input buffer (if running in container, it can be seen in field 'sock' from /sys/fs/cgroup/memory.stat). With enabled FileHandleCache in Velox we actually get a memory leak (in some cases hundreds of megabytes).
The hdfsPread() hasn't this bug ( but has 'seek' and 'while' ).

--- a/HdfsReadFile.cpp
+++ b/HdfsReadFile.cpp
@@ -31,13 +31,7 @@ void HdfsReadFile::preadInternal(uint64_t offset, uint64_t length, char* pos)
   if (!file_->handle_) {
     file_->open(hdfsClient_, filePath_);
   }
-  file_->seek(offset);
-  uint64_t totalBytesRead = 0;
-  while (totalBytesRead < length) {
-    auto bytesRead = file_->read(pos, length - totalBytesRead);
-    totalBytesRead += bytesRead;
-    pos += bytesRead;
-  }
+  file_->pread(pos, length, offset);
 }
 
 std::string_view HdfsReadFile::pread(

--- a/HdfsReadFile.h
+++ b/HdfsReadFile.h
@@ -34,7 +34,13 @@ struct HdfsFile {
 
   int32_t read(char* pos, uint64_t length) const {
     auto bytesRead = hdfsRead(client_, handle_, pos, length);
-    VELOX_CHECK(bytesRead >= 0, "Read failure in HDFSReadFile::preadInternal.");
+    VELOX_CHECK(bytesRead >= 0, "Read failure in HDFSFile::read.");
+    return bytesRead;
+  }
+
+  int32_t pread(char* pos, uint64_t length, uint64_t offset) const {
+    auto bytesRead = hdfsPread(client_, handle_, pos, length, offset);
+    VELOX_CHECK(bytesRead >= 0, "Read failure in HDFSFile::pread.");
     return bytesRead;
   }
 };

[zhouyuan]

hi @ayyatsenko maybe below patch is useful for the case you mentioned?
ClickHouse/libhdfs3#31

[ayyatsenko]

Unfortunately, no. I tested with this patch.
Problem with hdfsRead() is here:
https://github.com/ClickHouse/libhdfs3/blob/master/src/client/InputStreamImpl.cpp#L389
The length of the requested data from the datanode does not consider a 'length' arg of the hdfsRead() call. The hdfsPread() uses another version of InputStreamImpl::setupBlockReader(), which works correctly.

[FelixYBW]

Any possible to upstream the fix to clickhouse/libhdfs3?

[ayyatsenko]

I'll try.

[FelixYBW]

Another issue we met recently is the java configures are not fully honored by c++ version. Not only libhdfs3, but the aws, azure and GCS all. We are thinking to use the other remote storage's java API as well like libhdfs in Gluten.