CouchDB docker image's port 5984 is not exposed with containerd-1.6.x

[sshedi]

CouchDB-2.3.0 docker image doesn't work as expected with containerd-1.6.6

Expected Behavior

Port 5984 should start listening.

Current Behavior

Once image is deployed, port 5984 is not listening.

More info at vmware/photon#1379

Steps to Reproduce (for bugs)

1. Issue is reproducible in Fedora-36 and containerd-1.6.9 as well.
2. systemctl start docker && CONTAINER=$(docker run -d --rm couchdb:2.3.0) && docker exec -it ${CONTAINER} /bin/bash -c "apt update ; apt -y install net-tools && sleep 5 && netstat -pantu" && docker rm -f ${CONTAINER}

Your Environment

vmware/photon#1379, Photon OS 4.0

Fedora 36, containerd-1.6.9, docker-20.10.21

PS:
This works as expected with containerd-1.4.13

[big-r81]

Does it work with the latest image? You shouldn't use this old version.

[aad-griff]

It's important to install the containerd-1.6.6-4.ph4.x86_64 and reboot the VMware Photon guest OS before testing.

After more testing I have discovered that I can only reproduce this issue on a VMware hypervisor specifically with the "VMXNET 3" network adapter (changing to E1000E resolves the issue)

When running the container manually, netstat should report port 5984 listening:

CouchDB 2.3.1:

docker run -it --rm couchdb:2.3.1 /bin/bash;
apt update;
apt install -y net-tools;
/docker-entrypoint.sh /opt/couchdb/bin/couchdb &
sleep 10; # Wait for couchdb to start and initialise a listening port
netstat -pantu # This should list port 5984 as listening but it doesn't start listening at all!

CouchDB 3.1.2:

docker run -it --rm -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password couchdb:3.1.2 /bin/bash;
apt update;
apt install -y net-tools;
/docker-entrypoint.sh /opt/couchdb/bin/couchdb &
sleep 10; # Wait for couchdb to start and initialise a listening port
netstat -pantu # This should list port 5984 as listening but it doesn't start listening at all!

CouchDB 3.2.2:

docker run -it --rm -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password couchdb:3.2.2 /bin/bash;
apt update;
apt install -y net-tools;
/docker-entrypoint.sh /opt/couchdb/bin/couchdb &
sleep 10; # Wait for couchdb to start and initialise a listening port
netstat -pantu # **This should list port 5984 and works as expected**

I strongly suspect this is an issue with the kernel module responsible for controlling the "VMXNET 3" network adapter and not couchdb but I am reporting it here for completeness.

VMWare Environment:
Version: 7.0.3
Build: 20395099

Kernel: 5.10.152-3.ph4-esx #1-photon x86_64

[big-r81]

What's shown in the logs of CouchDB?

[aad-griff]

Appologies!!! This does work using CouchDB 3.2.2! But not CouchDB 2.3.1 or CouchDB 3.1.2.
For versions 2.3.1 and 3.1.2, I get absolutely no output after executing /docker-entrypoint.sh /opt/couchdb/bin/couchdb it appears to completely stall during startup.