Unable to connect to Docker TCP service

[crawford]

Issue by MarcDiethelm

---

• MacOSX 10.9.3
• OpenSSH_6.2p2
• Virtualbox 4.3.12
• Vagrant 1.6.3
• Coreos v343.0.0 and v349.0.0

Setting $expose_docker_tcp=2375 in config.rb and invoking export DOCKER_HOST='tcp://127.0.0.1:2375' does not give the expected results.

local shell:

$ docker images
2014/06/17 10:37:10 Get http://127.0.0.1:2375/v1.12/images/json: EOF

VM:

$ cat /etc/systemd/system/docker-tcp.socket
[Unit]
Description=Docker Socket for the API

[Socket]
ListenStream=127.0.0.1:2375
Service=docker.service
BindIPv6Only=both

[Install]
WantedBy=sockets.target

Set via project user-data:

$ cat /var/lib/coreos-vagrant/vagrantfile-user-data
#cloud-config

coreos:
 etcd:
 #generate a new token for each unique cluster from https://discovery.etcd.io/new
 #discovery: https://discovery.etcd.io/<token>
 addr: $public_ipv4:4001
 peer-addr: $public_ipv4:7001
 units:
 - name: etcd.service
 command: start
 - name: fleet.service
 command: start
 runtime: no
 content: |
 [Unit]
 Description=fleet

 [Service]
 Environment=FLEET_PUBLIC_IP=$public_ipv4
 ExecStart=/usr/bin/fleet
 - name: docker-tcp.socket
 command: start
 enable: true
 content: |
 [Unit]
 Description=Docker Socket for the API

 [Socket]
 ListenStream=127.0.0.1:2375
 Service=docker.service
 BindIPv6Only=both

 [Install]
 WantedBy=sockets.target
 - name: enable-docker-tcp.service
 command: start
 content: |
 [Unit]
 Description=Enable the Docker Socket for the API

 [Service]
 Type=oneshot
 ExecStart=/usr/bin/systemctl enable docker-tcp.socket

On the VM docker commands are working. But the docker commands in the local shell fail.
Probably unrelated but I'm also seeing a ssh connection error in vagrant up. See: #131

[crawford]

Comment by philips

---

@MarcDiethelm when you login to the machine and see if docker-tcp.socket is running?

[crawford]

Comment by philips

---

systemctl status docker-tcp.socket

[crawford]

Comment by philips

---

If it isn't running do:

systemctl stop docker.service
systemctl start docker-tcp.socket docker.socket

Then try again.

[crawford]

Comment by MarcDiethelm

---

Yeah, as mentioned in the VM the Docker commands work. I did manually stop and start the service previously but no joy.

[crawford]

Comment by MarcDiethelm

---

I'm on the road currently but will try the status check when I can.

[crawford]

Comment by MarcDiethelm

---

Yup.

$ systemctl status docker-tcp.socket
● docker-tcp.socket - Docker Socket for the API
 Loaded: loaded (/etc/systemd/system/docker-tcp.socket; enabled)
 Active: active (listening) since Fri 2014-06-20 07:58:53 UTC; 1min 29s ago
 Listen: 127.0.0.1:2375 (Stream)

Jun 20 07:58:53 core-01 systemd[1]: Starting Docker Socket for the API.
Jun 20 07:58:53 core-01 systemd[1]: Listening on Docker Socket for the API.
Jun 20 07:58:54 core-01 systemd[1]: Listening on Docker Socket for the API.

But no response in the local shell.

$ docker images
2014/06/20 09:59:25 Get http://127.0.0.1:2375/v1.12/images/json: EOF

[crawford]

Comment by razic

---

vagrant version

~/coreos-vagrant master vagrant --version
Vagrant 1.6.5

user-data:

#cloud-config
coreos:
 etcd:
 addr: $public_ipv4:4001
 peer-addr: $public_ipv4:7001
 discovery: https://discovery.etcd.io/e792d9ee556fddc746ca93aa6202cadf
 fleet:
 public-ip: $public_ipv4
 units:
 - name: etcd.service
 command: start
 - name: fleet.service
 command: start
 - name: docker-tcp.socket
 command: start
 enable: true
 content: |
 [Unit]
 Description=Docker Socket for the API

 [Socket]
 ListenStream=2375
 Service=docker.service
 BindIPv6Only=both

 [Install]
 WantedBy=sockets.target
 [Socket]
 ListenStream=2375
 Service=docker.service
 BindIPv6Only=both

 [Install]
 WantedBy=sockets.target

config.rb:

# To automatically replace the discovery token on 'vagrant up', uncomment
# the lines below:
if File.exists?('user-data') && ARGV[0].eql?('up')
 require 'open-uri'
 require 'yaml'

 token = open('https://discovery.etcd.io/new').read

 data = YAML.load(IO.readlines('user-data')[1..-1].join)
 data['coreos']['etcd']['discovery'] = token

 lines = YAML.dump(data).split("\n")
 lines[0] = '#cloud-config'

 open('user-data', 'r+') do |f|
 f.puts(lines.join("\n"))
 end
end

# coreos-vagrant is configured through a series of configuration
# options (global ruby variables) which are detailed below. To modify
# these options, first copy this file to "config.rb". Then simply
# uncomment the necessary lines, leaving the $, and replace everything
# after the equals sign..

# Size of the CoreOS cluster created by Vagrant
$num_instances=3

# Official CoreOS channel from which updates should be downloaded
#$update_channel='alpha'

# Log the serial consoles of CoreOS VMs to log/
# Enable by setting value to true, disable with false
# WARNING: Serial logging is known to result in extremely high CPU usage with
# VirtualBox, so should only be used in debugging situations
#$enable_serial_logging=false

# Enable port forwarding of Docker TCP socket
# Set to the TCP port you want exposed on the *host* machine, default is 2375
# If 2375 is used, Vagrant will auto-increment (e.g. in the case of $num_instances > 1)
# You can then use the docker tool locally by setting the following env var:
# export DOCKER_HOST='tcp://127.0.0.1:2375'
$expose_docker_tcp=2375

# Setting for VirtualBox VMs
#$vb_gui = false
#$vb_memory = 1024
#$vb_cpus = 1

Building the Cluster

~/coreos-vagrant master vagrant up
Bringing machine 'core-01' up with 'virtualbox' provider...
Bringing machine 'core-02' up with 'virtualbox' provider...
Bringing machine 'core-03' up with 'virtualbox' provider...
==> core-01: Importing base box 'coreos-alpha'...
==> core-01: Matching MAC address for NAT networking...
==> core-01: Checking if box 'coreos-alpha' is up to date...
==> core-01: Setting the name of the VM: coreos-vagrant_core-01_1410922149559_88128
==> core-01: Clearing any previously set network interfaces...
==> core-01: Preparing network interfaces based on configuration...
 core-01: Adapter 1: nat
 core-01: Adapter 2: hostonly
==> core-01: Forwarding ports...
 core-01: 2375 => 2375 (adapter 1)
 core-01: 22 => 2222 (adapter 1)
==> core-01: Running 'pre-boot' VM customizations...
==> core-01: Booting VM...
==> core-01: Waiting for machine to boot. This may take a few minutes...
 core-01: SSH address: 127.0.0.1:2222
 core-01: SSH username: core
 core-01: SSH auth method: private key
 core-01: Warning: Connection timeout. Retrying...
==> core-01: Machine booted and ready!
==> core-01: Setting hostname...
==> core-01: Configuring and enabling network interfaces...
==> core-01: Running provisioner: file...
==> core-01: Running provisioner: shell...
 core-01: Running: inline script
==> core-02: Importing base box 'coreos-alpha'...
==> core-02: Matching MAC address for NAT networking...
==> core-02: Checking if box 'coreos-alpha' is up to date...
==> core-02: Setting the name of the VM: coreos-vagrant_core-02_1410922168730_19927
==> core-02: Fixed port collision for 22 => 2222. Now on port 2200.
==> core-02: Clearing any previously set network interfaces...
==> core-02: Preparing network interfaces based on configuration...
 core-02: Adapter 1: nat
 core-02: Adapter 2: hostonly
==> core-02: Forwarding ports...
 core-02: 2375 => 2376 (adapter 1)
 core-02: 22 => 2200 (adapter 1)
==> core-02: Running 'pre-boot' VM customizations...
==> core-02: Booting VM...
==> core-02: Waiting for machine to boot. This may take a few minutes...
 core-02: SSH address: 127.0.0.1:2200
 core-02: SSH username: core
 core-02: SSH auth method: private key
 core-02: Warning: Connection timeout. Retrying...
==> core-02: Machine booted and ready!
==> core-02: Setting hostname...
==> core-02: Configuring and enabling network interfaces...
==> core-02: Running provisioner: file...
==> core-02: Running provisioner: shell...
 core-02: Running: inline script
==> core-03: Importing base box 'coreos-alpha'...
==> core-03: Matching MAC address for NAT networking...
==> core-03: Checking if box 'coreos-alpha' is up to date...
==> core-03: Setting the name of the VM: coreos-vagrant_core-03_1410922188019_86776
==> core-03: Fixed port collision for 22 => 2222. Now on port 2201.
==> core-03: Clearing any previously set network interfaces...
==> core-03: Preparing network interfaces based on configuration...
 core-03: Adapter 1: nat
 core-03: Adapter 2: hostonly
==> core-03: Forwarding ports...
 core-03: 2375 => 2377 (adapter 1)
 core-03: 22 => 2201 (adapter 1)
==> core-03: Running 'pre-boot' VM customizations...
==> core-03: Booting VM...
==> core-03: Waiting for machine to boot. This may take a few minutes...
 core-03: SSH address: 127.0.0.1:2201
 core-03: SSH username: core
 core-03: SSH auth method: private key
 core-03: Warning: Connection timeout. Retrying...
==> core-03: Machine booted and ready!
==> core-03: Setting hostname...
==> core-03: Configuring and enabling network interfaces...
==> core-03: Running provisioner: file...
==> core-03: Running provisioner: shell...
 core-03: Running: inline script

Trying to use Docker Over the TCP Port

~/coreos-vagrant master export DOCKER_HOST=tcp://localhost:2375
~/coreos-vagrant master docker ps
2014/09/16 19:52:04 Get http://localhost:2375/v1.14/containers/json: EOF

Checking the Docker Services with systemctl on core-01

~/coreos-vagrant master vagrant ssh core-01
CoreOS (alpha)
core@core-01 ~ $

core@core-01 ~ $ systemctl status docker
● docker.service - Docker Application Container Engine
 Loaded: loaded (/usr/lib64/systemd/system/docker.service; disabled)
 Active: inactive (dead)
 Docs: http://docs.docker.io

core@core-01 ~ $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

wtf?

docker.service:

core@core-01 ~ $ systemctl status docker
● docker.service - Docker Application Container Engine
 Loaded: loaded (/usr/lib64/systemd/system/docker.service; disabled)
 Active: active (running) since Wed 2014-09-17 02:53:54 UTC; 2s ago
 Docs: http://docs.docker.io
 Process: 914 ExecStartPre=/bin/mount --make-rprivate / (code=exited, status=0/SUCCESS)
 Main PID: 916 (docker)
 CGroup: /system.slice/docker.service
 └─916 /usr/bin/docker --daemon --storage-driver=btrfs --host=fd://

Sep 17 02:53:54 core-01 docker[916]: [info] Listening for HTTP on fd ()
Sep 17 02:53:54 core-01 docker[916]: [473afc16.init_networkdriver()] getting iface addr
Sep 17 02:53:54 core-01 docker[916]: [473afc16] -job init_networkdriver() = OK (0)
Sep 17 02:53:55 core-01 docker[916]: [info] Loading containers:
Sep 17 02:53:55 core-01 docker[916]: [info] : done.
Sep 17 02:53:55 core-01 docker[916]: [473afc16] +job acceptconnections()
Sep 17 02:53:55 core-01 docker[916]: [473afc16] -job acceptconnections() = OK (0)
Sep 17 02:53:55 core-01 docker[916]: [info] GET /v1.14/containers/json
Sep 17 02:53:55 core-01 docker[916]: [473afc16] +job containers()
Sep 17 02:53:55 core-01 docker[916]: [473afc16] -job containers() = OK (0)

docker.socket:

core@core-01 ~ $ systemctl status docker.socket
● docker.socket - Docker Socket for the API
 Loaded: loaded (/usr/lib64/systemd/system/docker.socket; disabled)
 Active: active (running) since Wed 2014-09-17 02:49:19 UTC; 6min ago
 Listen: /var/run/docker.sock (Stream)

Sep 17 02:49:19 localhost systemd[1]: Starting Docker Socket for the API.
Sep 17 02:49:19 localhost systemd[1]: Listening on Docker Socket for the API.

docker-tcp.port:

core@core-01 ~ $ systemctl status docker-tcp.port
● docker-tcp.port.service
 Loaded: not-found (Reason: No such file or directory)
 Active: inactive (dead)
core@core-01 ~ $

core@core-01 ~ $ systemctl stop docker.service
Failed to stop docker.service: Access denied
Warning: Stopping docker.service, but it can still be activated by:
 docker.socket
core@core-01 ~ $ sudo !!
sudo systemctl stop docker.service
Warning: Stopping docker.service, but it can still be activated by:
 docker.socket
core@core-01 ~ $ systemctl start docker-tcp.socket docker.socket
Failed to start docker-tcp.socket: Access denied
Failed to start docker.socket: Access denied
core@core-01 ~ $ sudo !!
sudo systemctl start docker-tcp.socket docker.socket
Job for docker-tcp.socket failed. See 'systemctl status docker-tcp.socket' and 'journalctl -xn' for details.

why do i have to sudo? and why won't it start damn it....

Looking at journalctl

core@core-01 ~ $ journalctl -xn
-- Logs begin at Wed 2014-09-17 02:49:17 UTC, end at Wed 2014-09-17 02:58:58 UTC. --
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:omaha_request_action.cc(433)] No update.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:action_processor.cc(82)] ActionProcessor::ActionComplete: finished OmahaR
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:omaha_response_handler_action.cc(36)] There are no updates. Aborting.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:action_processor.cc(68)] ActionProcessor::ActionComplete: OmahaResponseHa
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:action_processor.cc(73)] ActionProcessor::ActionComplete: finished last a
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:update_attempter.cc(531)] Processing Done.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:update_attempter.cc(582)] No update.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:update_check_scheduler.cc(84)] Next update check in 42m10s
Sep 17 02:57:59 core-01 locksmithd[494]: LastCheckedTime=0 Progress=0 CurrentOperation="UPDATE_STATUS_IDLE" NewVersion=0.0.0.0 NewSize=
Sep 17 02:58:58 core-01 sudo[998]: core : TTY=pts/0 ; PWD=/home/core ; USER=root ; COMMAND=/bin/journalctl -xn

[crawford]

Comment by razic

---

Referenced from #169 this is the cause of this bug!

[crawford]

Comment by BugRoger

---

For what it's worth I solved this by using the following enable-docker-tcp.service. I'm pretty sure you can do this properly using systemd primitives. I monkeyed around with it quite a while but having the docker service already started when cloud-init runs and with something keep on restarting it through the socket activation I caved in and just manually orchestrated the socket start order.

 - name: enable-docker-tcp.service
 command: start
 content: |
 [Unit]
 Description=Enable the Docker Socket for the API

 [Service]
 ExecStartPre=/usr/bin/systemctl stop docker.socket
 ExecStartPre=/usr/bin/systemctl stop docker-tcp.socket
 ExecStartPre=/usr/bin/systemctl stop docker
 ExecStart=/usr/bin/systemctl enable docker-tcp.socket
 ExecStartPost=/usr/bin/systemctl start docker.socket
 ExecStartPost=/usr/bin/systemctl start docker-tcp.socket
 Type=oneshot
 RemainAfterExit=true

[crawford]

Comment by jeremyd

---

Thanks @BugRoger , +1 to your solution as the official copy/paste of cloud-config.yml does not appear to enable Dockersocket by itself. The service you posted is required to actually restart docker with the API support on boot.

[ender74]

I did create a pull request for the solution from BugRoger

coreos/coreos-vagrant#296