|This guide is based on an old version of Docker. The instructions you find below may be already removed or changed in the Docker codebase.|
In my last blog post I explained how to run a Docker installation across multiple hosts. In the comments I was asked if it would be possible to use a DHCP server to assign IPs to the containers. I thought immediately — why not? In fact, assigning IPs using DHCP is a nice way to overcome the IP assignment issue I talked about before.
The set up is pretty similar to my earlier work.
The difference is that I want to run a DHCP server on each of the hosts.
It will be responsible for assigning the IPs to all of the containers connected to the
docker0 bridge on that host.
You may ask why I want to run so many DHCP servers? Of course one server would do the job of assigning the IP’s (in fact, this was my initial set-up). Then I realized that it would be easier to manage routing of the container’s traffic if we have a DHCP server on every host.
Since we want to have the containers run on the same network, there
will be many DHCP servers working on that network. The trick is to reserve a
bunch of IP’s for each host so as not to interfere with other hosts (in my case it’s
10 addresses per host) and drop the DHCP requests on each
br0 Open vSwitch
Additionally I rewrote my earlier script that prepares the virtual machines for me. I decided to drop the use of cloud-init entirely in favor of libguestfs — a great Swiss Army knife written by Rich. If you’re not familiar with it — it’s a tool for offline manipulation of disk images. And it does the job damn well.
You can install libguestfs tools on your system by running this command:
yum -y install libguestfs-tools
liguestfs-tools before you use my scripts.
The first run of
The new scripts are avilable in the docker-dhcp repository on GitHub.
The logic of preparing and registering the image as a libvirt domain was split into two scripts:
prepare-image.sh— update the system, modify the image,
register-domain.sh— copy the image to a custom location and register a domain under the new name.
The split made it possible to create the image once and create as many domains as you want in seconds.
I placed a lot of comments in these scripts, so I hope everything is self-explaining. Let me know if that’s not the case!
To create a VM with everything preinstalled to make the setup easier the only thing you need to do is to download the QCOW2 image from cloud.fedoraproject.org and run the script providing the cloud image as the parameter, like this:
$ ./prepare-image.sh Fedora-x86_64-20-20131211.1-sda.qcow2 Wed, 29 Jan 2014 12:20:57 +0100 Cleaniung up... Wed, 29 Jan 2014 12:20:58 +0100 Modifying the image... Wed, 29 Jan 2014 12:25:15 +0100 Resizing the disk... Wed, 29 Jan 2014 12:26:06 +0100 Image 'image.qcow2' created!
|Executing the above script can take some time. It took about 5 minutes for me to prepare the image. Please keep in mind that it does the full system update and it installs some additional software as well.|
This script injects the
network.sh script which will be used later to
configure the network inside the hosts.
Now we have the base image prepared:
image.qcow2. Time to make use of it and
register two domains based on it. For this purpose I use the
$ ./register-domain.sh image.qcow2 host1 Wed, 29 Jan 2014 13:52:51 +0100 Installing the domain and adjusting the configuration... Wed, 29 Jan 2014 13:52:51 +0100 Domain host1 registered! Wed, 29 Jan 2014 13:52:51 +0100 Launching the host1 domain... Wed, 29 Jan 2014 13:52:52 +0100 Domain started, waiting for the IP... Wed, 29 Jan 2014 13:53:30 +0100 You can ssh to the 192.168.122.144 host using 'fedora' username and 'fedora' password or use the 'virsh console host1' command to directly attach to the console
You can log-in using the
If you don’t want to run the domain immediately after creation — use the
register-domain.sh script twice with
host2 as the arguments.
DHCP configuration explained
The DHCP server will be run on every host and listen only for requests on the
interface since it’s configured to look for the
172.16.42.0/24 network only.
The first 9 IP addresses from this network are reserved (can be assigned
manually to additional hosts), the rest will be available to the DHCP clients.
Every DHCP server will only be responsible for a part of the network. For
example, the server on
host1 will assign addresses from
host2 will will assign addresses from
|This is just an example — you can expand the default values to run more than 10 containers on one host.|
docker0 network interface will have the
host2 will have
172.16.42.2, and so on.
Make it work!
I assume that we have already started
host2 as explained above.
Now it’s time to prepare the networking on both hosts. Log-in to both hosts and
get the IP addresses of the
eth0 network interfaces. Now run the
script (it’s located in the
/home/fedora directory) on both hosts.
I assume that
host1 run the script with the IP of
$ sudo ./network.sh 1 192.168.122.2
And do the opposite on
$ sudo ./network.sh 2 192.168.122.31
When you run the
The GRE tunnel should now be established and a DHCP server should be running on
host1. You can confirm this by pinging the
docker0 bridge addresses on each host.
There is one requirement for the container image — it needs to have a DHCP
client installed. Sadly the default
fedora image does not have the
package installed. To make things easy I prepared the
image. The only difference between
fedora image is the addition of
Download this image on both hosts:
docker pull goldmann/fedora-dhcp
If you run the
goldmann/fedora-dhcp image you’ll see that there is no network
interfaces beside the loopback. This is because Docker is run with the
-b=none flag and it does not know about any network interfaces to bind to, so
it does not create the ethernet adapter in the container.
But we still want to have networking. The only option at the moment is to use
-lxc-conf flag when running the image, like this:
docker run -i -t \ -lxc-conf="lxc.network.type = veth" \ -lxc-conf="lxc.network.link = docker0" \ -lxc-conf="lxc.network.flags = up" \ goldmann/fedora-dhcp /bin/bash
This will start a new container with a virtual ethernet adapter which is
attached to the
docker0 bridge. Sweet!
Obtaining the IP address
Since the Docker container does not run anything besides the command you
specify (in our case
/bin/bash) — it does not run the scripts that
configures the network too. We need to do it by hand.
|I hope this will change in the near future. One option is to make systemd run well in the Docker containers.|
After you get the prompt from the container, you can simply run the
dhclient command. This will obtain the address from the DHCP server, exit and
leave a shell just for you.
bash-4.2# ip a s dev eth0 17: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 1e:d4:13:c7:9d:fd brd ff:ff:ff:ff:ff:ff inet6 fe80::1cd4:13ff:fec7:9dfd/64 scope link valid_lft forever preferred_lft forever bash-4.2# dhclient bash-4.2# ip a s dev eth0 17: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 1e:d4:13:c7:9d:fd brd ff:ff:ff:ff:ff:ff inet 172.16.42.14/24 brd 172.16.42.255 scope global dynamic eth0 valid_lft 43197sec preferred_lft 43197sec inet6 fe80::1cd4:13ff:fec7:9dfd/64 scope link valid_lft forever preferred_lft forever bash-4.2# ping -c 1 google.com PING google.com (220.127.116.11) 56(84) bytes of data. 64 bytes from ee-in-f139.1e100.net (18.104.22.168): icmp_seq=1 ttl=39 time=55.6 ms --- google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 55.672/55.672/55.672/0.000 ms
You can also use the
You can (should!) try it on
host2. You should get the same
result with no IP conflict and be able to access the Internet as well as other
containers on the network.
Things to improve
There are of course some things that could be improved to make this setup easier.
Make systemd available in the container — this would boot the networking and get the IP address automatically for us.
Stop Docker from (blindly) assigning IP addresses when we specify the
-b=BRIDGEflag. Docker currently assumes that it manages the container network and nothing else is allowed to do so. I hope this will change in the future.