This seemed like a good idea, as VXLAN has been in the Linux kernel since 3.7. TL;DR – this doesn’t work as I’d hoped. The two major issues being:

1. VXLAN needs a multicast enabled network, which rules out most public clouds.
2. Instability – I’ve managed to provoke multiple kernel panics on stock Ubuntu 14.04.

Background

As Docker deployments outgrow a single machine it can make sense to join container networks together. Jérôme Petazzoni covers the basics of using Open vSwitch in his documentation for pipework, and Marek Goldmann goes further with a worked example of Connecting Docker containers on multiple hosts.

What I did

Setting up VXLAN

Alexander Papantonatos posted last year on VXLAN on Linux (Debian). Using Ubuntu 14.04 most of the preamble stuff isn’t necessary, as the right kernel modules and a recent iproute2 are already present, so I was able to get right on with configuring interfaces and bringing them up:

sudo ip link add vxlan10 type vxlan id 10 group 239.0.0.10 ttl 4 dev eth1
sudo ip addr add 192.168.1.1/24 broadcast 192.168.1.255 dev vxlan10
sudo ifconfig vxlan10 up

I went through a similar process and assigned 192.168.1.2 to a second host, and confirmed that it was pingable.

Connecting the Docker network to the VXLAN interface

Using Marek’s Open vSwitch script as a template I ran through the following steps (after installing the bridge-utils package[1] and Docker):

sudo ip link set docker0 down
sudo brctl delbr docker0
sudo brctl addbr docker0
sudo ip a add 172.16.42.1 dev docker0
sudo ip link set docker0 up
sudo brctl addif docker0 vxlan10

After repeating with a different IP on the second host I tried to ping the docker0 IPs between hosts, which didn’t work. I tried the IPs assigned to the vxlan10 interfaces, which were no longer working. I tried deleting the docker0 bridges and starting over, and that’s when the kernel panics started. I’m now at the point where as soon as I try to use the VXLAN network between VMs one of them blows up :( It seems that I was lucky that the original ping test worked. On subsequent attempts (including rebuilds) I’ve been able to provoke kernel panic as soon as VXLAN is brought up on the second host.

Conclusion

I don’t think VXLAN is fit for this purpose. Even if it was stable it wouldn’t work in public cloud networks.

Please comment

What did I get wrong here? If I’m doing something stupid to provoke those kernel panics then I’d love to hear about it.

Note

[1] sudo apt-get install -y bridge-utils