(Not) Connecting Docker containers between VMs with VXLAN
This seemed like a good idea, as VXLAN has been in the Linux kernel since 3.7. TL;DR – this doesn’t work as I’d hoped. The two major issues being:
- VXLAN needs a multicast enabled network, which rules out most public clouds.
- Instability – I’ve managed to provoke multiple kernel panics on stock Ubuntu 14.04.
Background
As Docker deployments outgrow a single machine it can make sense to join container networks together. Jérôme Petazzoni covers the basics of using Open vSwitch in his documentation for pipework, and Marek Goldmann goes further with a worked example of Connecting Docker containers on multiple hosts.
What I did
Setting up VXLAN
Alexander Papantonatos posted last year on VXLAN on Linux (Debian). Using Ubuntu 14.04 most of the preamble stuff isn’t necessary, as the right kernel modules and a recent iproute2 are already present, so I was able to get right on with configuring interfaces and bringing them up:
sudo ip link add vxlan10 type vxlan id 10 group 239.0.0.10 ttl 4 dev eth1 sudo ip addr add 192.168.1.1/24 broadcast 192.168.1.255 dev vxlan10 sudo ifconfig vxlan10 up
I went through a similar process and assigned 192.168.1.2 to a second host, and confirmed that it was pingable.
Connecting the Docker network to the VXLAN interface
Using Marek’s Open vSwitch script as a template I ran through the following steps (after installing the bridge-utils package[1] and Docker):
sudo ip link set docker0 down sudo brctl delbr docker0 sudo brctl addbr docker0 sudo ip a add 172.16.42.1 dev docker0 sudo ip link set docker0 up sudo brctl addif docker0 vxlan10
After repeating with a different IP on the second host I tried to ping the docker0 IPs between hosts, which didn’t work. I tried the IPs assigned to the vxlan10 interfaces, which were no longer working. I tried deleting the docker0 bridges and starting over, and that’s when the kernel panics started. I’m now at the point where as soon as I try to use the VXLAN network between VMs one of them blows up :( It seems that I was lucky that the original ping test worked. On subsequent attempts (including rebuilds) I’ve been able to provoke kernel panic as soon as VXLAN is brought up on the second host.
Conclusion
I don’t think VXLAN is fit for this purpose. Even if it was stable it wouldn’t work in public cloud networks.
Please comment
What did I get wrong here? If I’m doing something stupid to provoke those kernel panics then I’d love to hear about it.
Note
[1] sudo apt-get install -y bridge-utils
Filed under: Docker, networking | 2 Comments
Tags: bridge, Docker, fail, gre, iproute2, multicast, network, open vswitch, tunnel, Ubuntu, vxlan
I tried this out on Hyper-V (having previously used VirtualBox) in order to eliminate any hypervisor issues from being possible causes of the kernel panics. I got a kernel panic straight way, so my finger of blame isn’t pointing at the hypervisor.