Connecting Docker containers between VMs with VXLAN
I wrote a few days ago about my first failed attempt to do this.
After some perseverance, and with some lessons learned along the way I’m pleased to say that I now have it working.
Given that VXLAN (at least in the Linux kernel implementation) needs multicast I’m still not sure that this is a good idea, as it won’t work in (almost every one of the) public clouds.
Stability at last
The main thing that stopped me on my first try was repeated kernel panics when connecting a couple of VMs together over VXLAN.
I was using stock Ubuntu 14.04, with a 3.13.0-24 kernel and iproute2-ss131122 – a configuration that was unusable.
Going back to 12.04 with the trusty backport kernel 3.13.0-27 and the latest iproute2-ss140411  seems to give me a stable platform for experimentation.
Getting it going
First take down and delete the existing docker0 bridge:
sudo ifconfig docker0 down sudo brctl delbr docker0
Then create a new bridge (using the MAC address of the old one), and give it an IP:
sudo brctl addbr docker0 sudo ip link set docker0 address 56:84:7a:fe:97:99 sudo ip address add 172.17.42.1/16 dev docker0
Add the VXLAN adaptor and give it a MAC address:
sudo ip link add vxlan10 type vxlan id 10 group 126.96.36.199 ttl 4 dev eth1 sudo ip link set vxlan10 address 54:8:20:0:0:1
Then plug the VXLAN adaptor into the bridge and bring both of them up:
sudo brctl addif docker0 vxlan10 sudo ifconfig vxlan10 up sudo ifconfig docker0 up
The process then needs to be repeated on a second machine, taking care to change MAC and IP addresses to avoid conflicts. I used 56:84:7a:fe:97:9a and 172.17.42.2 for docker0 and 54:8:20:0:0:2 for vxlan10.
With that done I could ping between machines on their 172.17.42.x addresses
I already had a container running Node-RED on the second machine, which I (re)attached to the docker0 bridge using:
sudo brctl addif docker0 vethb909
I could then ping/wget stuff from it on its IP of 172.17.0.2
A new container that I brought up on the first VM was similarly reachable from the second VM at 172.17.0.3
IP assignment remains a problem
Just as with Marek Goldmann’s bridging using Open vSwitch it’s still necessary to do something to manage the container IPs, and I have nothing to add to his recommendations. I’m sure it’s just a matter of time before people come up with good orchestration mechanisms and DHCP that works across machines.
Something is badly wrong with VXLAN in Ubuntu 14.04.
Using a working VXLAN implementation it is possible to connect together containers across multiple VMs :)
1. I followed Alexander Papantonatos’s guide for building iproute2, but went for the latest version (3.14 at the time of writing).
2. I’m linking to the Google Cache version as at the time of writing the link is dead on the Docker.io docs (which seem to be having a major overhaul – perhaps Docker will go 1.0 at DockerCon tomorrow?).
Filed under: Docker, networking | 3 Comments
Tags: 14.04, bridge, Docker, iproute2, multicast, network, open vswitch, tunnel, Ubuntu, vxlan