I wrote a few days ago about my first failed attempt to do this.

After some perseverance, and with some lessons learned along the way I’m pleased to say that I now have it working.

Given that VXLAN (at least in the Linux kernel implementation) needs multicast I’m still not sure that this is a good idea, as it won’t work in (almost every one of the) public clouds.

Stability at last

The main thing that stopped me on my first try was repeated kernel panics when connecting a couple of VMs together over VXLAN.

I was using stock Ubuntu 14.04, with a 3.13.0-24 kernel and iproute2-ss131122 – a configuration that was unusable.

Going back to 12.04 with the trusty backport kernel 3.13.0-27 and the latest iproute2-ss140411 [1] seems to give me a stable platform for experimentation.

Getting it going

First take down and delete the existing docker0 bridge:

sudo ifconfig docker0 down
sudo brctl delbr docker0

Then create a new bridge (using the MAC address of the old one), and give it an IP:

sudo brctl addbr docker0
sudo ip link set docker0 address 56:84:7a:fe:97:99
sudo ip address add 172.17.42.1/16 dev docker0

Add the VXLAN adaptor and give it a MAC address:

sudo ip link add vxlan10 type vxlan id 10 group 239.0.0.10 ttl 4 dev eth1
sudo ip link set vxlan10 address 54:8:20:0:0:1

Then plug the VXLAN adaptor into the bridge and bring both of them up:

sudo brctl addif docker0 vxlan10
sudo ifconfig vxlan10 up
sudo ifconfig docker0 up

The process then needs to be repeated on a second machine, taking care to change MAC and IP addresses to avoid conflicts. I used 56:84:7a:fe:97:9a and 172.17.42.2 for docker0 and 54:8:20:0:0:2 for vxlan10.

With that done I could ping between machines on their 172.17.42.x addresses

Connecting containers

I already had a container running Node-RED on the second machine, which I (re)attached to the docker0 bridge using:

sudo brctl addif docker0 vethb909

I could then ping/wget stuff from it on its IP of 172.17.0.2

A new container that I brought up on the first VM was similarly reachable from the second VM at 172.17.0.3

IP assignment remains a problem

Just as with Marek Goldmann’s bridging using Open vSwitch it’s still necessary to do something to manage the container IPs, and I have nothing to add to his recommendations. I’m sure it’s just a matter of time before people come up with good orchestration mechanisms and DHCP that works across machines.

Recommended reading

I hadn’t previously found the full Docker network configuration documentation, but it’s very good (and it’s a shame that it’s not linked from the ‘Advanced Docker Networking‘ documentation’ [2]).

Conclusion

Something is badly wrong with VXLAN in Ubuntu 14.04.

Using a working VXLAN implementation it is possible to connect together containers across multiple VMs :)

Notes

1. I followed Alexander Papantonatos’s guide for building iproute2, but went for the latest version (3.14 at the time of writing).
2. I’m linking to the Google Cache version as at the time of writing the link is dead on the Docker.io docs (which seem to be having a major overhaul – perhaps Docker will go 1.0 at DockerCon tomorrow?).