Connecting Docker containers between VMs with VXLAN


I wrote a few days ago about my first failed attempt to do this.

After some perseverance, and with some lessons learned along the way I’m pleased to say that I now have it working.


Given that VXLAN (at least in the Linux kernel implementation) needs multicast I’m still not sure that this is a good idea, as it won’t work in (almost every one of the) public clouds.

Stability at last

The main thing that stopped me on my first try was repeated kernel panics when connecting a couple of VMs together over VXLAN.

I was using stock Ubuntu 14.04, with a 3.13.0-24 kernel and iproute2-ss131122 – a configuration that was unusable.

Going back to 12.04 with the trusty backport kernel 3.13.0-27 and the latest iproute2-ss140411 [1] seems to give me a stable platform for experimentation.

Getting it going

First take down and delete the existing docker0 bridge:

sudo ifconfig docker0 down
sudo brctl delbr docker0

Then create a new bridge (using the MAC address of the old one), and give it an IP:

sudo brctl addbr docker0
sudo ip link set docker0 address 56:84:7a:fe:97:99
sudo ip address add dev docker0

Add the VXLAN adaptor and give it a MAC address:

sudo ip link add vxlan10 type vxlan id 10 group ttl 4 dev eth1
sudo ip link set vxlan10 address 54:8:20:0:0:1

Then plug the VXLAN adaptor into the bridge and bring both of them up:

sudo brctl addif docker0 vxlan10
sudo ifconfig vxlan10 up
sudo ifconfig docker0 up

The process then needs to be repeated on a second machine, taking care to change MAC and IP addresses to avoid conflicts. I used 56:84:7a:fe:97:9a and for docker0 and 54:8:20:0:0:2 for vxlan10.

With that done I could ping between machines on their 172.17.42.x addresses

Connecting containers

I already had a container running Node-RED on the second machine, which I (re)attached to the docker0 bridge using:

sudo brctl addif docker0 vethb909

I could then ping/wget stuff from it on its IP of

A new container that I brought up on the first VM was similarly reachable from the second VM at

IP assignment remains a problem

Just as with Marek Goldmann’s bridging using Open vSwitch it’s still necessary to do something to manage the container IPs, and I have nothing to add to his recommendations. I’m sure it’s just a matter of time before people come up with good orchestration mechanisms and DHCP that works across machines.

Recommended reading

I hadn’t previously found the full Docker network configuration documentation, but it’s very good (and it’s a shame that it’s not linked from the ‘Advanced Docker Networking‘ documentation’ [2]).


Something is badly wrong with VXLAN in Ubuntu 14.04.

Using a working VXLAN implementation it is possible to connect together containers across multiple VMs :)


1. I followed Alexander Papantonatos’s guide for building iproute2, but went for the latest version (3.14 at the time of writing).
2. I’m linking to the Google Cache version as at the time of writing the link is dead on the docs (which seem to be having a major overhaul – perhaps Docker will go 1.0 at DockerCon tomorrow?).

3 Responses to “Connecting Docker containers between VMs with VXLAN”

  1. Chris,

    Thanks for sharing how you figured out connecting containers using VXLAN. This is definitely good info.

    Have you considered using L2TPv3? This should work with any public IaaS if you use UDP encapsulation rather than directly over IP. NAT doesn’t play nice with the latter.

  2. Cumulus Network and MetaCloud released VXFLD as open source last year. VXFLD is intended to solve the BUM (broadcast, unknown, multicast) problem that usually required MultiCast support for the VxLAN.
    VXFLD enables VXLAN with just uni-cast.


    read the .RST files as they document both VXSND and VXRD daemon’s and their use

  1. 1 Technology Short Take #44 - - The weblog of an IT pro specializing in virtualization, networking, cloud, servers, & Macs

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: