I was helping a colleague troubleshoot a deployment issue recently. He’d set up a virtual private cloud (VPC) in Amazon with a public subnet and a bunch of private subnets:

• 10.0.0.0/16 – VPC (the default)
• 10.0.0.0/24 – Public subnet
• 10.0.0.1/24 – Private subnet 1
• 10.0.0.2/24 – Private subnet 2
• 10.0.0.3/24 – Private subnet 3

Everything was behaving itself in the first two private subnets, but nothing in the third one was reachable from the hosts in the public subnet. After lots of fine toothed combing through the AWS config we took a look at the routing table on a host in the public subnet:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-10-0-0-1.eu- 0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth0
10.0.3.0        *               255.255.255.0   U     0      0        0 lxcbr0
172.16.10.0     *               255.255.255.0   U     0      0        0 docker0
172.31.0.0      m1              255.255.0.0     UG    0      0        0 tun0
172.31.0.0      *               255.255.0.0     U     0      0        0 tun0
192.0.2.0       192.0.2.2       255.255.255.248 UG    0      0        0 tun1
192.0.2.2       *               255.255.255.255 UH    0      0        0 tun1
192.0.2.8       192.0.2.10      255.255.255.248 UG    0      0        0 tun3
192.0.2.10      *               255.255.255.255 UH    0      0        0 tun3
192.0.2.254     *               255.255.255.254 U     0      0        0 eth0
224.0.0.0       m1              240.0.0.0       UG    0      0        0 tun0

The problem here is a conflict between 10.0.0.3/24 configured as a VPC subnet, and another 10.0.0.3/24 configured on the host for lxcbr0 – a relic from an early Docker installation[1] that used LXC (and allowed it to install its default bridge)[2]. We worked around this by creating a 10.0.0.4/24 instead – an easy fix – this time.

You won’t be so lucky with VPC Peering

Amazon announced the availability of VPC Peering a little while ago. It’s a pretty cool feature, and it’s worth taking a look at the peering guide, and the recent Evolving VPC Design presentation from the London AWS Summit for more details. There is one key part that needs to be called out:

The VPCs to be peered must have non-overlapping CIDR blocks.

That means that if you’re using the default 10.0.0.0/16 then you can’t peer with anybody else (or indeed any of your own networks in other accounts) using that default, which right now is pretty nearly everything.

Amazingly it is possible to peer together overlapping CIDRs indirectly, so I can join my 10.1.0.0/16 network A to a number of 10.0.0.0/16 networks (B,C etc.); but there’s a couple of catches: firstly peering isn’t transitive (B can’t talk to C), and secondly I can have a subnet of A connected to B, and a different subnet of A connected to C, but not the same subnet in A connected to B and C[3].

Recommendations

It’s easy to say plan your network carefully, but that’s a bit like saying plan your family carefully. Circumstances change. Networks grow organically. Things have to be joined together later because company X bought competitor Y (which seemed unimaginable to both of their network engineering teams).

1. Avoid the defaults – using 10.0.0.0/16 for a VPC will pretty much guarantee trouble with peering.
2. Don’t use the next range along – 10.1.0.0/16 is only one better than the default; 10.254.0.0/16 isn’t 254 times better. There are 253 other networks to play with (just in the RFC1918 class A), and picking one at random is likely to be a good strategy.
3. Use smaller networks – a /16 is a LOT of IPs. The smaller the network the less chance of collision with another.

Notes

[1] Since version 0.9 Docker doesn’t actually use LXC any more by default, preferring its own native libcontainer.
[2] If you’ve been playing around with Docker since the early days and you have these bridges laying around then they can be removed using:

sudo ifconfig lxcbr0 down && sudo brctl delbr lxcbr0

[3] This kind of thing comes up all the time when making VPN connections in partner networks where it’s only a matter of time before overlapping RFC1918 ranges come along (usually 10.0.0.0/8 or 192.168.0.0/24 or close cousins). It is actually possible to deal with situations like this using source network address translation (SNAT) and network mapping (where one range gets mapped to another to avoid conflict). This is something we’ve supported in VNS3 for a little while.

This post originally appeared on the CohesiveFT blog as part of the Ask a Cloud Networking Expert series.