TL;DR

Whilst on vacation in Spain I’ve found networks that seem to be like something out of a Cory Doctorow novel – domestic WiFi routers hanging off domestic WiFi routers hanging off domestic WiFi routers. At first I thought it was my Airbnb host being cheap and having a cosy arrangement with a neighbour to provide Internet, but it’s much more systematic than that.

Six routers deep

Here’s a traceroute from my laptop:

Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops:

1 3 ms 1 ms 1 ms 192.168.0.1
2 6 ms 7 ms 5 ms . [192.168.2.1]
3 9 ms 8 ms 7 ms 192.168.1.20
4 963 ms 940 ms 697 ms 192.168.10.1
5 368 ms 464 ms 159 ms homestation.Home [192.168.1.1]
6 685 ms 728 ms 769 ms 192.168.144.1
7 * * * Request timed out.
8 1580 ms 658 ms 588 ms 109.Red-80-58-106.staticIP.rima-tde.net [80.58.106.109]
9 * * * Request timed out.
10 3538 ms 2147 ms 1566 ms GOOGLE-Ae2-GRAMADNO2.red.telefonica-wholesale.net [5.53.1.74]
11 723 ms 397 ms 877 ms 216.239.50.199
12 1975 ms 1198 ms 1047 ms 216.239.50.177
13 865 ms 431 ms 425 ms google-public-dns-a.google.com [8.8.8.8]

Trace complete.

That’s six different routers on RFC1918 class C private networks (and a lot of latency) before I hit the Internet proper. It’s also a whole ton of NATing and way too much potential flakiness. On a good moment I’ve seen 2Mb/s, but in reality it seems lucky when packets get through at all, and amazing when Skype works[1].

I’ll try to unpick what’s going on at each router in turn…

Router 1

The router in the house I’m renting is my old friend the TP-Link TL-WR841N. I have one of these at home running OpenWRT, but the one here has the (awful) stock firmware on it. Luckily the admin password hasn’t been changed, which came in handy when it needed some help reconnecting after a long power cut.

The WAN link of the router is connected to a TP-Link powerline adaptor

At first I thought this was connected through to a neighbour, but that was because I was looking downstairs for an ADSL modem or similar that wasn’t there. When I looked upstairs (in the laundry room) I found its twin attached to a Ubiquiti power over ethernet coupler:

and that was for a WiFi antenna mounted on the roof:

A quick detour to Solyaires Internet

I didn’t set the system up, and I don’t pay the bill, but my research would suggest that it’s connected to Solyaires Internet or some similar system for distributing Internet via 5GHz WiFi connections. So instead of a community effort where people create a mesh network to share, this seems to be a commercial endeavour (and it’s not a mesh – more like a spider’s web).

One amusing thing I’ve noticed is that my in law’s apartment building (which is miles away from the house I’m renting) has exactly the same egress IP onto the Internet. Here’s their tracert (oddly despite fewer layers of router they get much worse bandwidth):

Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops:

1 <1 ms <1 ms 1 ms 192.168.100.1
2 1 ms 1 ms <1 ms homestation [192.168.1.1]
3 107 ms 56 ms 42 ms 192.168.144.1
4 * * * Request timed out.
5 62 ms 72 ms 102 ms 109.Red-80-58-106.staticIP.rima-tde.net [80.58.106.109]
6 * * * Request timed out.
7 160 ms 142 ms 136 ms GOOGLE-Ae2-GRAMADNO2.red.telefonica-wholesale.net [5.53.1.74]
8 67 ms 59 ms 58 ms 216.239.50.197
9 58 ms 65 ms 60 ms 209.85.254.9
10 57 ms 59 ms 59 ms google [8.8.8.8]

Trace complete.

Router 2

The second router along is a Belkin F7D1301, which judging by the Amazon reviews is a very ordinary router indeed. It has no password set, so the admin interface is wide open, which is obviously a terrible idea from a security perspective. My best guess as to what’s going on here is that the WiFi distribution outfit use some of their customers as Internet mules, acting as a relay from one point to the next. It’s pretty shocking how amateurish the setup is though.

Router 3

The third router doesn’t have an open admin interface. Looking at its response headers I see a Boa 0.93.15 web server, which could suggest a Zyxel/Edimax piece of kit (which might be a full router, or might be some sort of ‘range extender’). That web server is susceptible to a basic authentication bypass exploit, but I wasn’t feeling nefarious enough to pwn it (this was a look but don’t touch exercise). The basic auth prompt was ‘Graham-New’ so I suspect it’s a wise home user (another relay mule?) rather than something professionally configured.

Router 4

This one has an airOS admin screen implying something from Ubiquiti networks, and likely kit that’s run by an actual service provider rather than sat in somebody’s home.

Routers 5&6

Neither of these had web admin screens on ports 80 or 443 so I have less to go on (but at least they’re somewhat secure)[2].

The home.homestation implies that we’re back to consumer ADSL gear, and my best guess is that the WiFi connections are being back-hauled by a bunch of consumer grade ADSL links.

The final 192.168.x.y router might just be the local telco being awful and aggregating many ADSL connections onto one public IP.

Part of a broader broadband problem?

I asked a friend who lives and works in Spain about her experiences, and she said ‘it’s unreliable, it’s slow, and the telephone companies are from the last century’. Flicking through local papers I also see that WiFi delivery is a pretty normal offering, and priced in line with ADSL services at around €24/month.

Whilst here I’ve been lucky enough to see Spain included in Three’s ‘Feel at home‘ roaming deal, which means I’ve also been able to check out 3G service. The 3G I’m getting is pretty typical of a mobile service – when it’s good it’s OK (~1Mb/s), when it’s bad it’s not there at all.

In general I’d say that the house WiFi and 3G are about on par in terms of bandwidth and reliability – good enough for keeping up with what’s going on in the world beyond, but not so good that I’d want to depend on it for any kind of business use.

Conclusion

Something must be very wrong with the Internet connectivity market in the Costa Tropical (and perhaps Spain more generally) for this type of arrangement to be tolerable (never mind commonplace).  I’ve been visiting Almuñécar for many years now, and back in the early days the ADSL provision seemed to be much the same as back home in the UK. I get the feeling that the FTTC connection I have at home now would be considered enough to serve hundreds of properties. It’s been great to see investment in infrastructure like roads over the past decade, but it’s a shame that the technology infrastructure hasn’t had the same attention.

Notes

[1] I got so sick of large downloads from my home network failing that I’ve lashed up a combination of autossh and bittorrent so that it will download things eventually, and I don’t have to burden the network (and my mouse finger) with redoing the same failed file time and time again.
[2] I’m guessing that the homestation in the path from my in-laws’ was a different one, as in addition to being ‘homestation’ rather than ‘homestation.Home’ in the traceroute it also serves up an admin GUI over the web.