Almost two weeks later than planned, and after almost three days without an Internet connection to the house it seems like I finally have a working Fibre To The Premises (FTTP) install. Phew! This is the story of how it took so long, and the regulatory failure that underpins the sorry state of affairs.

Background

My house was built in 2002. It’s utterly ridiculous that I didn’t get FTTP when I moved in, but as former British Telecom (BT) Chief Technology Officer Dr Peter Cochrane recounts in “How Thatcher killed the UK’s superfast broadband before it even existed” there’s an explanation for why FTTP wasn’t a thing in the UK 21 years ago.

At the time the best on offer was Asynchronous Digital Subscriber Line (ADSL), a service based on tech that BT originally developed to deliver on demand TV services over it’s copper telephone network. That got me 2Mbps, and it wasn’t long before ADSL2 came along and the speed bumped up to 8Mbps[1].

A decade or so rolled by, and along came Fibre To The Cabinet (FTTC) services based on Very-high-speed Digital Subscriber Line (VDSL). I switched to Plusnet (which is a BT subsidiary) as my provider, and they pretty reliably delivered 80Mbps download (~12Mbps up). This has been good enough for the 10+ years I’ve been using it, and when ‘full fibre’ services started to roll out I didn’t feel much pressure to upgrade, especially given the premium.

That cost equation changed though. When my contract came around for renewal this year the VDSL service was priced higher than FTTP, so it was time to change, even if that meant sacrificing the landline[2].

How things should work

The critical piece of equipment for a full fibre install is an Optical Network Termination (ONT). It’s essentially a type of Modulator/Demulator (Modem) taking flashing light from the fibre and turning it into electrical signals on an Ethernet cable that goes into a router (and turning electrical signals coming from the router into flashing light back down the fibre).

The ONT is connected to a Connectorised Block Terminal (CBT) out in the street, which makes use of a form of encryption key based multiplexing so that the fibre from many houses can be back hauled on a single fibre to the exchange.

To connect the ONT to the CBT an engineer needs to run some fibre from the ONT (inside the house) to a junction box (outside the house) and from that junction box to the CBT. In my case there was ducting for both cable runs (holding the copper telephone cable used by existing services), so it should have been very straightforward:

• Fish a couple of fibre optic cables
• Splice those cables together in the junction box
• Activate the new service[3]

“What’s Involved in an Openreach FTTP Full-Fibre Broadband Installation?” provides a much more detailed description of the process.

Engineer Visit 1 – 8 Dec – Matthew

When I was placing my upgrade order it was implied that installation would happen in a few days, but when it came to selecting a date, the earliest on offer was more than 3 weeks away – Dec 8.

When the day came around Matthew from Kelly Communications (an OpenReach contractor) arrived and set about doing both cable runs. He achieved neither. The internal duct was blocked, though he could just drill a hole through the wall to get around that. But also he couldn’t reach the CBT outside – his ‘cobra’ duct rod wasn’t getting through to the pit. He’d have to escalate the job to a specialist team with the equipment and skills to get past the block.

Clearing the internal duct

My colleague Colin (a former BT engineer) mentioned that he’d seen installers using compressed air to get cables through ducting so I gave that a go. To no avail – there was definitely a blockage. So I grabbed the vacuum cleaner and sucked out some debris (mostly bits of plaster) that likely found their way down their during the house build.

I was then able to get my metal fish line through the duct and leave a piece of string ready.

Engineers, but not for me

On Dec 14 I heard some engineers working outside, but my joy/hope was quickly quashed when they made clear they were working on my next door neighbour’s connection and didn’t have a job card to help me. I called Plusnet to find out when I could expect a visit, but didn’t get any clear answer.

Price hike

The next day (15 Dec) brought a fresh bill for my VDSL service, and as I’d run past the end of the 1yr contracted discount the price had leapt from £28.59 to £54.39. I called Plusnet customer service, and they were pretty swift in applying a £25.80 discount. I guess these things happen a LOT.

Engineer Visit 2 – 18 Dec – Josh

Josh arrived without any warning (having driven all the way from Kent), and was friendly and eager to get on with things. He quickly confirmed that he too couldn’t get from my house to the CBT, and went off to get a camera from a colleague to help navigate the blockage.

A little while later Josh figured out what was going wrong. The trunk from my house goes to a pit to the left of my front door, not to the right where the CBT is. The cable had to be run to pit (left) before going through another trunk to pit (right) where it could be connected.

Sadly the pull string I’d run broke when Josh tried to use it. But we worked together to fish the internal cable, and it wasn’t long until he had the ONT hooked up.

There was a problem though. Josh had only been given the job to get the cable connected. He wasn’t able to activate the service. Josh left assuring me that another engineer would be along ‘in a few days’ to finish things off, and that my VDSL would remain active until then.

Cut off

About 30m after Josh left my VDSL broadband stopped working. I called Plusnet, was quickly escalated to Michael in tier 2 support, and assured that my new service would be activated in the next few hours.

Overnight I got an email ‘Your broadband product change is now complete’, but the new service wasn’t working.

I called Plusnet again, and was told that another engineer visit would be needed, and it would be ‘in a few days’. Meanwhile it was suggested that I could use a ‘BT Hotspot’ to get on the Internet, and I was told about the compensation rates for days without service and missed engineer visits. Later that day I got a text:

We have identified an issue which requires an engineer to attend your property. We have booked the appointment for the engineer to arrive between 08 AM and 01 PM on 21/12/2023

Jury rig

I connected a GL.iNet travel router to an EE Hotspot[4] and then used that as the Internet source for my home router. It worked surprisingly well, showing ~10Mbps down and 20Mbps up on a speed test.

The main annoyance was getting through the EE captive portal to pay and then sign in, which had to be repeated each time a session was timed out :(

Engineer Visit 3 – 21 Dec – Mark

Mark arrived shortly after 8 and immediately saw that the ONT hadn’t been activated.

He was unable to activate it himself with the app on his phone, and had to call a help desk in India who took around 15m to manually activate the service; leaving us both to enjoy hold music.

At that stage things were looking good. The PON light was solid green on the ONT and the multi-colour light on the Plusnet ‘Hub 2’ router was blue indicating a connection. I did a quick speed test with my laptop (over WiFi) and ~400Mbps down and 80Mbps up looked pretty respectable.

Mark wasn’t happy with the test results he was seeing though, and eventually set about checking/cleaning all the connectors. After redoing the splice in the wall junction box line levels were within the acceptable range.

I put the Hub 2 back in its box where it belongs and connected the ONT to my Draytek 2866.

How this could have gone better

I’ll pick some specifics from my experience that will lead into some more general points:

• Matthew should have known which pit he was trying to get to.
• The guys doing next door should have been able to grab my job out of the queue and save one of their colleagues another trip.
• Josh should have been tasked with completing the job, not just doing 90% of it.
• The lack of ONT activation should have prevented the changeover from VDSL to FTTP that cut me off.
• ONT activation could have been handled without Mark’s visit.
• Mark shouldn’t have needed to escalate to the team in India.

The Goal

My DevOps friends will probably know where I’m going here…

I asked one of the engineers how often things just work as they should. I expected the answer to be 50-60%. OpenReach have a well deserved reputation for poor service, but surely things work like they should some of the time. His answer shocked me – ‘never’. He said he’d never seen a ‘textbook’ installation.

The disaster that unfolded over the past weeks bears many of the same hallmarks as my last encounter with OpenReach. As an organisation it seems that their processes and the systems that implement them are fragile, siloed and have poor escalation paths – even for mainstream stuff like FTTP. This is hugely wasteful of precious engineer resource, which leads to customer frustration and poor profitability.

It seems that the regulator OFCOM’s response to this is to allow competition on the ‘last mile’, which should be a natural monopoly. So in addition to OpenReach’s fibre I now have fibre from a bunch of other companies running past my front door. This is a disaster for consumers, as ultimately we’ll all pay for that duplication in higher service charges over the coming years.

Rather than neoliberal ideology of solving problems with ‘markets’ and ‘competition’ OFCOM should have straightened out OpenReach’s disastrous mismanagement[5]. The problems described above can all be solved:

• by understanding processes end to end
• by empowering staff to do the right thing
• by supporting people with well thought through platforms
• by providing effective means to deal with exceptions when they arise

This is exactly what we did to a bunch of Delivery when I was at CSC/DXC. It’s a hard slog, but worth it. There’s no magic, we simply followed the approach described by Eli Goldratt in The Goal – identify constraints, fix the worst one, repeat.

Updates

22 Dec 2023 – the initial version of this post stated that wavelength division multiplexing (WDM) was used between the ONT and CBT, which has been corrected.

Notes

[1] At around the same time my brother in law was getting his flat in Japan connected to the Internet. His service was 1Gbps symmetric full fibre, and cheaper than my ADSL :0
[2] Plusnet don’t offer the ability to transfer a landline number to a Voice Over Internet Protocol (VOIP) service (unlike BT’s consumer service). In theory I could have picked another service, and hopefully even kept my (easy to remember) ‘home’ number. But even at a few pounds a month it’s hard to justify any cost. I stopped answering that phone many years ago as it was never for me (and usually scammers).
[3] As far as I can tell the main purpose of activation is to bind the ONT encryption key to the upstream Optical Line Terminal (OLT). This is just an entry in a database somewhere. Details of this are pretty sketchy, and I suspect that’s because there’s a bunch of security by obscurity going on. The Wikipedia article on Passive optical network provides the best overview I can find.
[4] It seems the BT Hoptspots have been re-branded to EE (and nobody’s upgraded the Plusnet support scripts).
[5] I should make clear that all of the engineers I encountered were all great – professional, understanding, friendly, and as accommodating as they could be within the constraints inflicted on them. But they work for an organisation that management have allowed to be dysfunctional.