A/B Testing Into Mediocrity


I’ve been noticing that lots of the services I use online have been getting worse. My friends have been complaining too. I think I know why.

A/B testing is a great way for product managers to make decisions based on data (rather than their own gut feel). But what happens when A/B testing meets the classic technology adoption curve?

Rogers’ Bell Curve (Wikipedia)

As the Innovators and Early Adopters get started with something A/B testing works with them. It helps to explore the solution space, and smooth the rough edges off.

Just add moron

As the Early/Late Majority, or worse still the Laggards get on board the A/B testing starts tuning things to their preferences, which might generally be towards a simpler user experience where only the most used features are presented. Other features – the features that the Innovators and Early Adopters liked (we know this from earlier A/B testing) start to disappear, or get hidden away behind special ‘nerd knob’ selectors.

It doesn’t have to be like this

Pretty much every service lets me customise things to my preferences – stuff like text size, background colours/pictures etc. There’s absolutely no reason why users shouldn’t be able to override A/B testing when they find it taking their personal user experience in the wrong direction. Better still I expect that if A/B testing was done into adoption clades it would cause some very nice split evolution into different ecological niches. This might blow the brains of simplistic (simpleton) product managers who want to keep everything uniform, but people aren’t uniform – deal with it. The secret to good user experience is know your customer (KYC[1]) – wasn’t that the whole point of A/B testing in the first place. The point here is that some customers are different from others – get to know them and those differences and you can serve them better.

Here’s a great counter example of KYC from Barclays via Conor Ogle:

We have the technology

I was at the Andreessen Horowitz London tech summit last week and one of their portfolio companies gave a demo of customisation that they do for online retail. It was carving up customer groups by their various profiles six ways from Sunday for various online (A/B) tests, so clearly we’re at a place where this is a commodity for presenting product, so why isn’t it a commodity for presenting features.


A/B testing can be a great tool, but if its used too bluntly then it can actively destroy a well tuned user experience of early adopters as less demanding users follow them. Product managers need to get smarter about segmentation.


[1] Yes, I know KYC has a specific regulatory meaning in banking etc., but any services organisation should be trying to know their customers better, and striving to delivery a delightful user experienced based on that knowledge.

One Response to “A/B Testing Into Mediocrity”

  1. 1 The DXC Blogs – What Would Google Do (WWGD)? | Chris Swan's Weblog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: