I’ve been noticing that lots of the services I use online have been getting worse. My friends have been complaining too. I think I know why.

A/B testing is a great way for product managers to make decisions based on data (rather than their own gut feel). But what happens when A/B testing meets the classic technology adoption curve?

Rogers’ Bell Curve (Wikipedia)

As the Innovators and Early Adopters get started with something A/B testing works with them. It helps to explore the solution space, and smooth the rough edges off.

Just add moron

As the Early/Late Majority, or worse still the Laggards get on board the A/B testing starts tuning things to their preferences, which might generally be towards a simpler user experience where only the most used features are presented. Other features – the features that the Innovators and Early Adopters liked (we know this from earlier A/B testing) start to disappear, or get hidden away behind special ‘nerd knob’ selectors.

It doesn’t have to be like this

Pretty much every service lets me customise things to my preferences – stuff like text size, background colours/pictures etc. There’s absolutely no reason why users shouldn’t be able to override A/B testing when they find it taking their personal user experience in the wrong direction. Better still I expect that if A/B testing was done into adoption clades it would cause some very nice split evolution into different ecological niches. This might blow the brains of simplistic (simpleton) product managers who want to keep everything uniform, but people aren’t uniform – deal with it. The secret to good user experience is know your customer (KYC[1]) – wasn’t that the whole point of A/B testing in the first place. The point here is that some customers are different from others – get to know them and those differences and you can serve them better.

Here’s a great counter example of KYC from Barclays via Conor Ogle:

We have the technology

I was at the Andreessen Horowitz London tech summit last week and one of their portfolio companies gave a demo of customisation that they do for online retail. It was carving up customer groups by their various profiles six ways from Sunday for various online (A/B) tests, so clearly we’re at a place where this is a commodity for presenting product, so why isn’t it a commodity for presenting features.

Conclusion

A/B testing can be a great tool, but if its used too bluntly then it can actively destroy a well tuned user experience of early adopters as less demanding users follow them. Product managers need to get smarter about segmentation.

Note

[1] Yes, I know KYC has a specific regulatory meaning in banking etc., but any services organisation should be trying to know their customers better, and striving to delivery a delightful user experienced based on that knowledge.