Understanding the A/B Testing Market

Zarget got acquired by Freshworks back in August 2017, and I was meaning to write about the internals of how the A/B testing market functions. Here are my thoughts and learnings:

1 – A/B testing doesn’t work for those who have low traffic. When VWO/Optimizely started out in 2010, one could survive by serving the SMB market. In 2017, there are far too many competitors and this isn’t possible anymore. For a product to do well, it has to go after customers who have non-trivial traffic and budget to pay for an enterprise testing vendor.

2 – A/B testing products on their own will always have high churn. Testing requires time, technical help, marketing/psychology skills, design skills, meaningful traffic, and yet might not give you the wins that make it worthwhile. In such cases, it becomes difficult to prove the direct RoI of the tool investment. This situation becomes 10x worse for SMBs who don’t have the traffic or the tech/design resources, therefore the high churn in this segment.

3 – Adwords is an unprofitable channel for an SMB focused A/B testing tool. Enterprise competitors and agencies (who charge ~$10,000 per month) have driven up the CPC bids, and SMB customers anyway churn quickly. These two factors mean one cannot rely on Adwords to profitably acquire small, $49 – $99 per month customers.

4 – Another reason for high churn is that Javascript based tools are easy to remove or replace. They’re usually a script that sits just after the < head > tag. Don’t like it, just delete that line. Nothing breaks, and no process is disrupted.

5 – On the other hand, they’re also difficult to place on a website because you can’t use a tag manager to insert the testing tool’s JS. It has to be the first one to load, and therefore is placed before the tag manager’s JS. I’ve heard first hand accounts of marketers who’ve walked up to their IT teams asking to place a JS tag on the website, and been told to fill up a “Business Impact” form, so that this tag can be considered next quarter!

6 – The best market to go after are companies that have a conversion rate optimization (CRO) team in place. A CRO team or senior testing leader means the company is committed to conversion optimization and will run it as a process. Ideally, the testing tool wants to be the one that helps the company run and manage the process. Or, at least be an integral part of the process.

7 – The best customers for an A/B testing tool are those who think of A/B testing as a method of verifying their decisions, and not as a method of finding ‘testing wins’. These customers don’t just go after the ‘wins’, instead they are also happy when a losing test result helps them avoid a bad decision.

8 – Optimizely was probably the smartest A/B testing tool. Their good decisions include:

  • moving upmarket to serve enterprises, though this is a natural progression of most martech vendors
  • having a server side testing product that is difficult to install, but also very difficult to remove
  • giving out a basic free plan which made it the default choice for beginners and SMBs
  • asking agencies to connect them directly to the client (the end buyer of the product), and then owning that relationship, vs. depending on the agency to champion Optimizely to the client
  • adding personalization to their product suite (see next point)

9 – Personalization is a good space to be in, because of two reasons:

  • the average internet buyer today expects a more personal experience when compared to a few years ago. And it is claimed that personalization increases conversion rates. Therefore, most brands today implement some form of personalization on their ecommerce stores.
  • a personalization product takes time to implement, but once in place, it upholds the visitor’s website experience… unplanned removal of the product frequently breaks the website and the customer’s experience, due to which most orgs don’t remove it easily.