Understanding the A/B Testing Market

Zarget got acquired by Freshworks back in August 2017, and I was meaning to write about the internals of how the A/B testing market functions. Here are my thoughts and learnings:

1 – A/B testing doesn’t work for those who have low traffic. When VWO/Optimizely started out in 2010, one could survive by serving the SMB market. In 2017, there are far too many competitors and this isn’t possible anymore. For a product to do well, it has to go after customers who have non-trivial traffic and budget to pay for an enterprise testing vendor.

2 – A/B testing products on their own will always have high churn. Testing requires time, technical help, marketing/psychology skills, design skills, meaningful traffic, and yet might not give you the wins that make it worthwhile. In such cases, it becomes difficult to prove the direct RoI of the tool investment. This situation becomes 10x worse for SMBs who don’t have the traffic or the tech/design resources, therefore the high churn in this segment.

3 – Adwords is an unprofitable channel for an SMB focused A/B testing tool. Enterprise competitors and agencies (who charge ~$10,000 per month) have driven up the CPC bids, and SMB customers anyway churn quickly. These two factors mean one cannot rely on Adwords to profitably acquire small, $49 – $99 per month customers.

4 – Another reason for high churn is that Javascript based tools are easy to remove or replace. They’re usually a script that sits just after the < head > tag. Don’t like it, just delete that line. Nothing breaks, and no process is disrupted.

5 – On the other hand, they’re also difficult to place on a website because you can’t use a tag manager to insert the testing tool’s JS. It has to be the first one to load, and therefore is placed before the tag manager’s JS. I’ve heard first hand accounts of marketers who’ve walked up to their IT teams asking to place a JS tag on the website, and been told to fill up a “Business Impact” form, so that this tag can be considered next quarter!

6 – The best market to go after are companies that have a conversion rate optimization (CRO) team in place. A CRO team or senior testing leader means the company is committed to conversion optimization and will run it as a process. Ideally, the testing tool wants to be the one that helps the company run and manage the process. Or, at least be an integral part of the process.

7 – The best customers for an A/B testing tool are those who think of A/B testing as a method of verifying their decisions, and not as a method of finding ‘testing wins’. These customers don’t just go after the ‘wins’, instead they are also happy when a losing test result helps them avoid a bad decision.

8 – Optimizely was probably the smartest A/B testing tool. Their good decisions include:

  • moving upmarket to serve enterprises, though this is a natural progression of most martech vendors
  • having a server side testing product that is difficult to install, but also very difficult to remove
  • giving out a basic free plan which made it the default choice for beginners and SMBs
  • asking agencies to connect them directly to the client (the end buyer of the product), and then owning that relationship, vs. depending on the agency to champion Optimizely to the client
  • adding personalization to their product suite (see next point)

9 – Personalization is a good space to be in, because of two reasons:

  • the average internet buyer today expects a more personal experience when compared to a few years ago. And it is claimed that personalization increases conversion rates. Therefore, most brands today implement some form of personalization on their ecommerce stores.
  • a personalization product takes time to implement, but once in place, it upholds the visitor’s website experience… unplanned removal of the product frequently breaks the website and the customer’s experience, due to which most orgs don’t remove it easily.

Unsolicited Career Advice For Tech Marketers

TL;DR be a business person doing marketing, not a marketer doing marketing


When marketers ask me for advice about what they should do to further their careers, they ask about the skills they should acquire, and the functional areas they should specialize in.

Mostly, they receive advice from others that they should be a T-shaped marketer, i.e. have a broad set of marketing skills where they’re decently good, with a couple areas where they have deep experience and knowledge. I agree and propound this view. Over a period of 30 to 40 years of a professional career, functional/technical expertise is one of your best bets to grow. There’s research to indicate that reportees prefer bosses who could easily do their job, and Andy Grove, the legendary founder and CEO of Intel built a company where he fostered “knowledge power” over “position power”, something that you see all across the flat hierarchies in today’s tech startups.

However, I also think that most marketers end up becoming “just marketers”, and the same happens to folks in other departments who dedicate themselves to only their craft. They end up being just sales people, or just customer success people, frequently capping their growth to the Director or VP of their respective functions.

Instead, I recommend you become a businessperson who is currently handling marketing. By a businessperson I mean you understand your entire business deeply, you understand the different levers that make it what it is, the customer behavior, the delivery of value, the metrics, the industry, what keeps this entity up and ticking, and finally, you understand how your function contributes to it all.

I say this because I see marketers speak in meetings with leadership, and their ideas are not accepted, they’re not given the importance they feel they deserve, or they’re just a part of the side conversation. While the CEO is busy thinking about big-picture stuff like cash flow, burn rate, inside sales process, efficiency, competition, product strategy, the marketer limits himself by capping his thinking at “how do I increase my budget”, or “what if we tried to do more social media”.

Marketers are taught to understand their target audience, but most of them repeatedly fail to understand what is it that’s bothering their CEO. And if you want a seat at the boardroom sometime in the future, you have to start talking their language today.

Advice on how to become a ‘businessperson’

Start by understanding the big picture, and then get an excellent grasp on how that drills down to the day-to-day activities performed by your organization. Basically, understand the levers that run your business. This will necessitate that you understand some key metrics, and once you do, you’ll be able to quickly size up any business in your industry if you get to know their key metrics.

For example, if you know that a SaaS startup has an Annual Contract Value (ACV) of USD 3000, then you should be able to guess that an inside sales team doesn’t make sense for them, churn is an important number to contain, and most of their growth will be coming from new business being added every month, instead of upsells.

Similarly, if a firm has an ACV of USD 200,000, then they’re likely to have a field sales team, extremely strong account management, and a large part of their growth, if not most of it, will be from account expansion (getting existing customers to pay more over time).

Ask yourself why, and what form of marketing is needed today.

If it’s a bootstrapped SaaS, then you should be able to infer that the CEO won’t spend heavily on paid marketing channels, because she doesn’t have that kind of money. Instead, she’ll focus on low-cost inbound marketing, and maybe outbound email/phone prospecting. For her, profitability is more important than outright growth.

If it’s a consumer business that wants to grow fast, then they better have a marketing war chest to spend, or incredible word-of-mouth virality. Aside: differences between B2C and B2B marketing.

If the business is heavily VC funded, then it is basically a financial product, not for you or me, but for the VCs who invested in it. Everyone’s job is to get it to an exit event in a defined(ish) timeline. That event is heavily dependent on growth, so expect a lot of budget for marketing.

If it’s an enterprise play, then marketing’s job is to support Sales with full-funnel sales collateral, product marketing (testimonials, case studies, whitepapers, spec sheets, knowledgebase), and make an impact at events where your target decision makers congregate. Essentially, you want your very specific potential customers to know you, trust you enough to give Sales an opportunity to talk to them, and then be able to evangelize you internally to various stakeholders. Aside: read this interesting post by Mike Volpe on difference between Enterprise and SMB CMOs.

Essentially, when you look at a given situation, you should be able to guess if marketing is needed at all, and if yes, then what kind and form of marketing.

Understand the business you’re in as a system

Systems thinking is a management discipline that concerns an understanding of a system by examining the linkages and interactions between the components that comprise the entirety of that defined system.

Source

All businesses are complex systems, and these systems interact with each and other, and impact each other. As business people, it’s important that we marketers understand how these different systems affect each other, and whether things need correction or not.

1) A SaaS started out with a basic product that mostly catered to SMB customers. Over the years, competitors came in so they kept adding new features to stay ahead of the curve. So many features in fact, that the product isn’t meant for SMB customers anymore who prefer simple, easy-to-use tools. It is now better suited to larger customers with more detailed needs.

However, the brand, website, copy and collateral still retain that SMB positioning. Sales keeps trying to talk to enterprise prospects, but the conversion rate isn’t great, or the ACV isn’t as large as it could be. As a ‘businessperson’ marketer, you should be able to understand and rectify this mismatch.

2) Sales is incentivized to develop a new territory, but no marketing budget or team has been assigned to that new territory. Soon enough, Sales complains that it isn’t going as well as they hoped, and they need geography specific collateral and programs to generate demand.

Similar alignment issues crop up multiple times during lifetime of a business, and marketers must see the big picture, figure out how everything comes together, and push towards that.

Become friends with your head of Finance

When Wingify first hired a person to head up Finance, I thought ok, we probably need this role, but I wasn’t sure what value they’d provide. This was a classic “If you don’t think you need it, you haven’t seen greatness” moment for me. I’ve learnt tons from him since then about financial metrics, what they mean, and the underlying stories they tell. Understanding these metrics is one of the best ways to get a grasp the whole business.

While finance execs are hired a little later in the life of a company, if you have one in yours, I recommend to at least have monthly one-on-one’s with them to discuss the latest tech IPO S-1 and comparing those with others tech businesses. Or, you can get a headstart on this by going through Tom Tunguz’s benchmarking of 7 key SaaS metrics for various companies that have gone public.


If this is good stuff, I’d certainly appreciate you sharing it with your network:


[mc4wp_form id=”1168″]

Create new communication channels

Want to create an incredibly successful startup and make the world a better place in the process? Create a communication channel for businesses to reach their potential customers, and they’ll pay you handsomely for it.

Your approach to creating the communication channel doesn’t matter as long as you acquire a large enough user base with some spending power. There’s only one caveat: the users have to stick.

Let me explain what I mean with examples.

Facebook

Facebook did this by creating a social network where it gave people the power to share and make the world more open and connected. Then it built an ad platform so that businesses could reach out to all these potential customers. The hard part was and still is acquiring and keeping people on the platform because the day they leave, so will the businesses, and so will the revenue.

Based on their Q4 2015 and Q1 2016 results, Facebook is doing a damn good job of keeping consumers on the network. Where it’s stumbling is getting new users from the “Rest of the World” category (the developing world). And they need new users to continue their growth rate. The desperation with which they need these users was evident in how they went bat-shit crazy trying to defend Free Basics (Internet.org) in India. Read more about that on link 1, link 2, link 3 and link 4.

Instagram

Instagram is interesting. They acquired users but didn’t give businesses access to these users for a long time. Smart brands quickly found a way to still get to consumers through some women with large followers. Instagram is making no revenue out of that and recently launched the Instagram for Business service.

Youtube

Youtube is slightly different from the pack because others are responsible for creating the content that brings in users, but they share some of the revenue with original content creators who attract and keep potential customers on the platform.

B2B industry news websites

Take for example iEntry, which owns WebProNews. They churn out “technology news” and somehow convince people to visit their website and signup for their email newsletter. Then these two channels are provided to businesses to advertise to their potential customers. Almost every B2B niche has its own news website where others can advertise.

Chrome push notifications

Google was concerned that unlike native apps, websites had no way of engaging mobile users. It responded by allowing websites to send push notifications through the Chrome browser installed on user’s Android phones. Firefox soon followed suit.

While Google was responding to a different threat, some smart people immediately figured that a new channel for businesses to talk to customers had just opened up. And now you have a whole bunch of SaaS products (25 as on date of publishing) that basically bring marketing automation capabilities to website push notifications.

This channel is similar to white-hat email marketing, where businesses have to build their own personal email database.

Google Search

The largest “business to potential customer” advertising channels because of one reason: relevance. Think of every search query as a person looking for an answer to a problem. Google lets businesses bid for access and priority in the advertising channel with appropriate measures in the system to promote relevance.

Chat bots

Facebook’s big announcement at its annual F8 Developer conference, was to make Messenger, its 900-million-user messaging app into a full-fledged platform that allows businesses to communicate with users via chatbots.” – 3 trends driving the chatbot revolution, VentureBeat

How to search for flights with Skyscanner’s new Facebook Messenger bot – Skyscanner

Wrapping up

The hard part is getting and keeping users. Businesses will happily come after you’ve acquired users.

Or you could simply provide the means to the communication and let businesses build their own audiences… like website push notifications or email marketing software.

Think of all the interesting ways in which people have acquired users and then made them available to businesses: Chat bots, Amazon, Linkedin, Zomato, Yelp, Woot, mobile games, content and news websites.

The difficult part, as always, is a product that attracts and retains users.

Why every MBA wants to be in “Strategy”

Image source: Wikimedia Commons

Here’s an interesting phenomenon across Indian b-schools: everyone wants to be doing some kind of “strategy”. You ask a fresh-as-fuck 22 year old engineering grad what his latest live project is, he’ll say “brand strategy”. Or ask the 27 year old with 4 years experience in Infosys/HCL/Accenture/Wipro/Cognizant/you-get-my-gist, and he’ll say “IT Strategy”. This thought process manifests itself in the inter-college competitions that students organize. See http://www.dare2compete.com/events/search-result/0/0/0/0/12/strategy for all the fun strategy events.

Here’s a fun look at why this happens.

What MBAs mean when they say strategy

“Lots of armchair gyaan and suggestions, but not one muscle movement worth of execution.”

Let me clarify. The MBA course is filled with “real life” case studies. Most are from the Harvard Business Review, but a few are culled from other sources too. These case studies can almost always be broken out like this

  • Business problem
  • Data (usually incomplete)
  • Few generalized directions that the business can take
  • A bunch of MBAs eager to provide their suggestions

In this situation, almost no one knows what the right approach or answer is, so the MBA (or the professor) will take out his most celebrated tool; the case framework. This he will use to analyze the problem from all angles that the framework affords and will finally come up with what he deems to be the solution. Similar mental gymnastics will then be performed by the other MBAs and the professor, followed by a discussion of what the organization did in real life and why it worked/did not work for them.

However, no one ever talks of the execution part of it. Everyone’s busy imagining themselves as the CMO or CEO, yet no one ever tries to think of it from the perspective of the other 20 years of their careers when all they’ll be is Chief of Jackshit.

Why does this happen?

Because to be able to execute better, you need to have some sort of previous experience in that field. In India however, most people don’t join an MBA program because they want to further their career in their field of specialization, no sir, they do it to switch tracks. Why do they want to switch tracks? Because most often their pre-MBA tracks are

  • Engineering (B. Tech, B.E. followed by some IT company)
  • B. Com. followed by CA preparations and a job in some bank or financial institution as an Analyst
  • Fresher, no work experience

Yet, these often trudged pre-MBA tracks are hardly the ones that most people want to stick to. They’re simply those that were available and paid what then seemed like good money. So since you have no previous knowledge in that field, you obviously can’t talk jackshit about execution. However, anyone can talk about “strategy”. It’s so easy because not even the prof can question you (after all, he himself doesn’t know the answer) and there are a hundred ways any strategic decision can pan out, and yours just might be the correct one.

Glorification by Management Consulting

“Strategy” is also coveted because it is what Management Consulting claims to do. And I’m sure you know how well those companies pay.

In fact, I think one of the reasons why Management Consulting is such a hot career path is because you get to meet such important people and influence such important decisions without any actual “get-stuff-done” involved.

Glorification by b-schools

“Strategy” beautifully ties in with the vision sold by b-schools to students and the inflated expectation and sense of entitlement that the students consequently carry with them. B-schools will never say that in most cases, post MBA you’ll have to slog it out for at least two decades. That just isn’t as exciting as convincing young people that they’re going to become “leaders” the day they step out. And as you all know, leaders don’t get stuff done. They just provide overriding strategic direction and play golf. Just for shits and giggles, checkout IIM Shillong’s golf cup website.

Due to all these reasons, I think the word “strategy” has been put up on some kind of false pedestal. More so since it covers up for a lot of bullshit.

A real life example

Back in first year of MBA, I got selected by one of the companies who were offering a live project on campus. The project outline was

Prepare an entry strategy for XYZ consultants to enter – South
Africa and Kenya. This should include:

  • Local Economy, currency, laws, regulations, etc
  • Target Customers (including geographical spread and key decision maker)
  • Possible Indian Firms
  • Possible MNCs
  • Possible Local Firms with owners having Indian roots
  • Others
  • Target areas in HR Consulting (non-recruitment)
  • Current consulting eco-system
  • Locally based HR Consulting firms
  • MNC firms
  • Indian HR Consulting firms who are doing business there
  • Strategy details
  • What to do in the time period – including how to visits, which companies to approach

· First 3 months

· First 6 months

· First 1 year

· First 1-2 year

Business projections

Dos and Don’ts

Possible local partners

Others

Time fame – 3-4 weeks.

I hope you notice the “entry strategy” right in the beginning. In terms of ego and bragging rights, such a project is pure gold. After all, I got to write “Market Entry Strategy” in my resume. Back then, I remember thinking that just this project would qualify me to get through the CV screening of any consulting firm which came to hire on campus.

But before I get too carried away, let me point out the bullshit to you.

  1. Local economy – the company would hardly be bothered by local economy. They’re a smallish firm that could easily outmaneuver anything happening at the level of the “economy”.
  2. Currency – They’d only be interested in exchange rate and what is the best way to bring cash back to India.
  3. Regulations – It would be silly for an organization to ask MBA students to find out what the laws and regulations for their business in South Africa are. This would be the job of a law firm that specialized and had a lot of experience in this kind of thing.
  4. Key decision makers – No way someone doing secondary research could find out the dynamics of an organization and who it’s key decision makers are. For this, best person would be an experienced Sales professional who understand B2B buying and goes and meets the various stakeholders within that organization.
  5. And so much more that I’ll stop.

Basically, I could go on with pointing out why this is essentially a BS project outline. Oh, and we didn’t submit any of this shit to the company. Our first week, we delivered a “Macroscopic Analysis of South Africa”, all of 5.5 pages long. The contact person reached out to us with what he really wanted: what kind of soft skills training happens in SA, who are the major players and what kind of training programs do they provide? And the fun part is, we didn’t deliver that either, because there is no way you can find out all of this without being in the market yourself.

Eventually, he asked us to find the cost of flying out to SA, camping over there for a few days and moving around trying to setup talks with a few people. If you can’t imagine how that works, it basically means googling “India to South Africa airfare” and surfing all the travel sites for average hotel and taxi rates.

That, my dear reader, along with some hashed up data and analysis pulled from various “Doing business in South Africa” reports was the market entry strategy project that we did.

So what can you do?

If you’re a recruiter

Be very wary of anything “strategy”. Look for simple indicators of employability instead of being swayed by high impact terms that are essentially fluff. Also, realize that persons best suited to understanding things from a strategic perspective are those who have been there and done that. If an ex-Infosys person talks at length on “Brand Strategy”, drill her on the basics of branding. Fundamentals are solidly in place? Good to go. Trips up? Provide her the swift kick she should have gotten much earlier.

If you’re an MBA grad

Stop bullshitting. Get something more substantial on your CV. Seek out projects where you actually get your hands dirty. Stop thinking of ground work as dirty work. Start your career by peeling potatoes.