Experiment design, an introduction
An introduction to experiment design: a two-week sprint – taking an evidence-based decision to embrace or kill an idea
Imagine this. You’ve been playing around with an idea for a new product or company for a while. Analysis has shown you that a lot of the questions asked & actions taken by your customers are the same, very repetitive. You come up with the brilliant idea of what you call ‘the digital assistant’. This virtual assistant helps clients to solve common problems, stores all answer on FAQ’s in one place, …
The only thing is, you don’t know where to start. Sound familiar for you?
We don’t do sentiments, we do numbers
The only way to figure it out: testing it. Find out exactly which segment of your customers might be into such a solution. And if they are open to the idea, which features would they be tempted to use? To map and measure their willingness, you have to simulate a V1 of this product, allowing you to track the behaviour of potential users.
That is what ‘experiment design’ is all about. Running (small) tests allows companies to make evidence-based decisions. Experiment design validates assumptions. It’s a way to translate gut feeling into hard-boiled facts and figures.
Aren’t companies already testing new products and services? Yes, they do. But they test them in a qualitative way, that’s open for interpretation. They set up surveys with open questions, they organize focus groups, they bring together user panels. Experiment designs aims for quantitative results. Think traffic, think clicks, think downloads, think subscriptions. You don’t get sentiments, you get numbers.
Wanted: the best failed idea
Experiment design is an interesting way to discover things quickly. We run sprints, not marathons. Too many companies still make decisions based on last year’s marketing plans. But technology evolves with the light of speed, and consumer behavior is unpredictable. Ideally, the experiment runs no longer than two weeks. After the experiment is done, you evaluate the result, and you either embrace and implement the idea, or you relentlessly kill it. No regrets, the numbers have shown it won’t work.
The Indian giant Tata has a prize for the best failed idea. Each year, the company rewards an idea that terribly failed. Not to make fun of the employees behind the idea, but to convince everyone working at Tata from the importance of trying out new things. Even if they fail miserably.
What experiment design essentially does, is radically speeding up that process of trying. And failing. Or succeeding. Experiment design done right can save you a tremendous amount of time and money. By implementing great ideas. Or by NOT implementing bad ideas.
Canary in the mine
Experiment design has an impact on your organization that goes beyond just the experiment. It makes you question whether you have the innovative power to compete, it exposes the roadblocks that stand in the way of innovation, and it forces you to switch gear. It’s like the canary in the mine.
Booking.com had dozens, even hundreds of different versions of its website running, focused on all kinds of different markets and target groups. When it takes you weeks to get the approval of your legal department for setting up even one landing page, you know you have a problem you need to solve. Asap.
The golden formula: if-then-because
In order to get the right answers, you need to ask the right questions. If you wish to validate certain assumptions, you need to make them measurable. You do so by translating them into hypotheses. An assumption is a claim without proof, a hypothesis is testable. If the assumption is nobody clicks on the ‘call to action’ buttons on your website, the hypothesis is that the number of clicks will increase by 10% if it doesn’t say something generic like “learn more”, but something really practical like “request a demo”.
A hypothesis always follows the same pattern: if-then-because. IF I change the call the action, THEN more people will click, BECAUSE it’s clear for them what they will get in return. To help you translate vague assumptions into measurable hypotheses, we developed experimentation cards.
What are you waiting for, corporates?
There are a few tough misconceptions about experiment design.
No, it’s not just for startups. On the contrary, especially big companies benefit from experiment design. Startups are agile and flexible by definition, which gives them a huge competitive edge over corporates. For these corporates, experiment design is a way to bypass slow and sluggish internal procedures.
And no, experiment design is not the playground of the innovation team. Experiment design is suited for every team and every department, that’s the beauty of it.
Let’s imagine that an abnormally high percentage of applicants drop out between round one and round two of your application procedure. The HR department can easily run one or more experiments to find ways to raise the level of candidates pushing through. Will the tone of voice of your e-mails make a difference? Maybe candidates will stay on board when you give them a call instead of dropping them an impersonal e-mail? Or when you change the time of their meeting? Only one way to find out.