At the conference, he’ll be talking about transforming your organization’s culture to support innovation and running two workshops: one on Mastering Experiment Design, and one on Customer Discovery. He’s also written books on both of those topics: Talking to Humans, published in 2014, and now, a follow-up volume called Testing with Humans, just out this week. Frank Rimalovski of NYU collaborated on both. We recently spoke about customer discovery, experiments, and his writing habits and process.
Testing with Humans uses a fictional story of two entrepreneurs building a soccer ball sensor company. How did you come up with that example and what makes it useful for discussing experiments?
The story is really a composite of multiple real situations. I just blended them into a unified arc. Business fiction can be tricky to get right, but it did allow me to get a lot of ideas across with one coherent example rather than twenty fragmented ones.
As for why a soccer ball? That's easy — I'm a huge English Premier League fan (go Bournemouth!), so how could I resist? In all seriousness, in both this book and its prequel, I wanted the entrepreneurs to have a product idea that many people could relate to. I chose physical products both times so that the concepts didn't feel specific to tech or software. Experiments and customer discovery are powerful across all types of business.
What was the most surprising thing you discovered while gathering the research for the book?
I guess the most shocking thing came from talking to my former colleague David Bland. I was surprised to hear him say how many people, across his coaching, are still struggling with the basic concept of landing pages. I would like to think that we're further along than that. I always have to remind myself how long it takes for things to seep into widespread practice, especially things that fight against our cognitive biases.
What step do people most often try to skip in the experimentation process?
I'm going to give you a three-part answer. The first is experiments themselves. Everyone knows that they should run experiments, just like they know they should talk to customers, but that doesn't mean they do it. People chase a perception of speed, when in actuality they go slower because of mistaken decisions, big and small, that could have been avoided.
Within experiments, however, it really depends on one's psychological profile. "Do-ers" jump right into experimenting without prioritizing what they really need to learn. "Seat of the pants" types ignore structure and change too many variables in parallel, making it really hard to interpret results. The high-charisma founder usually has a huge problem resisting confirmation bias. The introverts hide behind the experiment, putting data on a pedestal while missing out on the huge insights that come from talking to participants. To do lean startup well, you need to understand your own makeup and challenge your weaknesses.
Lastly, I've learned that for any big, important experiment, it's worth running an experiment about the experiment. In other words, test the design first, before scaling (just like a startup!). This small investment pays dividends with speed and better decisions.
If you had to pick one element that every single experiment needs to have, what would it be?
A very specific quantitative goal, even if it is a guess! I've always seen that if you don't go into an experiment with a clear goal, everything becomes mushy from that point on.
What's the most challenging audience question you've ever had to answer?
That has to be: "How do I know when to kill an idea?" I share a lot of tactics in the book, but there's one area where I absolutely refuse to be prescriptive: decision-making. Everyone has their own context. I certainly do not believe that one experiment should make or break someone's vision. However, I firmly do believe that good experiments will arm you with far better information to make those big, heavy decisions. Unfortunately, they don't let you off the hook of having to make them.
What's one of the best experiments you've seen and why?
The non-profit Taproot Foundation ran an excellent Wizard-of-Oz experiment as they sought to scale, by way of an online application, their historical service of connecting non-profits with pro-bono experts.
Because they had been manually providing this service for years, they could have been cocky about their expertise. Instead, they challenged themselves to ask hard questions. They ran a Wizard-of-Oz experiment that was sharp from top to bottom. They had tight hypotheses statements and quantitative goals. They ran it at an intense pace. They tracked and openly shared their metrics. They talked to participants of the experiment, rather than relying solely on metrics to tell the story. The team also brought their entire organization along for the ride with excellent internal communication. In the process, they ran into unexpected challenges, learned a ton, and ultimately informed a much better final product (found at https://www.taprootplus.org).
It was particularly impressive to see a non-profit, which as a field can be pretty hidebound at times, attempting to work in such an agile and forward-thinking way.
Do you have a daily writing routine when you're working on a book?
This book took years to get out, because I was busy selling a company and then helping to build another startup. I was finally able to build momentum by getting up regularly at 5am. I could get an hour or two of good writing in while fresh, before startup pressures invaded my brain. And I'll admit, my family has been very patient as I've disappeared for weekends at a time.
----------
You can buy Talking to Humans here. You can buy Testing with Humans here. Both books are available to schools and non-profits for free, via a form you can fill out either here or here. Follow Giff on Twitter at: @giffco.
