Highlights from Our Webcast on Lean Startup for Engineers

Guest post by Lisa Regan, writer for The Lean Startup Conference.

Last week, Eric sat down with developer Dan Milstein--one of the most popular speakers from the 2012 Lean Startup Conference--for a webcast conversation on getting engineers into the Lean Startup cycle. If you missed it, go ahead and watch it here. And if you have questions, post them in the comments here; Eric and Dan will be jumping in to answer.

There were some great take-aways from this conversation, not just for engineers, but for anyone trying to rethink how a startup approaches its defining problem—uncertainty. As Eric put it in the webcast, “A startup is a human institution designed to do something new, under conditions of extreme uncertainty. We don’t really know what’s going to work in advance.” The question then becomes: What can engineers, and everyone else at a startup, do to reduce that uncertainty? As Eric and Dan talked this through, they focused on three areas: first principles, organizational shifts, and methods.

First Principles

Eric began by offering a basic principle for engineers interested in exploring Lean Startup ideas: Start with yourself. “Whatever work you’re in the middle of doing right now, ask how you could treat that as an experiment. Have an explicit hypothesis. What is what you’re building supposed to be doing *specifically*? Then define how that will manifest in real life. Then think of how to test and measure – just for yourself – that effect. Do that and you will automatically graduate to a higher level.” What’s at that higher level? Changes beyond yourself, to the organization more broadly. But, as Eric repeated, that has to begin with the individual: “If you yourself are not able to apply scientific thinking in small ways, you’re going to fail at your vaunted organizational change.”

Organizational change
Once you’ve agreed to change yourself, and to start testing and experimenting, you really need the organization to change as well. Engineers can only do so much on their own. Here’s Eric: “If you’re trying to do engineering under conditions of extreme uncertainty in order to succeed in delighting customers and having an impact on the world, you need to build cross-functional collaboration into the process from the beginning, and to see marketing and engineering really integrated into a unit that’s operating together to try to retire some of those risks early on.”

But this is where things can get tricky. Eric: “Organizational change requires the explicit buy-in of senior management. How do we convince senior management that their pet projects, their beautiful babies, are not as beautiful as they think they are?” By asking them to test those projects. Indeed he has a test for determining whether leaders are visionaries: “So many entrepreneurs cloak themselves in the myth of the visionary precisely to avoid having to confront these kinds of questions. The easiest way to tell the difference between a visionary and a charlatan is to ask them to put their ideas to the test. The visionary who actually believes they’re right will agree to test their ideas 10 times out of 10…. My experience is that the ratio of charlatans to visionaries in our world is 10 to 1 or 100 to 1.” 

Here’s a specific example of organizational impediments that Eric and Dan hashed out. The fact that engineers hate to throw away their work — and everyone else is afraid to ask them to. The solution has two points, which Eric laid out: “First is to realize that if we don’t build something people want, the work will be thrown away *anyway*. All this testing and experimentation is a path toward _preventing_ work from being thrown away.” If the product doesn’t get users, it won’t reveal its flaws – but when the company fails it will still be thrown out. As Dan pointed out, “A product outage is a good sign because it means that people care. You have a great stable system when no one’s using it.”

(There’s another, surprising reason Eric says engineers shouldn’t fear throwing work away, but we’ll leave it to you to watch the webcast for that!)

Dan also spoke about engineers’ investment in the business’ overall success as the key element in persuading them to let their work go: “Part of what’s useful is to understand engineers are profoundly driven by wanting to solve meaningful problems. And they may have forgotten that if they’re working at an organization that doesn’t let them do that. But if the leadership of the organization can share the fundamental question of whether the business is going to work as a problem to be solved – let’s all go solve that together – engineers can get really excited about solving that problem, and then they don’t so much mind work getting thrown away.”
 

Real-world Applications
In the last segment of their conversation, Eric shared two examples of organizations that have integrated experimenting and testing as their defining methods, not just in terms of engineering, but across the business. The first of these is an anonymous mobile app whose team does a new build every week, loads it on their phones on Friday, tests it all weekend, and on Monday meets to decide what’s working. Often, they wind up throwing out the last week’s work. Eric: “With mobile apps, especially, the products you love the most are the simplest. They on the surface have very few features and all the complexity is hidden behind a really gentle user experience. Those products can only be produced by teams that are willing to throw code away. If you’re afraid to delete you can only add complexity. Simplification is the removal of complexity.”

For one final example, Eric mentioned last year’s Lean Startup Conference talk by Diane Tavenner of Summit Charter Schools. Her weekly rounds of experimentation and metric-gathering led to the conclusion that, when it comes to teaching kids math, lecture doesn’t work. Eric: “You think it’s hard for an engineer to let code go? You’re a professional educator and you’re going to let lecturing go? That’s hard. And it was only through this rhythm of regular experimentation that they were able to let that go.”

Here’s Diane’s video if you missed it – it’s really worth watching if you’re considering the role engineering can play within an entire organizational shift toward testing and experimentation:

Check out the webcast and then post your questions in the comments here. 

Lean Startup Implementation Lessons

Guest post by Lisa Regan, writer for The Lean Startup Conference.

The Lean Startup Conference was founded four years ago to bring together real entrepreneurs who have applied Lean Startup techniques and have useful advice to share with each other. Last year’s conference was no exception: speakers talked about their direct experience with practices you may need help implementing—things like metrics, feedback, and experimentation. This year’s conference will go deeper on advanced entrepreneurship topics, and we wanted to get you up to speed on some of the key ideas from 2012.

Metrics play a big role in any conversation around implementation. But which numbers matter, and which are vanity metrics--numbers that make you feel good, but don’t actually tell you how you’re doing? At last year’s conference, Red Room CEO and co-founder Ivory Madison pointed out that vanity metrics are among those numbers most commonly cited in investor decks: page views, new members, unique visitors, conversions and, vainest of them all, Twitter followers. None of these, she says, provides information about how well your company is delivering on your business model, nor about the true lifecycle of a customer. Instead, Ivory says, track metrics that capture revenue, observe the behavior of real, individual customers, and demonstrate cause and effect, or how you can replicate a result. These show success at your core business. For the details, here’s Ivory:

Running Lean author Ash Maurya seconded Ivory’s advice. He counsels tracking only five kinds of behaviors: Acquisition (how do users find you?), Activation (do they have a great first experience?), Retention (do they return?), Revenue (how do you make money?), and Referral (do users tell others?). He also advised making it your goal to move just one of these measures at a time, and described his own effort to improve activation with an experiment that yielded unexpected results at pretty much every turn. Check out his talk for the specifics of how Ash tests a metric:

On the subject of experimentation, few have learned more than Matt Brezina, whose company Sincerely launched nine products in its first year to form a group of apps that are the world’s largest gifting network. Matt has deep experience in mobile development and nailed down ten steps to rapid app development, noting that mobile is an environment not obviously compatible with Lean Startup techniques. His talk covered building a minimum viable product (in his case, an app that let people send their kids a photo of Santa) to test basic processes like billing and delivery, reasons for testing in Canada rather than the US (really!), and buying cheap users—plus it encompassed team structure and time management. The talk is detailed and specific—a must-see for anyone in app development:

For one last example of experimentation, we point you toward a short, inspiring talk from Alejandro Velez and Nikhil Arora of Back to the Roots. These two Berkeley grads heard from one of their professors that it was possible to grow mushrooms in coffee grounds. Based on that thin evidence, they became full-time urban mushroom farmers. But what got them into retailers like Bed Bath & Beyond was a series of experiments they ran at farmer’s markets, where they used a timer to test whether they could get more engagement from customers if, in addition to fresh mushrooms, they also offered mushroom growing kits. Results: “For the first time, we were having kids and families engage with us,” Alejandro said, “and they would be with us for minutes.” The kits enormously increased engagement—but it’s important that this product change did not actually alter their mission. Here’s Nikhil: “Our vision has remained the same since day one: making food personal again. But that test…actually shifted our vision from growing fresh product, fresh mushrooms, to educational products that can inspire people to grow their own food."

Alejandro and Nikhil were listening to their customers, a theme Lean Startup constantly returns to. Last year we heard a variety of strategies for getting customer feedback, beginning with Tendai Charasika of Greater Louisville, Inc. offering ten ways founders can get out of their offices. His tips include some crucial reminders (i.e., don’t just ask your mom for her opinion) to some things you may never have considered, like moving your offices to where your clients are, rather than to where the other startups are.

We also had one very specific tip from Adam Goldstein of travel site Hipmunk: install live chat on your site. Live chat gets real-time feedback and accurate information about where customers are getting hung up. “People don’t email to say that something is tricky to use,” Adam pointed out.

At this year’s conference, we’re bringing in speakers who are ready to describe the exact methods they use to implement Lean Startup methods, what numbers they look at, how they pivot from them, how they test their product, and how they find out what their customers want. In upcoming posts we’ll fill in the details about our breakout sessions where you can directly ask successful entrepreneurs, “So how do I do this?” We sell tickets in blocks—when one block sells out, the price goes up. Our current block of tickets is nearly sold out, so register today.

Five Questions for Lean Content Creator Stephanie Hay

Guest post by Lisa Regan, writer for The Lean Startup Conference, who has created content for a number of startups.

In our ongoing exploration of how people actually implement Lean Startup methods in their workplaces, we’re asking practitioners to share their techniques. We’re starting here with a conversation about content with Stephanie Hay, the leading voice behind Lean Content, which she describes as “finding the ‘right’ words to acquire customers before building products.” Her advice is the kind of nuts-and-bolts information you’ll find at this year’s Lean Startup Conference, which will feature in-depth sessions on the how-tos of Lean Startup—including a session lead by Steph.

Steph has incredible energy, as you’ll see in this video of her 2012 Lean Startup Conference lightning talk, where she discussed testing language that “helps people find you, understand you, and choose you.” When not at CrossFit, Steph tests content at Work Design Magazine, which she co-launched in 2010. She has also mentored at 500 Startups, co-organized the DC Lean Startup Circle, and co-founded FastCustomer. We asked her five questions about how, exactly, clear content gets created. Among her answers is a simple test for assessing whether your company has a content problem.

1. How do you iterate on the content side without confusing customers, or seeming as if you lack focus, commitment or are just plain flaky?

Steph: Segmentation and timing. Work Design Magazine has 5,000 email subscribers who comprise a variety of audience types within the commercial architecture and design industry. So we segmented the list by audience type, and secondarily decided which ideas we were going to circulate when and to whom.

For example, we decided to test whether a print edition was feasible. The assumption was that architects and designers would pay a premium for a beautiful, high-quality journal full of the most popular content from the previous quarter, complete with tons of high-resolution images. A secondary assumption was that furniture manufacturers would want to advertise in that journal.

We segmented the audience between likely readers and likely advertisers. We created eight different versions of "sell" content—describing different value propositions of why they'd want to subscribe to receive it (or advertise in it), what they'd pay, and when they had to decide (a deadline).

We then sent it to targeted groups of about ten users each, followed up with them personally two and four weeks later, and stuck to the deadline as our drop-dead date for learning.

Within a month, we learned that both audiences were lukewarm on the product. So, essentially we saved time and energy associated with the massive overhead of printing a publication—not to mention the risk of keeping it going—and punted on it for now. 

The next month, we tested the ideas of events taking the same approach—and it turns out, this was a big revenue source. Readers pay to get tickets for high-quality, in-person content (panel discussions) in cities around the country, and manufacturers and other vendors pay to be associated with that content.

That's the iteration part. If we just sent all our ideas to everyone, whenever we had that idea, it wouldn't leave room for iteration. It would be a blanket approach and, ultimately as you suggested, dilute the messaging and demonstrate a lack of focus.

I think the hardest part for any startup is to rein in the desire to "throw shit against the wall and see what sticks" versus being more strategic in their focus and truly iterating. A content-first approach is a lower-cost, lower-risk way of achieving the same.


2. How does lean content integrate into the rest of the lean product development cycle?

Steph: For me, it leads. If you create a Google doc that describes a small chunk of an interaction—the language you'd use to "sell" an upgrade from a premium model, for example—and you circulate that to some power users, what do they say? Do they ask questions? Do they want to sign up?

Same thing with the email; if you send an email "selling" an idea, and you see the open rates are super low, that means people aren't interested in what you have to say (or at least, your subject line isn't compelling).

I try to make the content focused on what I can learn from it. Normally, that means including two possible conversions: (1) "learn more" and (2) "get on an email list." If people go straight to (2), that's a great sign.

If they open it but don't click, that's a sign your content isn't hitting.

If they click through more often on "learn more," I direct them to a page with more "sell" content. And that page of course includes a conversion to "get on the email list." If the user lands here but doesn't convert by signing up, I expect the content isn't compelling enough. The sell isn't there.

I iterate on the email content and page content over a course of time, maybe one to two months, usually based on the number of people in my test group and/or uniques on a landing page.

You can learn a lot before even sketching wireframes, and you should, because the layout doesn't matter if the "sell" doesn't hit.

Content needs to "sell." The words need to resonate. Once you nail those, it gives the team wind in its sails to keep moving forward.

Plus, UX designers and developers love it when they already have the "right" content done. It makes their jobs easier to not have to think about what to say—so ultimately they can run faster, and what they create is then more effective right out of the gate.

3. What do you need to know before you create content that is able to be found, understood, and helps you get chosen? How do you go about collecting and analyzing this data?

Steph: I do a ton of research in the Google Keyword tool. Seeing monthly search numbers for organic searches is revealing of the kind of language people use—and the kind of possibilities (numbers-wise) for specific markets.

For example, for Work Design Magazine, do people search for "work design" or "workplace design" or "workspace design" or "designing for work" or … you get it. I put all of the terms into the Google Keyword to better understand the lexicon of my target market, which of course I can then work into H1s, title tags, and even subject lines or blog-post headlines. This is all catered toward trying to speak the user's language.

Using the "top content" tab in Google Analytics helps me understand what qualified visitors are looking for. When they come to the site, let's say through organic optimization, do they stay and read more articles? I'm focused on the ones who do. They see something intriguing. So I want to write more of the kind of content people are finding and staying to read.

Getting them to "choose" requires the kind of strategic, iterative approach to learning what content "sells" as described in the first two answers.

4. You've spoken about the fact that people on the inside of a product fail to explain it clearly to others. Why do you think clarity is so difficult and how can people start to recognize that they're being unclear?

Steph: Clarity is difficult because we know how it's built (or what we want to build), and we have motives for how we want things to go (really well, blowup style).

So naturally, we say too much that the prospective customer doesn't care about, or we don't say enough of the "right" words that they need to hear to arrive at the "aha" moment.

But this is where the notion of "lean content" comes into play. Before you have a product, it's possible to find the language that "sells" the product. It's the minimum amount that needs to be said in order for a person to say, "Holy shit, why didn't I think of that?" or "How do I pay for this?" or "Where do I sign up?"

Those are a-ha moments. They're ready to buy.

Responses like "Interesting, good for you," or "Cool, email me more information" or "I'll get back to you" are lukewarm, non-committal responses. There's no urgency, specificity, or indication of action. They indicate you either haven't sold the real value or you're selling to the wrong person.

It's a challenge to hear these not as compliments (and false optimism that there's a business behind what you're saying) but instead as signals that you're not communicating value obviously with the "right" prospective customer.

But with awareness and discipline, you'll only arrive at the real language (and buyer) faster.
It means humbling yourself to really hear what people are telling you, politely, rather than hiding behind the lukewarm.

5. What tests can people run right now to decide if they have a content problem?

Steph: One lightweight test is to print out the content you've got on your website, probably your homepage and sign-up page. Then call your mom or friend and try to have a conversation—a natural conversation—using your own content. See how that works.

If you find that the conversation is one-sided, or that the person you're talking to can't understand what you're promising/asking/selling, then you have a content problem.

The mistake startups make is thinking they need a copywriter who can "polish" their words. Great content effectively sells; the only way to write effective content that helps build businesses is to tie content to conversions. Most startups don't get this. They think it's the layout of the page or the function of the product. Of course, these are important, but they assume someone understands your words enough to sign up in the first place.

Have any startups failed because they've had too many people scrambling to sign up?
So as another test, go out on the street and find anyone willing to listen. The person doesn't have to be your target audience -- you're just trying to test if you can be understood when selling your own product.

If you tell people what you do in 10 seconds and ask them, "Would you need something like that," and the person says, "I'm not sure I understand," then you have a content problem.

If the person says, "I don't need it," then you *might* have a content problem (of pitching the real value), but at least the person understands what you're selling.

If the person says, "Tell me more," then you *might* have a content problem of pitching the real value quickly enough, but at least you're heading in the right direction.

If the person says, "OMG I NEED THAT," you're winning. Stick with that.

Then the job shifts to discover only what you need to say to keep the conversation going and to direct it to a conversion that grows a business. Inherent to this is then also to begin discovering who the market is that understands the conversation and converts quickly and at the highest value. This naturally refines the messaging over time.

(Notice that both of these tests require no code. Just conversations with friends and strangers. Imagine if everyone started here and decided which businesses to build based on their ability to explain them in the first place.)

Learn more from Steph and other practitioners at this year's Lean Startup Conference. We sell conference tickets in batches; when one batch sells out, the price goes up. We've got a new batch on sale today, so register now for the best price.

Getting Engineers into the Lean Startup Cycle

Guest post by Lisa Regan

On August 20, Eric will sit down with developer and Hut8Labs co-founder Dan Milstein for a webcast you can join to discuss “Getting Engineers Into the Lean Startup Cycle.” This conversation will be a great opportunity for engineers and engineering managers to learn more about implementing Lean Startup ideas. It’s also for founders who want to think about how engineering and the rest of the business team can realign around a shared set of goals. This webcast is the first of a series held by The Lean Startup Conference and will take place on August 20th at 10am PST/1pm EST; it’s free—you just need to register in advance.

The conversation will be free-ranging, and it will include live Q&A with attendees, so we don’t know exactly what will happen. But there are a few themes Dan and plan to hash out. Dan comes to this conversation as an engineer, product developer, and startup veteran with a career-long interest in how people interact with the complex systems they themselves build. He investigates the peculiar conditions present in a startup, where, by definition, everyone, including the engineers, is operating in extreme uncertainty. He is interested in asking, given that environment: How can the team measure progress? How can engineers establish benchmarks or make predictions? Should they even try? How do you move the business in the right direction—and how do you find out what that direction might be? All of which amounts to: How do you reduce that inherent uncertainty? Dan and Eric will also talk about how engineers can contribute to containing uncertainty—including in the way they interact with leadership.

These issues don’t lend to easy answers. Indeed, at a startup, the entire business is one big question: “Can we offer a product that fills the customers’ needs such that they’re willing to pay for it?” As Dan described in an interview last year, many businesses inadvertently turn away from this question by splitting the teams for product, marketing, and engineering apart from each other, with each pursuing a tailored set of questions. But this often allows everyone to run from the problems of risk and failure. Instead, Dan says, all of the teams, as well as leadership, need to collectively accept that “in conditions of extreme uncertainty, you’re going to get things really badly wrong,” so it’s important to learn to “focus on the risks rather than pretend they aren’t there.” To do this, you’ll need to overcome basic psychological and structural barriers—ones that Dan and Eric will discuss—and rethink the way engineers and leadership communicate.

For a specific example of how Dan’s been thinking about those barriers in his own work, check out his presentation from the 2012 Lean Startup Conference, in which he broke down the way he runs a Five Whys post-mortem following a product or process failure. As Dan described in his talk, post-mortems are hindered by our human sense of guilt, shame, and blame. Because, he notes, even as we are asking, “Why?”, we are also putting a focus on, “accusing, identifying villains, punishing. And this is going at it all wrong.” We’ll let Dan talk directly about his experiences, but among his solutions is changing the question itself. For example, instead of using a Five Whys to find out, “Whose fault is this?” or even “What was the root cause?”, use it to ask yourself, “What were the multiple contingencies that led to this problem, and which of them can we efficiently address right now?”

How can engineers and leadership (and product and sales) realign around the right questions? What human limitations and structural hindrances will they need to overcome to do so? And, how will they achieve this? On August 20, join Dan and to discuss these issues. Register today.