Case Study: Rapid iteration with hardware

(I am often asked to explain how to apply Lean Startup approaches to domains beyond software. In order to answer, I have taken to drawing a two-axis diagram. 

On one axis we have the degree of market uncertainty for a given industry. For "cure for cancer" type businesses, there is no question about who the customer is and what the customer wants, and therefore there is no market uncertainty. On the other extreme, modern web-based applications face almost no technical risk, and are governed by high market uncertainty.

On the other axis we have the underlying cycle time of the industry in question. Slow-moving cycles, like drug discovery or new automobile models, govern the slow part of the axis. On the extreme opposite end are rapid iteration businesses like software or fashion.

The key to understanding Lean Startup is to recognize two things:

1. Lean Startup techniques confer maximum benefit in the upper-right quadrant, namely high market uncertainty coupled with fast cycle time.
2. Every industry on Earth is currently undergoing a disruption that is causing it to move along both axes: more uncertainty and faster cycle times.

I am aware of no industries that are moving "backwards" on either dimension. Thus, more and more industries are starting to look like the software business. Of course, the underlying root cause of this worldwide disruption is the software and semiconductor revolution. Industries are disrupted as their traditional work process is "infected" by software. And, as a result, more and more companies are able to benefit from Lean Startup practices. 

The following case study looks at one such industry, consumer electronics, where the pace of iteration has taken a marked turn towards high speed. It is written by Ronald Mannak, who is currently the CEO of a startup named Yobble. What follows are solely his opinions. -Eric)

In a bar in Amsterdam in 2005, my two cofounders and I came to the sad conclusion that startup we tried to built for two years was doomed. In 2003 we started developing a martial arts motion sensing toy, a full three years before the Nintendo Wii changed the world of motion sensing. The toy (we called it Ninja Master) consisted of two hardware units, attached to both wrists. When a child would perform a perfect karate move (or better yet: a combo of several karate moves in a row), Bruce Lee-like karate sounds would emerge from a small speaker in the device. We loved the product. Test users loved the product. It was way ahead of its time. We thought we were visionaries and believed the future was motion control. Yet, we failed to sell the toy. We talked to every toy company imaginable, but none wanted to license our toy. " Kids nowadays don't want to move, they play Playstation" was the most often heard reply, even though our user tests suggested otherwise. To make matters worse, we lived in a country (Holland) without a proper functioning startup, VC and angel ecosystem. The company was doomed. My co-founders decided startup life wasn't for them.

However, one new idea emerged at that meeting. What if we could make an air drum? Drum sticks with sensors in them. Now that was an idea. Music is much easier to sell (to toy companies) than the abstract martial arts Ninja Master toy. Besides, we could easily expand the line with with an air guitar and a device to link the air instruments to a PC. How cool. I loved the idea so much that I decided to pursue the idea.

I envisioned the product would be popular with 8 to 12 year old boys. I thought the price couldn't be higher than $40. I already knew how the product would be used. Boy, was I wrong.

Waterfall
I previously worked on a couple of IT projects that used the 'waterfall model' where specifications were written down by one team, thrown over an imaginary wall and implemented by another team. Every single waterfall project I encountered turned out to be a disaster in every way. Specifications turned out to be open to multiple interpretations, usability was the last priority (if a priority at all). It Just Did Not Work. As a beta tester of the first Borland Delphi, I learned the wonders of rapid prototyping and fast iterations. I wondered if we could do the same for hardware development. It turned out we indeed could.

The first hire
The first hire was critical. I wanted somebody who was creative first and technical secondly. I found the perfect person at the department of Industrial Design Engineering of the Delft University of Technology. Joris. Joris was creative and eager to learn. Better yet, he plays drums. Even better than that: he likes to tinker with electronics. Hiring him was a no brainer, and he didn't disappoint.

The internship only lasted six months. That's not much time, considering the scope of the project. I convinced the university that Joris should not be writing specifications and other nonsense first, but start right away building prototypes. And he did.

Joris suggested that before he started working on electronics, we should invite children, give them wooden drum sticks and let them pretend they were playing air drums. It turned out to be an excellent idea. Children are perfect test subjects. To our surprise, every single child did something we didn't anticipate. Without any exception, they all whacked the wooden drum sticks *sideways* and made 'crashing sounds'. I certainly didn't think of sideways movements when I created the first ideas, but apparently it was a good idea to implement.

The prototypes
The next day we started building the first prototype to see if the sensors actually behaved like they were supposed to, and to see if we could measure the sideway movements. The prototype was crude. Joris taped sensors on his arms with duct tape and started drumming in the air with wooden drum sticks (that did not contain any electronics). We connected the sensors to a seven year old pc with an Arduino-like interface that ran a simple drum program we developed. The results were amazing. It actually worked. (A video of the first prototype can be found here.)

We now knew what kids liked and we knew the product was technically feasible. Yet, I still felt we didn't know all we needed to know and wanted to test more. And I'm glad we did.

For the next prototypes we placed the sensors in PVC pipes to optimize sensor angles and added features to the pc software.

We made another discovery we did not anticipate. We found out that parents (who came along with their children for user testing) often liked the prototype as much as their kids. We decided to interview the parents and quickly found out that the parents who like our product were video games players. Of course we liked our product, but we never would have guessed other grown ups would like our product too. Knowing this, we invited test users from 12 to 30. They also loved the prototypes. Our target audience just exploded in size. We decided to make a few changes that would make the product less 'toy' and more 'gadget'.

Over a period of six months, we made eight generations of prototypes, each version adding more features and making the product more reliable. By testing each generation, we learned that a lot of hypotheses were correct, but a large number of hypotheses were incorrect. By testing early and often, we were able to adjust the product. I believe that we demonstrated that it is indeed possible to iterate fast and often with hardware development.

Product Launch
After some financing-related delays, the products went on sale in Europe and Asia in the summer of 2008. The retail selling price was $40, exactly what we targeted. In less than six months, we sold over 90,000 units. All shops sold out our products two months before christmas, all without spending one penny on marketing. The products were voted 'best music gadget' on television program The Gadget Show, became the best selling music toys on Amazon.co.uk and the best selling products on Firebox.com. Best of all, users love the products. On Firebox.com, the average user rating (740 users) is 4.5 out of 5 stars (link). We couldn't have been happier.

Post mortem
We demonstrated it is possible to iterate often and fast. I believe a lot of the product's success can be attributed to the iterative development process. We didn't find every issue though. We didn't test the price and we didn't see the Nintendo Wii or Guitar Hero coming. We chose to enter the market though the low margin toy market, where (in hindsight) we should have positioned the products as video games with higher video games margins.

Another thing we missed: after we launched we received many requests to add double bass drum, as often used in metal. The drums include two drum pedals and a double bass drum could have been added by a simple and minor change to the embedded software. However, updating the embedded software in sold devices isn't possible with the microcontroller we used. We could have included the feature in a 1.1 version of the product, but the toy manufacturer we licensed the toy to, wasn't interested in a new version, as the original version continues to sell well to this day.

Tools to develop hardware get better and cheaper. Open source projects like Arduino and SuperCollider make iterative hardware development cheaper and faster than ever. We learned that connecting the prototypes to PCs and user the PC to run the program is a very good way to test hardware (developing on a PC is still much faster than embedded developing on a standalone hardware device).

In the summer of this year, I moved to San Francisco and founded a new startup that makes music related games and hardware controllers that connect to the iPhone. There are a lot of new opportunities. New cheap flashable micro controllers make firmware updates possible for low cost hardware. With hardware connected to the internet (in our case through the iPhone) it should be possible to use continuous deployment: small and very frequent updates of the firmware instead of less frequent large updates. Bugs in firmware could be fixed within minutes or hours instead of weeks or months.

(Continuous deployment of hardware is an exciting new capability. In addition to continuous deployment of firmware via the Internet, it is also possible to do continuous deployment by taking advantage of a small-batch production process. When the complete cycle time of assembly is low and the design can be specified mostly through software, it's conceivable that each batch rolling off the line could have a different design. -Eric)

As a final thought, I am convinced iterative design depends mostly on the mindset of the team and the company culture and less on the tools. I was lucky to have a great team of A-players that were willing to take responsibility and risk. If the company culture is such that mistakes are punished, I am pretty sure iterative development won't work.

The Lean Startup Bundle

People often ask me if they can buy "Lean Startup in a box" from me. It's a strange thing, to be asked by a potential customer if they can give you copious amounts of money, and then to have to refuse. Startups run into this problem all the time: not every possible way of making money is equally useful. On the other hand, I try to keep in mind the idea that feedback is always about them and never about you. I recognize the need being expressed by this request. People want to get started with Lean Startup but aren't exactly how. If you're in that boat, today my friends at Appsumo are trying a new experiment that just might help you out.

It's called the "The Lean Startup Bundle" and it is available this week only. It is the ultimate product development get-started guide to Lean Startup. We did our best to package as much content (including almsot a dozen books and ebooks) as well as tools and apps, into one low-priced magic offer. Naturally, the bundle includes every essay I have written for this blog, as part of the Startup Lessons Learned ebook series.

But I am even more excited to report that it will include a hardcover copy of my new Lean Startup book. That's right, I am writing an old-fashioned, traditionally-published book full of all new material. Because of the traditional publishing industry's commitment to rapid iteration, the book won't come out until Fall, 2011. But if you buy The Lean Startup Bundle this week, you'll get a pre-order of the book included. That means you will get a hardcover copy of the book in the mail when it is eventually published. (Believe me, you will be hearing a lot more about this book over the course of the next year, so that's all I will say about it right now.)

But that's not all! I'm even more excited to announce two brand-new ebooks from two of my favorite bloggers: Sean Ellis and Andrew Chen. These ebooks were created by my friends at Leanpub especially for this bundle, and include the best essays written by both authors (and a foreword from yours truly). Also included is the incredible Venture Hacks Bible, which is worth the price of admission on its own, and The Entrepreneur's Guide to Customer Development (which I previously reviewed here).

And there's so much more. For two awesome books which have been traditionally published, we have several ebook chapters: Inbound Marketing by Dharmesh Shah, Do More Faster: TechStars Lessons To Accelerate Your Startup by Brad Feld & David Cohen. And also making its debut, you'll get a sample of Ash Maurya's forthcoming ebook Getting Lean.

"But," I hear you say, "so far this sounds like just a bunch of gurus trying to sell me books." Indeed, but - as they say - that is not all. We also reached out to companies that make products useful to accelerating your Build-Measure-Learn feedback loop, from cloud-hosted Selenium provider Sauce Labs (disclaimer: where I have equity) to ScrumPad (where I don't) and many in-between (a sampling is below, but you should really just click through to see the whole smorgasbord). (And even if you don't buy this bundle, one beneficial side-effect of my research for it is this cool list of Lean Startup tools curated by twtpick.in. Because of the short timeline in putting this all together, we couldn't include them all. But, if this bundle is successful, perhaps we'll have a chance to do another.)

Phew, I'm exhausted. "But," you are sure to ask, "how much will all this bundling cost me?" How about $1000? $500? $250? No! All of those prices are much too high. "Ok," you surely say, "how about $100?" Absolutely not. "$75?" No. "$65?" Absurd.

No, all of those prices are incorrect. If you act now, you will pay not one dollar more than $42. Don't panic. Buy now.

Stop lying on stage

Entrepreneurs crave information about successful startups, and they should. Most of the received wisdom about business and entrepreneurship is simply wrong. Many journalists and conference organizers attempt to fill this demand by giving successful entrepreneurs the opportunity to tell their stories: in magazines, on blogs, and on stage. And yet, most of the time, those opportunities are wasted, because the protagonists tell lies. And while this may sound like a harmless phenomenon – after all, most of these are simple lies of omission – I think the consequences are quite harmful.

I know the word “lie” sounds harsh. But I think our industry has to face up to this unpleasant reality. Luckily, some people have started to collect evidence of misleading behavior on the part of speakers. Paul Graham wrote this just the other day:

I didn't consciously realize how much speakers at more public events censored themselves till I was able to compare the same people speaking off the record at YC dinners and on the record at Startup School. YC dinner talks are much more useful, because the details people omit in more public talks tend to be the most interesting parts of their stories. About half the interesting things I know about famous startups, I learned at YC dinners.

I commend Paul for his honesty, something I have always admired about him. Paul has been – intentionally or unintentionally – running a perfect science experiment to answer the question: do startup speakers tell the whole truth on stage? His results are the same as I’ve observed on many occasions: the simple answer is no. I think everyone who’s been around these events long enough knows this to be true. None of the really successful people I know take what’s said at public events seriously. This is the same issue we see with vanity metrics: companies are giving the appearance of sharing information while actually engaging in spin or outright deception.

I call this the vanity ratio: the amount of apparently interesting information given divided by the amount of useful information contained therein. The higher the vanity ratio, the more effective the PR. Unfortunately – also – the more misleading the story is as a help to others.

And it’s not as if the stories we hear about entrepreneurs are biased in a random way. Paul quotes one of his founders like this: “That's the actual beauty in the off-the-record-ness: you hear just how screwed up most of these successful startups were on the way up.” In my experience, the official stories are always more linear, make the founders and investors look smarter, and dramatically overstate the level of certainty everyone had at every stage of the process. Failures, pivots, and crappy minimum viable products are generally elided. And the kinds of failures that do get airtime are usually failures to adequately plan, anticipate, or design in advance. So, naturally, the kinds of inferences we make from these stories are: we need heroic entrepreneurs, with absolute certainty in a brilliant idea, and we need to plan and execute well.

I call this the mythological-industrial complex, because it serves the interests of many players in preserving the status quo. It sells newspapers and magazines. It helps investors boost their profile and convince entrepreneurs that they offer value-add. It helps companies with PR. It makes successful founders famous. I've certainly been a beneficiary of these forces. Yet I worry that it deters new entrants from disrupting incumbents: if your idea looks a little dubious, your career a little messy, your team a little dysfunctional, if you lack superhuman design skills, maybe you should just give up. You don’t have the Right Stuff to become an entrepreneur.

The reason this is harmful is that it benefits another kind of person even more: the startup advisor. Yup, that’s me. When all of the stories floating around are bogus, there is a huge opportunity for people to claim to have arcane or esoteric knowledge. Paul does it right in his essay: “About half the interesting things I know about famous startups, I learned at YC dinners.” Doesn’t that sound good? Don’t you believe that having that special knowledge would help Paul give you better advice? I emphatically do. Most of my mentors have had this arcane knowledge, and I have benefited from it immensely. They know how the game is really played. They know what really happened in those pivotal moments that shaped famous companies. And I have done my best to pay it forward, by sharing that arcane knowledge with as many startups as I can.

However, the problem is this: how can you tell if someone who claims to have arcane knowledge is telling you the truth? After all, the fact that what they advise you contradicts all public sources of knowledge is part of the pitch. And, of course, it sounds good. All talking heads (including me!) face an overwhelming incentive to say whatever it is that will sound good to their audience, regardless of whether it is true. Believe me, it is hard to resist (as far as I can tell, that’s why cable news is so awful: pretty much everyone on cable news is giving in to this temptation every day). But I have met many startups that are making key decisions based on arcane knowledge that is simply not true. It is extremely hard to help them, because they are following voodoo advice that is nearly impossible to disprove. In fact, in some of my early entrepreneurial ventures, I was taken in by people just like that.

I have tried really hard to hold myself to a high standard in all of my public presentations: to give the unvarnished truth about what it was really like to succeed and fail. I do my best and yet I probably fail sometimes, too. In fact, I rely on readers and attendees to ask me the hard questions that challenge me to root out and eradicate these errors. And this was an absolute requirement of the speakers at sllconf. The reason we had such a non-diverse lineup (much to my chagrin) is that I insisted on hosting speakers whose stories I already knew – because I had been there, as a friend, advisor, or investor. Thus, I knew they were telling the truth and I knew they shared a commitment to speak with integrity even when it’s uncomfortable. This is also why I get so outraged when people treat Lean Startup like a religion. The whole point of transforming advice into the form of a testable theory is to allow people to evaluate whether it works. You can test theories like Lean Startup in your own practice, and discover if they deliver the outcomes they predict – and you can do it in small batches.

That’s why I find lying on stage so upsetting. By misleading future entrepreneurs, we put them in an untenable position. Either take the stuff you read and hear at face value or gain access to a wizard who can guide you to the true path (and hope you don’t get taken in). But this false choice is not the only way forward. As usual, I think there’s an opportunity for our industry to do better.

To be clear, I don’t blame any particular actor in the system for doing what they’re doing. Paul Graham is getting his startups the best advice he can in the best way he knows: by giving them exclusive access to off-the-record conversations that nobody else is privy to. And it’s not reasonable to blame him for their behavior when he gives them the stage at events like Startup School. Similarly, journalists print vanity metrics because that’s all companies will release. And companies crave positive PR and control over their message – all for rational reasons.

And yet you who are reading this right now have tremendous power. You are the intended audience for all of those expenditures of energy. You are the “hits” that websites crave, the followers and the RSS subscribers. Where you put your attention and what you do with it matters.

So I’d like to suggest the following: let’s stop giving lying on stage and vanity metrics a free pass. I think if we can delegitimize this behavior, we can make it stop. We’ll need to ask tougher questions, though: how do you know your company’s success was caused by X? what else was happening around that time that might have caused Y? what did you think was going to happen when you did Z?

Journalists have the highest obligation to ask these kinds of questions, and conferences organized by journalists ought to be the exemplars the rest of us look up to. And yet, today many are not. I am sympathetic, because I have faced this problem, too. If you ask tough questions, are unwilling to help people craft their message, and are skeptical of vanity metrics, you can’t get the high profile guests. That means less attention, less coverage, fewer readers, and lower sponsorship dollars. Assuming, of course, that all of you give your attention disproportionately to famous people.

But if you’re attending a conference or reading a magazine, you aren’t bound by those rules. You can ask any question you want. And you can reward the speakers who tell the whole truth with your support.  Every time you do that, you’re helping make our industry a better place.

(Have a favorite real startup story? Share it in the comments and we can start giving those people some much-deserved attention right away.)