Lo, my 18891 subscribers, who are you?

Way back when I first started blogging, I was amazed to have any subscribers at all. In fact, when I crossed the five subscriber threshold, I was moved to write about my excitement and about the advantages of having a pathetically small number of customers. And, in the survey that was attached, I learned a lot of useful information about how to make this blog better. Since then, I've tried to make it a regular habit to collect real data about what you think. Welcome to the latest installment.

Since the last time, your support has been overwhelming: your ranks have grown almost a full order of magnitude. To all of you who've taken the time to subscribe, who have convinced a friend to subscribe, or even talked your co-workers into it, let me say thank you. Your support makes everything I do possible, and I won't ever get tired thanking you for it.

For those who have been readers for some time, the survey below will seem familiar. I sometimes use these surveys to test new product ideas and even to engage in some customer validation. Today is no exception. If you're curious, please take the survey to learn more. I don't want to say anything here that would bias the results.

As usual, I'll share what I learn and how it affects what I'm up to next. Thanks again for stopping by. Without further ado:

Click here to take survey

Case Study: Continuous deployment makes releases non-events

The following is a case study of one entrepreneur's transition from a traditional development cycle to continuous deployment. Many people still find this idea challenging, even for companies that operate solely on the web. This case presents a further complication: desktop software. Without being able transparently modify the software in situ, is it still possible to deploy on a continuous basis? Read on to find out.


Ash Maurya is the founder of WiredReach, a bootstrapped startup that he has been running for seven years. Recently, he was bitten by the lean startup bug and has started writing about his experiences attempting to apply lean startup and customer development principles. I've previously named his post Achieving Flow in a Lean Startup as one of my favorite blog posts of 2009. 

What follows is his own account of the challenges he faced as well as the solutions he adopted, lightly edited for style. If you're interested in contributing a case study for publication here, consider getting started by adding it to the Lean Startup Wiki case study section. -Eric

Of all the Lean Startup techniques, Continuous Deployment is by far the most controversial. Continuous Deployment is a process by which software is released several times throughout the day – in minutes versus days, weeks, or months. Continuous Flow Manufacturing is a Lean technique that boosts productivity by rearranging manufacturing processes so products are built end-to-end, one at a time (using singe-piece flow), versus the more prevalent batch and queue approach.

Continuous Deployment is Continuous Flow applied to software. The goal of both is to eliminate waste. The biggest waste in manufacturing is created from having to transport products from one place to another. The biggest waste in software is created from waiting for software as it moves from one state to another: Waiting to code, waiting to test, waiting to deploy. Reducing or eliminating these waits leads to faster iterations which is the key to success.

My transition to Continuous Deployment
Prior to adopting continuous deployment, I used to release software on a weekly schedule (come rain or shine) which I viewed as pretty agile, disciplined, and aggressive. I identified the must-have code updates on Monday, official code cutoff was on Thursday, and Friday was slated for the big release event. The release process took at least half a day and sometimes the whole day. Dedicating up to 20% of the week on releasing software is incredibly wasteful for a small team. This is not counting the ongoing coordination effort also needed in prioritizing the ever-changing release content for the week as new critical issues are discovered. Despite these challenges, I fought the temptation to move to a longer bi-weekly or monthly release cycle because I wanted to stay highly responsive to customers (something our customers repeatedly appreciate). Managing weekly releases got a lot harder once I started doing customer development. Spending more time outside the building, meant less time for coding, testing, and deploying. Things started to slip. That is when I devised a set of work hacks to manage my schedule (described here) and what drove me to adopting Continuous Deployment.

My transition from staged releases to continuous deployment took roughly 2 weeks. I read Eric Ries' 5 step primer to getting started with Continuous Deployment and found that I already had a lot of the necessary pieces. Continuous integration, deployment scripts, monitoring, and alerting are all best practices for any release process - staged or continuous.

The fundamental challenge with Continuous Deployment is getting comfortable with releasing all the time.

Continuous deployment makes releases non-events and checking in code is synonymous with triggering a release. On the one hand, this is the ultimate in customer responsiveness. On the other hand, it is scary as hell. With staged releases, time provides a (somewhat illusory) safety net. There is also comfort in sharing test responsibility with someone else (the QA team). No one wants to be solely responsible for bringing a production system down. For me neither was a consideration. I didn't have time or a QA team.

I took things easy at first - made small changes and audited the release process maniacally. I started relying heavily on functional tests (over unit tests) which allowed me to test changes as a user would. I also identified a set of events that would indicate something terribly going wrong (e.g. no users on the system) and built real-time alerting around them (using nagios/ganglia). As we built confidence, we started committing bigger and multi-part changes, each time building up our suite of testing and monitoring scripts. After a few iterations, our fear level was actually lower than how we used to feel after a staged release. Because we were committing less code per release, we could correlate issues to a release with certainty.

These days, we never wonder if unexpected errors could have been introduced as a result of a large code merge (since there is no branching. We also rely on more testing and monitoring automation, which is way more robust and consistent than what we were doing before.

All that said, mistakes are still made and we commit bad code now and then. None that have taken the system down (not yet anyway). Rather than seeing these as a shortcoming of the process, we view it as an opportunity to build up our Cluster Immune System. We try and follow a Five Whys approach to keep these errors from recurring. There is always some action to take: writing more tests, more monitoring, more alerts, more code, or more process.

Looking back, struggled to balance the opposing pulls of "outside the building" versus "inside the building" activities. Adopting Continuous Deployment has allowed me to build "flow" into my day which allows me to do both. But easier releases are not the only benefit of Continuous Deployment. Smaller releases lead to faster build/measure/learn loops. I've used these faster build/measure/learn loops to optimize my User Activation flow, delight customers with "near-instant" fixes to issues, and even eliminate features that no one was using.

While it is somewhat easier to continuously deploy web based software, with a little discipline, desktop based software too can be built to flow. Here's how I am implement continuous deployment for my desktop-based application (CloudFire).

My Continuous Deployment process

Don't push features

If you've followed a customer discovery process, identified a problem worth solving, and built out your minimum viable product, DON'T keep adding features until you've validated the MVP, or more specifically the unique value proposition of the MVP. Unneeded features are waste and not only create more work but can needlessly complicate the product and prolong the "customer validation" phase.

Every new feature should ideally be pulled by more than one customer before showing up in a release.

Build in response to a signal from the "customer", and otherwise rest or improve.

As a technologist, I too love to measure progress based on how much stuff I build. But instead of channeling all my energy towards building new features, I channel roughly 80% of it towards measuring and optimizing existing features. I am not advocating adding no features at all. Users will naturally ask for more stuff and your MVP by definition is minimal and needs more love. Just don't push it.

Code in small batches

I've previously described my 2 hour blocks of maker time for maximizing my work "flow". Prior to starting any maker activity, I clearly identify what needs to get done (the goal) and sketch out how it needs to get done (the design).

It is important to point out that the goal of the maker activity need not be a user facing feature or even a complete feature. There is inherent value in committing incremental work into production to diffuse future integration surprises. During the maker activity, I code, unit test, and create or update functional tests, as needed. At the end of the maker activity, I check-in code which automatically triggers a build on a continuous integration server that is then run through a battery of unit and functional tests. The artifacts created at the end of the build are installers for mac and windows (for new users) along with an Eclipse P2 repository (OSGI) for automatic software updates (for current users). The release process takes ~15 minutes and runs in the background.

Prefer functional tests over unit tests whenever possible

I don't believe in blindly writing unit tests just to achieve 100% code coverage as reported by some tool. To do that I would have to mock (simulate) too many critical components. I deem excessive unit testing a form of waste. Whenever possible, I rely on functional tests that verify user actions. I use Selenium, which lets me control the application on multiple browsers and OS platforms, just as a user would. One thing to be wary of is that functional tests are longer running than unit tests and will gradually increase the release-cyle-time. Parallelization of tests with multiple test machines is a way to address this. I am not at that point yet but Selenium Grid looks like a good option. So does Go Test It.

Always test the User Activation flow
After the integration tests are run and the software packaged, I always verify my User Activation flow before going live. The user activation flow is the most critical path towards achieving initial user gratification or product/market fit. My user activation flow is automatically tested on both a mac and windows machine.

Utilize automagic software updates
A major challenge with desktop-based (versus web-based) software is propagating software updates. Studies have shown that users find traditional software update dialogs annoying. To overcome this, I am using a software update strategy that works silently without ever interrupting the user, much like an appliance. Google Chrome utilizes a similar update process. The biggest risk with this approach is that users will find it Orwellian. So far no one has complained, and many users like the auto-update feature. It helps that CloudFire, being a p2web app, runs headlessly with a browser-based UI.

This is how the software update process currently works:

1. At the end of each build, we push an Eclipse P2 repository (OSGI) which is a set of versioned plug-ins that make up the application. Because the application is composed of many small plug-ins, coupled with the fact that we commit small code batches, the size of each software update can be downloaded quickly.
   
2. Every time the user starts up the application, it checks for a new update, downloads and installs one if available. Depending on the type of update, it could take effect immediately or require an application restart. If an application restart is required, we wait until the next user initiated relaunch of the application or trigger one silently when the system is idle.
   
3. If the application is already running, it periodically polls for new updates. If an update is found, it is also downloaded and installed in the background (as above) without interrupting the user.
   

Alerts and monitoring

I use nagios and ganglia to implement both system and application level monitoring and alerting on the overall health of the production cluster. Examples of things I monitor are the numbers of user activations, active users, and aggregate page hits to user galleries. Any out-of-the-norm dip in these numbers immediately alerts us (via twitter/SMS) to a potential issue.

Application level diagnostics

Despite the best testing, defects still happen. More testing is not always the answer as some defects are intermittent and a function of the end-user's environment. It is virtually impossible to test all combinations of hardware, OS, browsers, and third party apps (e.g. Norton anti-virus, Zone Alarm, etc.).

Relying on users to report errors doesn't always work in practice. To compensate, we've had to build  basic diagnostics right into the application itself. They can notify both the user and us of unexpected errors, and allow us to pull configuration information and logs remotely. We can also do remote rollbacks this way.

Tolerate unexpected errors exactly once

Unexpected errors provide the opportunity to learn and bullet-proof a system early. Ignoring them or implementing quick-and-dirty patches inevitably lead to repeat errors which are another form of waste. I try and follow a formalized Five Why's process (using our internal wiki) for every error. This forces us to really stop, think, and fix the right problem(s).

My continuous deployment process is summarized below:

So why is Continuous Deployment so controversial?

Eric has addressed a lot of the objections already on his blog. One that I hear a lot is the belief that you need a massive team to pull off continuous deployment. I would argue that the earlier in the development cycle and the smaller the team, the easier it is to implement a continuous deployment process. If you are a start-up with a MVP, there is no better time to adopt a continuous deployment process than the present. You don't yet have hundreds of customers, dozens of peers, or dozens of features. It is a lot easier to lay the groundwork now with time on your side.

If you enjoyed Ash's writing in this case study, I suggest you subscribe to his blog. -Eric

Two Ways to Hold Entrepreneurs Accountable (for Harvard Business Review)

The next part in the series I am writing for Harvard Business Review is online. This time, I'm discussing the challenge for corporate CFO's and VC's alike in holding entrepreneurs accountable. Of course, the method I recommend, that of using quantified learning as a yardstick, is of equal interest to disciplined entrepreneurs. After all, the hardest question to answer in entrepreneurship is: are we making progress?

As usual, if you would take an extra two minutes of your day to click through and join the discussion on HBR's site, I would appreciate it. Leave a comment or reply to someone else. Fostering that dialog is important.

Two Ways to Hold Entrepreneurs Accountable - The Conversation - Harvard Business Review

Way back when the money was doled out, the team made a compelling pitch about the large market that was going to adopt their new innovative product or service. The rules of a fundraising pitch are straightforward: pitch the largest "total available market" that you can credibly claim to be capturing. Behind this analysis is a spreadsheet model, complete with detailed metrics for a set of customer behaviors that show just how valuable the new product will be.

And when companies pursue sustaining innovations — like a product line extension or a new technology designed to serve an existing customer segment — this procedure makes complete sense. A good general manager is expected to use all the tools at their disposal — market research, competitive analysis, customer focus groups — to figure out a plan that will work. The logic of the CFO when presented with such a plan is also straightforward: once the plan is approved, either the team will deliver the promised results, or the project can be safely shut down. Safely because it is clear that the manager in question didn't do his homework.

This whole framework breaks down when teams confront entrepreneurial situations in which they're trying to build something new under conditions of extreme uncertainty. This is the domain of disruptive innovation, projects which are inherently low-ROI in the short term. They are long-term bets on the development of a new line of business, a new technology platform, or the creation of a new market.

Let's return to our team that's failing to hit their targets. They are on-schedule and on-budget, but their gross metrics are way off. Usually, they are delivering only a fraction of the revenue they promised. For a little while, the team can resort to the last defense of entrepreneurs in trouble: the promised hockey-stick.

One thing that is often overlooked about the hockey-stick growth shape: its most distinctive characteristic is the long, flat part.

 Read the rest here... Two Ways to Hold Entrepreneurs Accountable

Amazing lean startup resources

A year ago, there was no lean startup movement. The term was known by maybe a few dozen people. Being an evangelist for these ideas earned me a regular diet of funny looks and patronizing comments. At that time, my wildest dreams did not include even a fraction of what's happened since.

I continue to believe that the explosion of interest in the lean startup has very little to do with me. Rather, I see it as a reflection of a worldwide openness to new ideas about entrepreneurship. Recent economic events, technological change, and the rapid diffusion of information about the old models have combined to help us all realize just how important entrepreneurship is - and just how little we really know about it.

Beyond my own efforts on this blog (and more), there is now an amazing variety of resources for lean startup practitioners. For those that are following me on Twitter, you've probably seen many of these. But I wanted to create a centralized directory. If you are attempting to apply lean startup ideas in your own business - you are not alone.

The Lean Startup Wiki (hosted by long-time lean startup pbWorks) contains a number of resources, including links to case studies, meetups, tools, and people who have volunteered to help. It's a wiki, feel free to consume and enhance it.

Rich Collins organized the Lean Startup Circle mailing list, the most robust discussion forum for lean startup ideas anywhere, with over two thousands members already. For the many entrepreneurs that send me cold emails asking for me to review a business plan or answer a strategic dilemma: I'm much more likely to answer if you've already tried getting an answer on the mailing list. You'll probably get a better result, anyway.

The #leanstartup hashtag on twitter has become a firehose of information, but seems to feature great new links almost every day. I also use it to collect feedback from all my talks and presentations, so it's a great place to find out what real people think about the ideas.

Rich also organized the first Lean Startup Meetup right here in San Francisco. The SF Meetup now has over five hundred members, and has spawned many other meetup events. There's probably one in a city near you. Some of the most active: SF, New York, DC, Chicago - but there are many more listed here. I also recommend SKMurphy's excellent Bootstrappers Breakfast series, which is now in multiple cities. Not coincidentally, they were one of the first organizations to host me as a speaker.

If you want to create or join a meetup near you, just leave a note in the appropriate section of the meetup directory. There are efforts underway in more than a dozen cities to start a regular meeting. If you need a first speaker, I'm happy to join by skype.

There are also an impressive array of bloggers writing about these ideas. When I first started blogging, the startup blogging mafia immediately came to meet me and find out who I was: Dave McClure, Andrew Chen, Sean Ellis and Venture Hacks. My success is due in no small part to their early and enthusiastic endorsement. And Steve Blank (a long-time mentor) has made the leap to blogging in impressive fashion.

Yet beyond these usual suspects are a huge number of up-and-coming bloggers, most of whom are practicing entrepreneurs willing to share their lean startup stories. Some of my favorites: Ash Maurya (whose Achieving Flow in a Lean Startup was one of my favorite posts of 2009), Brant Cooper, CindyAlvarez, Laura Klein, Kevin Dewalt, Giff Constable - and I must be missing many more. Have a favorite who I overlooked? Please share in a comment.

Last, I've tried to keep this blog updated with events, slides, audio, video and books that are helpful as well. 


Did I miss anything? Please feel free to use the comments on this post to share any and all resources you've found helpful. Most of all, thank you for your continuing support. Together we can change our  industry for the better.

Is Entrepreneurship a Management Science? (for Harvard Business Review)

I'm excited to have just published the first of several articles on entrepreneurship for the Harvard Business Review online. Although much of it will sound familiar to readers of this blog, what's new is my first step towards the "new entrepreneurship" that is my goal for 2010.

My explicit goal in working with HBR is to foster a dialog between entrepreneurs and more traditional general managers. I think we have a lot to learn from each other. So, I'd like to ask you a favor. Unlike previous cross-posting I've done for blogs, where a lot of the discussion happens here, I'd like to ask you to take a minute and click through to read the full article. If you can spare an additional two minutes, would you leave a comment or - better yet - reply to one? I'd be honored to have you join the conversation.

Without further ado...

Is Entrepreneurship a Management Science? - The Conversation - Harvard Business Review

For most of us, the phrase 'management science' conjures up a decidedly non-entrepreneurial image, and for good reason. The preeminent management science, for much of the twentieth century was general management, pioneered by twentieth century giants like Peter Drucker and Alfred P. Sloan. As an entrepreneur, I often saw the best practices of general management fail startups. Applied out of context, they were dangerous.

This conflict has played out in many companies that I've worked with. My most recent startup created a marketplace for customers to buy and sell virtual goods for their 3D avatar. So you can imagine how I expected some skepticism when pitching ideas about technology innovation to, say, the U.S. Army. This was my first surprise: they understood that innovation needs to be understood at the level of principles, not just tactics.

Read the rest at The Conversation - Harvard Business Review...

Towards a new entrepreneurship

When I started writing about the lean startup, my aspiration was to do more than just share a handful of tips and tricks that work for consumer internet startups. I believe the only way to improve our chances as entrepreneurs is to develop a working theory of entrepreneurship. This belief led me to the lean startup, and to an amazing 2009. I owe a tremendous debt of gratitude to everyone who's helped to make that a reality, especially you who are reading this right now. So far, 2010 is shaping up to be an even more amazing year; read on to find out why.

In a forthcoming article I've written advocating for the Startup Visa, I wrote this:

Like other industries – from publishing to automobiles – entrepreneurship is in the process of being disrupted by globalization. On the whole, this is a good thing for America and for our civilization. The cost of creating new companies is falling rapidly, and access to markets, distribution, and information is within the reach of anyone with an internet connection. The result is a profound democratization of the digital means of production.

Re-reading it recently, I was struck by an idea that I don't think is too widespread yet: that entrepreneurship is an industry. Sure, when entrepreneurs create startups that grow up into mature companies, they become part of an established industry, with its own ecosystem, norms, partners and best practices. But until that happens, we entrepreneurs have our own ecosystem, of investors and service providers, norms and even some "best" practices. The two ecosystems have diverged significantly in the past fifty years - and especially in the past ten. The reason is that the underlying theory that powers established business, the theory of general management, is increasingly inadequate for managing startups. And yet, so far, we lack a coherent theory to replace it. My belief is that the lean startup is that theory. Together, we are part of a movement that is redefining entrepreneurship.

Every once in a while, I get an inquiry from a startup asking if I'm ready to "jump back in the game" and start a company. My realization for 2010 is that I'm already in the game. This is just like a startup to me, complete with distribution and product development challenges, ecosystem design and many, many customers. Only now the stakes are higher: my aspiration is to impact our entire industry. So my priorities for 2010 are projects that will expand the lean startup industry-wide. As always, I'll look to you early adopters for guidance and help. Here's a preview of what I'm working on.

1. The Lean Startup Cohort program. The more I work in the realm of theory, the more I crave the messy realities of thorny startup problems. In 2010, I hope to work with a limited number of high-growth startups in an intense fashion. This is by far the most difficult project I've undertaken, but I believe the impact will be worth it.
   
   
2. Teaching at Berkeley (and beyond). Steve Blank has asked me to co-teach his MBA entrepreneurship class at Berkeley Haas, starting January 19, and to revamp the curriculum to make it a full introduction to the lean startup. This is just the first of several academic projects I hope to work on in 2010. Academia has a vital role to play in the new entrepreneurship, answering many questions that need urgent research attention.
   
   How you can get involved: Steve has always had a very liberal policy on auditors. If you'd like to attend this class at Berkeley, just let me know in a comment. If there's sufficient interest, I'll post details.
   
   Even if you can't attend, you can help shape the curriculum. Do you have a suggestion for a guest speaker or case study that illustrates a particular lean startup concept: minimum viable product, continuous deployment, validated learning, the pivot? Please leave your suggestions as a comment. Bonus points if you can find a relevant official HBS case study from their archives to suggest as well.
   
   
3. Startup Lessons Learned conference. This is still in the early planning stages, so I don't have many details to share yet. I've been blown away by the level of demand you all have expressed in having a forum for the whole community to come together and share what we're learning. My hope is to create a one-day event in San Francisco sometime in the spring. Details soon, I promise. In the meantime, feel free to nominate speakers or notable companies you'd like to hear from in the comments.
   
   For those that can't travel to SF, I'm also working on a way to simulcast to event to cities where there's sufficient interest. If there's one thing I've learned in all my travel in 2009, it's that entrepreneurship has gone global in a big way, and I'd like everyone to be able to participate in this event.
   
   How you can get involved: We're going to need volunteers to help spread the word and advise on topics we should cover. If you'd like to be part of this, let me know in a comment. Similarly, let me know if you'd like to host or attend a simulcast party in your town.
   
   
4. Writing a book. I'm just starting the publishing process that should lead to a book for a general business audience in 2011. Talk about long cycle times! Not much to share on this front yet, but I will especially need your help to get the word out about this.
   
   How you can get involved: If you'd like to be part of the early-reader and evangelizing team for this, let me know in a comment. I'm especially interested in people who are interested in helping with pre-orders.
   

Most of all, the best thing you can do to help this movement grow is to be an entrepreneur yourself. As you face new challenges, successes and failures - please share what you learn with the rest of us. As always, I'm eager to join the discussion in the new year. Have feedback on what you've heard so far? Let's hear it: good, bad, crazy? Let's get 2010 underway.

To kick it off, take a look at this propaganda video on the new entrepreneurship, courtesy of Dave McClure, KISSmetrics, and GoaP:



Update Jan 4: Replaced embedded video with YouTube version. Also, clarified details for auditing the Berkeley class in the comments below.