Achieving a failure

We spend a lot of time planning. We even make contingency plans for what to do if the main plan goes wrong. But what if the plan goes right, and we still fail? This is the my most dreaded kind of failure, because it tricks you into thinking that you're in control and that you're succeeding. In other words, it inhibits learning. My worst failures have all been of this kind, and learning to avoid them has been a constant struggle.

See if this plan sounds like a good one to you:

• Start a company with a compelling long-term vision. Don't get distracted by trying to flip it. Instead, try and build a company that will matter on the scale of the next century. Aim to become the "next AOL or Microsoft" not a niche player.
• Raise sufficient capital to have an extended runway from experienced smart money investors with deep pockets who are prepared to make follow-on investments.
• Hire the absolute best and the brightest, true experts in their fields, who in turn can hire the smartest people possible to staff their departments. Insist on the incredibly high-IQ employees and hold them to incredibly high standards.
  
• Bring in an expert CEO with outstanding business credentials and startup experience to focus on relentless execution.
• Build a truly mainstream product. Focus on quality. Ship it when it's done, not a moment before. Insist on high levels of usability, UI design, and polish. Conduct constant focus groups and usability tests.
• Build a world-class technology platform, with patent-pending algorithms and the ability to scale to millions of simultaneous users.
  
• Launch with a PR blitz, including mentions in major mainstream publications. Build the product in stealth mode to build buzz for the eventual launch.

I had the privilege, and the misfortune, to be involved with a startup that executed this plan flawlessly. It took years, tens of millions of dollars, and the efforts of hundreds of talented people to pull it off. And here's the amazing thing about this plan: it actually worked. I think we accomplished every one of those bullet points. Check. Mission accomplished.

Only this company was a colossal failure. It never generated positive returns for its investors, and most of its employees walked away dejected. What went wrong?

This company was shackled by shadow beliefs that turned all those good intentions, and all that successful execution, into a huge pile of wasted effort. Here are a few:

• We know what customers want. By hiring experts, conducting lots of focus groups, and executing to a detailed plan, the company became deluded that it knew what customers wanted. I remember vividly a scene at a board meeting, where the company was celebrating a major milestone. The whole company and board play-tested the product to see its new features first hand. Everyone had fun; the product worked. But that was two full years before any customers were allowed to use it. Nobody even asked the question: why not ship this now? It was considered naive that the "next AOL" would ship a product that wasn't ready for prime time. Stealth is a customer-free zone. All of the efforts to create buzz, keep competitors in the dark, and launch with a bang had the direct effect of starving the company for much-needed feedback.
  
  
• We can accurately predict the future. Even though some aspects of the product were eventually vindicated as good ones, the underlying architecture suffered from hard-to-change assumptions. After years of engineering effort, changing these assumptions was incredibly hard. Without conscious process design, product development teams turn lines of code written into momentum in a certain direction. Even a great architecture becomes inflexible. This is why agility is such a prized quality in product development.
  
  
• We can skip the chasm. As far as I know, there are no products that are immune from the technology life cycle adoption curve. By insisting on building a product for mainstream customers, the company guaranteed that they would be unhappy with the number and type of customers they got for the first version. Worse was the large staff in departments appropriate to a mainstream-scale product, especially in customer service and QA. The passionate early adopters who flocked to the product at its launch could not sustain this outsized burn rate.
  
  
• We can capitalize on new customers. As with many Silicon Valley failures, a flawless PR launch turned into a flawed customer acquisition strategy. The first version product wasn't easy enough to use, install, and pay for. It also had hardware requirements that excluded lots of normal people. Millions of people flocked to the website, but the company could only successfully monetize early adopters. As a result, the incredible launch was mostly wasted.
  
  
• We know what quality means. All of the effort invested in quality, polish, stability and usability turned out to be for nothing. Although the product was superior to its competitors in many ways, it was missing key features that were critical for the kinds of customers who never got to participate in the company's focus groups (or were represented on its massive QA staff). Worse, many of the wrong assumptions built into the technical architecture meant that, in the real world outside the testing lab, the product's stability was nothing to write home about. So despite the millions invested in quality, the end result for most customers was no better than the sloppy beta-type offerings of competitors.
  
  
• Advancing the plan is progress. This is the most devastating thing about achieving a failure: while in the midst of it, you think you're making progress. This company had disciplined schedules, milestones, employee evaluations, and a culture of execution. When schedules were missed, people were held accountable. Unfortunately, there was no corresponding discipline of evaluating the quality of the plan itself. As the company built infrastructure and added features, the team celebrated these accomplishments. Departments were built and were even metrics-driven. But there was no feedback loop to help the company find the right metrics to focus on.

These shadow beliefs have a common theme: a lack of reality checks. In my experience, great startups require humility, not in the personal sense, but in the organizational capacity to emphasize learning. A good learning feedback loop trumps even the best execution of a linear plan. And what happened with this ill-fated company? Although it failed, many of the smart people involved have accomplished great things. I know of at least five former employees that went on to become startup founders. They all got a tremendous first-hand lesson in achieving a failure, all on someone else's dime (well, millions of dimes). As we move into a new economic climate, it's my hope that our industry will stop this expensive kind of learning and start building lean startups instead.

The interesting thing about an analysis like this is that it seems obvious in retrospect. A lot of people say that they know that they don't know what customers want. And yet, if you go back and look at the key elements of the plan, many of them are in subtle conflict with the shadow beliefs. Understanding that tension requires a lot of reflection and a good teacher. I myself didn't understand it until I had the opportunity to view that failure through the lens of the customer development theory. I had a big advantage, because Steve Blank, the father of customer development, got to see the failure up close and personal too as an investor and board member. A much less painful way to learn the lesson is read his book: The Four Steps to the Epiphany.

Refactoring yourself out of business

Let me start out by saying I am a big fan of refactoring, the ongoing process of changing code so that it performs the same behaviors but has more elegant structure. It's an essential discipline of good software development, especially in startups. Nonetheless, I want to talk about a dysfunction I've seen in several startups: they are literally refactoring themselves to death.

Here's a company I met with recently. They have a product that has a moderate number of customers. It's well done, looks professional, and the customers that use it like it a lot. Still, they're not really hitting it out of the park, because their product isn't growing new users as fast as they'd like, and they aren't making any money from the current users. They asked for my advice, and we went through a number of recommendations that readers of this blog will already be able to guess: adding revenue opportunities, engagement loop optimization, and some immediate split-testing to figure out what's working and what's not. Most of all, I encouraged them to start talking to their most passionate customers and running some big experiments based on that feedback.

I thought we were having a successful conversation. Towards the end, I asked when they'd be able to make these changes, so that we could meet again and have data to look at together. I was told they weren't sure, because all of their engineers were currently busy refactoring. You see, the code is a giant mess, has bugs, isn't expandable, and is generally hard to modify without introducing collateral damage. In other words, it is dreaded legacy code. The engineering team has decided it's reached a breaking point, and is taking several weeks to bring it up to modern standards, including unit tests, getting started with continuous integration, and a new MVC architecture. Doesn't that sound good?

I asked, "how much money does the company have left?" And it was this answer that really floored me. They only have enough money to last another three months.

I have no doubt that the changes the team is currently working on are good, technically sound, and will deliver the benefits they've claimed. Still, I think it is a very bad idea to take a large chunk of time (weeks or months) to focus exclusively on refactoring. The fact that this time is probably a third of the remaining life of the company (these projects inevitably slip) only makes this case more exaggerated.

The problem with this approach is that it effectively suspends the company's learning feedback loop for the entire duration of the refactoring. Even if the refactoring is successful, it means time invested in features that may prove irrelevant once the company starts learning again. Add to that the risk that the refactoring never completes (because it becomes a dreaded rewrite).

Nobody likes working with legacy code, but even the best engineers constantly add to the world's store of legacy code. Why don't they just learn to do it right the first time? Because, unless you are working in an extremely static environment, your product development team is learning and getting better all the time. This is especially true in startups; even modest improvements in our understanding of the customer lead to massive improvements in developer productivity, because we have a lot less waste of overproduction. On top of that, we have the normal productivity gains we get from: trying new approaches on our chosen platform to learn what works and what doesn't; investments in tools and learning how to use them; and the ever-increasing library of code we are able to reuse. That means that, looking back at code we wrote a year or two ago, even if we wrote it using all of our best practices from that time, we are likely to cringe. Everybody writes legacy code.

We're always going to have to live with legacy code. And yet it's always dangerous to engage in large refactoring projects. In my experience, the way to resolve this tension is to follow these Rules for Refactoring:

• Insist on incremental improvements. When sitting in the midst of a huge pile of legacy code, it's easy to despair of your ability to make it better. I think this is why we naturally assume we need giant clean-up projects, even though at some level we admit they rarely work. My most important lesson in refactoring is that small changes, if applied continuously and with discipline, actually add up to huge improvements. It's a version of the law of compounding interest. Compounding is not a process that most people find intuitive, and that's as true in engineering as it is in finance, so it requires a lot of encouragement in the early days to stay the course. Stick to some kind of absolute-cost rule, like "no one is allowed to spend more time on the refactoring for a given feature than the feature itself, but also no one is allowed to spend zero time refactoring." That means you'll often have to do a refactoring that's less thorough than you'd like. If you follow the suggestions below, you'll be back to that bit of code soon enough (if it's important).
  
  
• Only pay for benefits to customers. Once you start making lots of little refactorings, it can be tempting to do as many as possible, trying to accelerate the compounding with as much refactoring as you can. Resist the temptation. There's an infinite amount of improvement you can make to any piece of code, no matter how well written. And every day, your company is adding new code, that also could be refactored as soon as it's created. In order to make progress that is meaningful to your business, you need to focus on the most critical pieces of code. To figure out which parts those are, you should only ever do refactoring to a piece of code that you are trying to change anyway.
  
  For example, let's say you have a subsystem that is buggy and hard to change. So you want to refactor it. Ask yourself how customers will benefit from having that refactoring done. If the answer is they are complaining about bugs, then schedule time to fix the specific bugs that they are suffering from. Only allow yourself to do those refactoring that are in the areas of code that cause the bug you're fixing. If the problem is that code is hard to change, wait until the next new feature that trips over that clunky code. Refactor then, but resist the urge to do more. At first, these refactoring will have the effect of making everything you do a little slower. But don't just pad all your estimates with "extra refactoring time" and make them all longer. Pretty soon, all these little refactorings actually cause you to work faster, because you are cleaning up the most-touched areas of your code base. It's the 80/20 rule at work.
  
  
• Only make durable changes (under test coverage). There's no point refactoring code if it's just going to go back to the way it was before, or if it's going to break something else while you're doing it. The only way to make sure refactorings are actually making progress (as opposed to just making work) is to ensure they are durable. What I mean is that they are somehow protected from inadvertent damage in the future.
  
  The most common form of protection is good unit-test coverage with continuous integration, becaus that makes it almost impossible for someone to undo your good work without knowing about it right away. But there are other ways that are equally important. For example, if you're cleaning up an issue that only shows up in your production deployment, make sure you have sufficient alerting/monitoring so that it would trigger an immediate alarm if your fix became undone. Similarly, if you have members of your team that are not on the same page as you about what the right way to structure a certain module is, it's pointless to just unilaterally "fix" it if they are going to "fix" it right back. Perhaps you need to hash out your differences and get some team-wide guidelines in place first?
  
  
• Share what you learn. As you refactor, you get smarter. If that's not a team-wide phenomenon, then it's still a form of waste, because everyone has to learn every lesson before it starts paying dividends. Instead of waiting for that to happen, make sure there is a mechanism for sharing refactoring lessons with the rest of the team. Often, a simple mailing list or wiki is good enough, but judge based on the results. If you see the same mistakes being made over again, intervene.
  
  
• Practice five whys. As always with process advice, I think it's essential that you do root cause analysis whenever it's not working. I won't recap the five-why's process here (you can read a previous post to find out more); the key idea is to refine all rules based on the actual problems you experience. Symptoms that deserve analysis include: refactorings that never complete, making incremental improvements but still feeling stuck in the mud, making the same mistakes over and over again, schedules slipping by increasing amounts, and, of course, month-long refactoring projects when you only have three months of cash burn left.

Back to my conversation with the company I met recently. Given their precarious situation, I really struggled with what advice to give. On the one hand, I think they have an urgent problem, and need to invest 100% of their energy into finding a business model (or another form of traction). On the other, they already have a team fully engaged on making their product architecture better. In my experience, it's very hard to be an effective advocate for "not refactoring" because you can come across as anti-quality or even anti-goodness. In any event, it's enormously disruptive to suddenly rearrange what people are working on, no matter how well-intentioned you are.

I did my best to help, offering some suggestions of ways they could incorporate a few of these ideas into their refactoring-in-progress. At a minimum, they could ensure their changes are durable, and they can always become a little more incremental. Most importantly, I encouraged the leaders of the company to bring awareness of the business challenges to the product development team. Necessity is the mother of invention, after all, and that gives me confidence that they will find answers in time.

Three freemium strategies

I'm excited to see so much attention being paid to freemium businesses lately. These are companies that generate revenue by offering a free product with an upsell or premium version. Their economics blends elements of the free, advertising-supported, "eyeballs" business with more traditional e-commerce and subscription businesses. For founders, I think it also has another big attraction: the ability to avoid a lot of "free vs paid" arguments. You can reach for all the scale of a free service and still make money. Pretty good deal, right?

Andrew Chen recently did a great article on the economic model underlying freemium (it includes a detailed spreadsheet, too!). In it, he mentions the core dilemma that all freemium companies face:

the key is to create the right mix of features to segment out the people who are willing to pay, but without alienating the users who make up your free audience. Do it right, and your conversion rates might be as high as 20%. Do it wrong, and your LTV gets very close to zero. This is why premium features have to be built into the core of a freemium business, rather than added in at the end. You want to be right at the balance between free and ‘mium!

Having worked in the freemium business for a number of years, I can safely report that struggling with these questions never goes away. I have sat in hundreds of meetings about what to give away and what to charge for, and they are not easy. Things get a lot worse when we don't have a coherent model of why we're using freemium in the first place. This is a common outcome I've observed among founding teams that take the "split the difference" route to freemium.

The way to resolve this tension is to agree on a fundamental freemium strategy, that leverages the free part of your business to add real value. In my experience, there are three basic choices:

1. Free serves paid. In this model, your free users trade their time for the benefit of your paying customers. For example, Puzzle Pirates does a great job of allowing trading between two different currencies: one that is earned with time, and another that is earned by paying. Both currencies are valuable, and free users can trade theirs to the paying customers, who are allowed to access benefits that would otherwise take a long time to achieve. Other examples are user-generated content sites, where free users create content that is valuable to paying customers. The key to this model is to make sure that customers who create value are heavily rewarded, and those that don't create value are marginalized, so that they have a strong incentive to pay.
   
   
2. Free trial. The original freemium model, where customers are given a certain amount of time before being forced to either quit the service or pay. The big question is when to pull the plug - too soon, and you risk customers not being sufficiently addicted to say yes; too late and you risk giving all the value away for free. Luckily, once you realize you are in this business, you can answer these questions empirically with good split-testing and linear optimization.
   
   
3. Free as inventory. Some businesses actually sell access to their free users. I think this is the right way to think about many advertising businesses, like Google AdWords. Some dating sites work this way too, where you can post your profile for free, and people pay only when they want to contact you. This ensures that the most popular people get lots of value without having to pay. Here the key is for your free customers to get value from the site that is greater than the costs they perceive by the fact that you're selling access to them. Google has shown their awareness of this issue by carefully managing the annoyance-factor of ads that are shown during search. What differentiates this model from "free serves paid" is that the free users don't need to consciously do anything special to be valuable. For network-effects businesses, like Skype, just being on the network is enough to create value.

Strategy is all about what you're not going to do; for a freemium business, it's about which users you're willing to turn away. Knowing which model you're in can make these decisions a little less excruciating. It's not that you shouldn't experiment - on the contrary, this is one of the most fundamental hypotheses you want to test early on. However, it's not very useful to mix and match features from different models. Better to set up complete experiments that are themselves internally coherent. Take a cohort of new customers and expose them to a completely different approach, and measure their behavior over time. Use that data to make an informed choice about whether you want to change strategies.

Lean hiring tips

In preparing for the strategy series panel this week, I have been doing some thinking about costs. Fundamentally, lean startups do more with less, because they systematically find and eliminate waste that slows down value creation. Sounds a little abstract, though, doesn't it? I want to talk specifics, and when you come right down to it, most technology startups don't have a very interesting cost structure. Almost all of it is in one giant fixed-cost bucket: salaries. And all of that cost was caused by one activity: hiring.

Hiring is no different from any other company process. I've written previously about how to conduct a good interview, how to build a process to assess fit, and, most importantly, how to do root cause analysis to continuously improve. But hiring does have some special challenges, and today I want to talk about the one that drives almost all of the cost (and most of the mistakes): deciding when and how to hire new people.

In the startups I've been involved with, there are few mistakes I regret more than hiring too many people too soon. It's one of the hardest things to see while it's happening, and it's one of the hardest mistakes to undo, because of the human and emotional costs of layoffs. Much of what I have found effective is counter-intuitive, because it requires investing more effort in each hire - all in the name of efficiency. This can be especially challenging for HR professionals who measure their success based on the number of jobs they fill. This is another sub-optimization caused by incentivizing the wrong metric.

We should begin by clearing up one common misconception: that hiring more people means you can create more value faster. This is a variant of the time-quality-money fallacy, and it is not necessarily true. There are two complicating factors. One, which is described in great detail in the Mythical Man-Month (in finding that link, I discovered that today is coincidentally the exact 11th anniversary of my first purchasing that book: Jan 19, 1998), is that as you add people to a team or project, there is an increase in communications overhead that makes everyone slightly less productive. Second, a lot of work that you do is actually not value-creating - it's waste. Hiring people to do more of that kind of work can actually slow down your whole company. In fact, unless you are adding capacity to a bottleneck in your development process, you probably should not be hiring at all. For more detail on this second point, I recommend The Goal: A Process of Ongoing Improvement.

In any event, in order to hire effectively, you have to have a strong understanding if what the unit of progress is for your company. For additional thoughts on that topic, see
Lean startups vs lean companies.

When to hire
Let's start with deciding when to add a new job to the payroll. Here's my suggestion: don't hire for any job unless you have tried, and failed, to do it yourself. There are lots of bottlenecks in any company, but only a few of them actually are on the critical path of a startup's success. For example, let's say you don't have an operations team, and so you're still getting paged late at night when servers crash. Is it time to make your first operations hire? That depends. If you're coming in to work groggy and therefore are not spending time with customers, I'm willing to bet it is. But if it's just an inconvenience, then you should focus on figuring out how to mitigate the pain rather than hire to alleviate it.

The main drawback of this approach is that most people in most startups are already too busy. As a mentor of mine always says, "Startups don't starve, they drown." If you're already too busy, how can you afford to take on yet another job? Wouldn't it be faster to just hire a new person and delegate to them? Not in my experience. If you hire a smart, competent person, they will inevitably find ways to be productive and keep busy. But you still won't know if the work they are doing is on the critical path. They may be dong good work, but you must be able to evaluate its contribution to the company's overall success to know if you've made a hiring mistake or not. If you've done the job yourself, you'll know that it's important and you'll know what success looks like. Oftentimes, in my experience, you'll realize that you don't need to hire for it at all.

Insisting that someone inside the company take on every new job has other benefits, too. It's a way of making necessity the mother of invention, and taking advantage of your limited resources to help the company decide not to do things it doesn't need to do. This tool is often hidden in plain sight. I didn't truly appreciate its power until I saw how larger startups can lose it if they're not careful. Having too few people forces companies to focus on working smarter and eliminating unnecessary projects.

Hiring experts
In many situations, the goal of hiring is to bring in expertise that the team doesn't already have. This might seem like another time where delegating makes sense, but again I recommend trying to have an existing person take on the job. In fact, I recommend putting one of your best people on the new job, even though that's especially painful. In my experience, having a generalist try to emulate an expert's job will result in three possible outcomes:

1. The person will fail. This may seem wasteful, but it's actually very valuable. Having a star performer attempt it will give the expert you eventually hire built-in credibility. You'll also have someone on staff who understands the problem, and can help the newcomer get integrated into the team faster.
   
   
2. The person will succeed, and it will matter. In this case, you now have an expert on staff already. This gives you the option to still hire an true expert, or to back-fill the person who has taken on the new job. In fact, in many situations, you'll actually find that the job is not as big as you thought, and you don't need to hire anybody. Or the person you get to do it will know how to invest in infrastructure that makes one of their many jobs automatic - this is one of the advantages of hiring smart people and building a culture of continuous improvement.
   
   
3. The person will succeed, and it won't matter. This is surprisingly common, and is the greatest savings of all. In many situations, people invest experts with wished-for magic powers, thinking they will "solve all our problems" even when that's not realistic. By having an in-house person succeed, you can evaluate whether having an expert on staff is really shortening the critical path of your organization's success. If not, you can avoid making the hire with confidence.

Returning to our example of the beleaguered founder who still has the pager, before hiring an operations guy, try promoting someone from within to take on the job. If you find yourself now spending time with customers, learning and guiding the learning of your whole company, you were probably right that the pager was getting in the way. But, if you find yourself getting caught up in some other distraction, and not really adding value, then the pager was a red herring. You need to get your priorities straight and, until you do, you probably may as well carry the pager too.


Relentlessly pursue passive candidates
Once they've decided to make a hire, startups should specialize in hiring candidates who are not actively looking for a job. This requires dramatically more work than hiring active candidates, but it usually generates positive returns.

The best candidates are those you (or someone on your team) have worked with before. Hiring them has substantial benefits over strangers: lower risk (because you know their strengths and weaknesses ahead of time), faster team integration (because they know someone already and that person can mentor them), and lower communication overhead (because they are more likely to speak and undersatnd a common language).

Most companies I work with think they are already pursuing these already-known passive candidates. I disagree. I am willing to bet that, right now, you can think of at least one person who is not working at your company who is an absolute star at what they do. Even if you have a referral bonus program, even if you've talked to that person about your company, and even if you've sent a recruiter to call her, I bet you could do more. Ask yourself this question: have you really made that person an offer they can't refuse? No Godfather-type violence is required. Telling someone "I really believe you can change the course of my company's life, and I'm willing to do whatever it takes to get you on board" is surprisingly powerful. Every person who you could make that offer to but haven't is a lost opportunity. Multiply that across everyone in your organization, and every superstar they know, and that's a huge potential pool of talent.

Even passive candidates that you don't already know have advantages that make them worth pursuing. You learn much more from trying to convince them to join, because they are more likely to ask you tough questions. You want people to turn you down if your pitch is bad, so you have more opportunities to learn how to get better (and is it possible that the problem isn't just with your pitch?). Given that most recruiting channels are pay-for-performance, it's usually cheaper, too, because it has a lower conversion rate. If you want to expand your pool of available passive candidates, give a site like NotchUp a try. They maintain a huge database of passive candidates, by offering to pay them when they interview. Again, by requiring you to invest more in each candidate you see, you learn to be much more careful about talking to the right people, and more focused on closing those candidates. (They also have a pay-for-performance model that's much cheaper than your typical contingency recruiter.)



Hiring well requires a lot of introspection in order to do it well. You have to really understand your company and what it needs. You need to be able to communicate the value your company creates to others. And you need to be able to ask yourself the tough questions to continue to refine your hiring process as you scale. But in lean times like these, I don't see any real alternative.

Why PHP won

When I first learned to program on the web, Perl + CGI was the dominant platform. But by the time I was building my first websites for commercial use, PHP had taken over. Since then, PHP (as part of the LAMP stack) has really been the dominant development platform, at least in the free software and startup worlds. Through my platform choices, I have forced many people to learn PHP and to work with it on a regular basis. Some of them are probably still cursing my name, because - let's face it - PHP can be pretty painful. As a language, it's inelegant. Its object-orientation support is "much improved" - which is another way of saying it's been horrendous for a long time. Writing unit tests or mock objects in PHP is an exercise in constant frustration.

And yet I keep returning to PHP as a development platform, as have most of my fellow startup CTOs. This post is about why.

Let's start with some circular reasoning. The number one reason I keep coming back to PHP is that it has overwhelming community support. I've written elsewhere that success in creating a platform is "becoming a function not of the size and resources of the company that builds it, but of the size of the community that supports it." When we started IMVU in 2004, we could rely on a staggering amount of open source software that jumpstarted our initial product offering. From e-commerce to blogs to photo galleries, we benefited from tens of thousands of lines of code that had already been battle-tested. Add onto that the incredible amounts of infrastructure code that we reused, and we were able to bring our product to market months earlier.

But that just begs the question: why does PHP have the most thriving community? It's an example of an information cascade - as PHP started to pull ahead of the platform pack, more and more people started working on it, increasing the size of the community and therefore making it more likely that even more people would choose it.

So, to understand how that cascade got started (and keeps going), we have to look at several key attributes of PHP. You'll notice a pattern: most of the key features that enabled PHP's success are considered among its defects by experts in language design. Ironically, as PHP has grown up, its designers have been busy "fixing" these shortcomings. It remains to be seen whether they will be able to do so without losing what enabled them to succeed in the first place. Or, perhaps they'll create an opening for another web platform to start a new cascade. If that happens, I expect you'll see some of these attributes in that new challenger:

1. Speed of iteration, thanks to reload-pages-every-time. CGI scripts had notoriously bad performance. That's because for every page request the web server had to spawn a new interpreter process, load the script, and return the output across process boundaries. So when people started using Apache's module system to build better web application platforms, it was a natural that they'd try to make the system more efficient. Take mod_perl, for example. In that system, the first time you loaded a script to handle a page, the whole script (as well as associated data) were kept resident in memory. The next time you needed to handle that page, you could take advantage of caching for excellent performance. There was only one real disadvantage: when the script source code changed, you needed to restart apache so that it could reload. This sounds like a good trade-off, but it turned out to be a classic sub-optimization.
   
   Memory-resident code made it slightly slower to iterate on scripts. Every change required a server restart. Even on a single server, those extra few seconds between iterations is costly. Writing code with PHP, even though the language is crufty, feels fast, because of the tight write-test-debug loop. Plus, with memory-resident code, it's pretty easy to forget which version of the code is actually running at any given time. The stuff in memory is invisible - you can't double check it. As a result, you have to be a lot more careful.
   
   This feature also supported multi-user environments much better. You could give someone a directory on your server as effectively a simple sandbox. You didn't need to worry about them breaking your server configuration or restarting it while you were in the middle of something important. This made it easier for ISP's to offer PHP hosting on shared servers, too. All of which meant it has always been exceptionally easy to get started with PHP. And when considering a new language or platform, the getting-started barrier is probably the single most important factor.
   
   
2. Direct mapping of outputs to inputs: URL's to files, code and presentation intermixing. PHP has a number of other seeming defects that make it easy to understand what's happening in a PHP application. This allows new people to get integrated into an existing PHP project faster. For example, I have repeatedly taught PHP to complete novices in several companies, including people who had never before programmed at all.
   
   Almost every PHP-based website is burdened with ugly URLs that contain the name of the file that implements each page. This is considered bad practice, because URLs are supposed to be about resources (meaning the content of a given page), not implementation. In theory, you should be able to completely change platforms, languages, and servers and not have to change your URLs. However this approach, while elegant, has a major drawback: it requires non-trivial understanding to figure out where to find the code for a given page. Just consider one way this is commonly done: having a table of regular expressions somewhere that maps URLs to pages. This sounds simple, but in order to use it everyone on your team has to: 1) know where to find the list of mappings and 2) be able to read regular expressions correctly. Every other system I've seen has some drawback like this. PHP has no such compunction: the name and location of the file are hanging out in the open, for everyone (including your teammates) to see.
   
   And then let's consider what happens when you get to the implementation file tiself. PHP encourages one of the worst programming heresies of all: intermixing code and presentation logic in a giant mish-mash. If you're emitting HTML, as 99% of all PHP apps are, it's just so damn convenient to throw in PHP tags along with your layout tags, all inline on the page. In many ways, this is a maintainability nightmare, but it has a compensating benefit. In most cases, simple changes are simple to make. At IMVU, when we'd hire a new engineer, we could get them to ship code to production on their first day, even if they had never programmed in PHP before. That's simply impossible on most platforms. That benefit carries over through the whole life of a project. The vast majority of features are actually only a few lines of code. All the difficulty and effort is in finding the right place to put them. PHP's transparency makes that easier, encouraging more experimentation and fine-tuning.
   
   All of which leads to a non-intuitive conclusion. PHP shines on one of the most important criteria for platform selection: readability. Yes, you heard me right. Even though PHP's syntax is an inelegant beast, the overall platform is impressively readable.
   
   
3. Incoherent (but huge) standard library. These days, successful programming is as much about processing data as creating algorithms. To be a good data-processing language, you need a large standard library. PHP has always scored well on this count, with lots of support for database drivers, URL parsing, HTTP fetching, regular expressions - you name it. And this all came bundled up in another bad practice: a big monolithic interpreter. For most of its existence, PHP didn't have a standard package-distribution system or very good module support. As a result, important features that were used widely almost had to be bundled into the interpreter itself. Although this was annoying for sysadmins, security consultants, and language purists (and for those who had proprietary modules that couldn't be bundled), it was a huge boon for developers. It meant that the PHP brand stood for the same set of tools always and everywhere. If someone offers you a library or script, it's a huge benefit to know that it will run in your environment, without having to worry about dependencies, which leads to a lot more code sharing and code reuse.
   
   
4. Bad OOP support. PHP began, as many scripting languages do, with pretty primitive language features. It was especially criticized for its lack of object orientated programming support. Although recent versions have added robust objects, inheritance, and interfaces, for most of its life PHP's objects were little more than decorated dictionaries. This is another feature that makes language purists cringe, but was key to PHP's success.
   
   One of the overriding benefits of OOP is encapsulation: the ability to take a chunk of code and data and put them behind an interface. At the cost of a little extra indirection, you can organize your code around a series of long-lived objects, each of whom communicates with the other objects in simple, easy-to-understand ways.
   
   In practice, though, this extra indirection imposes a steep penalty in transparency. Most PHP scripts are not long-lived, meaning that every object has to be constructed, used, and disposed for every request. And most PHP servers are run in a stateless mode, delegating storage to other processes like memcached and MySQL. So, in many cases, there's not much readability benefit to constructing an object, when all it is is a lightweight wrapper for external data access methods. A "module" of related functions does the job just as well, and is a lot easier to find when you're trying to understand how an application works. This is yet another way that PHP supports non-programmers well.
   
   There's another big benefit: one of the major strengths of the web is that it auto-encapsulates code behind URLs. That's part of its magic, and PHP takes advantage of it. Everything is request-oriented, short-lived, and stateless. A PHP script is like an object itself, encapsulating a bit of functionality behind an interface defined by HTTP. It's one of the most effective paradigms for software devlopment in history. The platforms that win on the web are those that mirror its fundamental structure, not those that try to morph it into a more traditional "elegant" shape.
   

I'll close with one last thought. The inimitable Paul Graham has an excellent essay called The Python Paradox in which he argues:

...that you could get smarter programmers to work on a Python project than you could to work on a Java project.

I didn't mean by this that Java programmers are dumb. I meant that Python programmers are smart. It's a lot of work to learn a new programming language. And people don't learn Python because it will get them a job; they learn it because they genuinely like to program and aren't satisfied with the languages they already know.

Which makes them exactly the kind of programmers companies should want to hire. Hence what, for lack of a better name, I'll call the Python paradox: if a company chooses to write its software in a comparatively esoteric language, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it.

As always, Paul is right. So, given that PHP is so popular, you might think it wouldn't be a great choice for companies that are trying to hire great programmers. Given that startups depend on superstars to survive, why stick with PHP? In my role as a CTO, I've always tried to choose the right tool for the right job. The IMVU downloadable client, just to name one example, is written primarily in python. This let us hire extremely high-caliber programmers to work on it. You might then assume that our backend was written in mod_python. But it's not.

It's easy to forget that these decisions need to be made, not on a language-by-language basis, but on a platform-by-platform basis. PHP may not be a great language, but the platform it enables does attract one particular kind of great developer: the cutting edge web gurus who work primarily in javascript, DHTML, and Ajax. For them, PHP is an ideal language precisely because it gets out of their way, allowing them to build a simple foundation for their complex and beautiful browser-based cathedrals.

When simple things are simple, and hard things are possible, great people can take it from there. So here's to the team that built PHP. Thank you.

CPI > CPC

Google revolutionized advertising by popularizing pay-per-click. This model has not translated well to the world of social networking, because customers of social networks engage sites in a different way than customers of search engines. The difference is intent: when you're searching for a specific topic, ads have a chance to fulfill your interest in that topic. Social network page views are much more likely to be internally focused; ads are more of a distraction.

Because social networks have generated staggering quantities of traffic without generating as much economic value from traditional advertising models, there's a big opportunity to buy traffic on a new model. Starting with Facebook's developer program, social networks are all jumping on the apps bandwagon. Those apps are generating even more page views, making it quite affordable to buy application installs (which is a cost-per-action ad that requires the user to install the app before the action is considered complete). These cost-per-install ads are a reasonable way to jumpstart traffic to a new app, which is what they've been used for so far. But I think there is a much more interesting way to use the model, and that's what I want to talk about today.

If you ask an expert in Facebook apps to come help build yours, you'll usually hear the same advice: build your app for virality. That's sensible advice; most people who have made a business on Facebook rely on the viral driver of growth. Those who have tried to build apps that exist just as glorified landing pages or marketing channels have generally failed. So why buy advertising on a cost-per-install basis? If you have a virally-growing app, you don't need the extra installs. And if your app is designed to drive traffic to some off-network site or product, the cost is usually prohibitive, because the traffic in your app doesn't convert any better than a traditional CPM or CPC-based ad anywhere else on the network.

As a result, CPI advertising is incredibly cheap. I have seen plenty of situations where CPI rates are as low as $0.25 to $1.00, and plenty of situations where you'd pay that much just for a click to your traditional landing pages.

The unexplored value of CPI ads is this: social networking apps give you the opportunity to practice advanced marketing techniques like true one-to-one marketing or permission marketing. When someone clicks on one of your traditional CPC or CPM ads, what do you know about them? You know the content of the ad, so you know a little bit about what they're interested in. Other than that, you know their IP address, maybe their browser version or what country they are in. In other words, not very much. That's why so much of traditional CPC marketing is focused on optimizing landing pages and registration forms.

By contrast, look at what it means to have someone install your Facebook app. Again, you know they have some level of interest in your ad. You already have an authenticated login. You know the person's age, gender, where they went to school, who their friends are, what their interests are. And, most importantly, you have several reliable communciation channels to talk to them including including email, profile notifications, feed items and status messages.

Imagine the kind of impact you could have with a potential customer if you didn't have to waste any energy at all on convincing them to invite their friends. You can actually communicate with them instead, learning more about them, getting better at serving their needs. If you have an e-commerce product, for example, you could try using the app to offer specialized product offers or to tune your recommendations. If you follow Seth Godin's original permission marketing concept, you could offer your customers entries in a contest in exchange for learning more about your products. You could find out what products your customer already uses, and find out when they run out, need to refill, are ready to switch, or it's time to upgrade.

Think of all the money car companies spend advertising to people who are not at all thinking of buying a car, just to reach the handful who are. What if, instead, you could self-select as someone who is thinking of getting a new car, and have the car company work with you to research and select the best option, instead of spamming you with ads all the time?

The specific form will depend on the kind of product or service you want to sell. But the general principle is to frame your marketing on social networks as an exchange of value, played in rounds. As the customer gives you a little of what you want, you give them a little of what they want. If you have the patience to walk down that path, you can convert prospects into customers for life. From their point of view, buying your product gives them not just the product itself, but membership in an exclusive club where they can get the inside scoop on what you're thinking and doing. Wouldn't you rather relate to customers in that frame of mind than on TV or in a banner ad? And, as a customer, wouldn't you rather be treated that way?

Lessons Learned office hours

For me, there were plenty of surprises in the survey I conducted a few weeks ago. One of them was that multiple readers suggested holding "office hours" Q&A sessions in the Bay Area. I'm not quite ready to put out a shingle and take over my neighborhood Starbucks, but I have accepted two invitations to speak in a similar format in the coming weeks. So, for those of you in the neighborhood, feel free to bring your questions to either event. There are no canned presentations, so you can come set the agenda.

The first event is January 21, and is being put on by Dealmaker Media as part of their Strategy Series events: In the Trenches: Cutting Costs in a Tight Economy. The format is Q&A around a conference table with four "speakers" and a moderator. One of the other speakers is someone I know well: Seth Sternberg, founder of Meebo. He's a very smart guy who, as a Sequoia portfolio CEO, can tell you all about the "RIP good times" strategy.

January 21, 2009 | 6:30 - 8:30PM
Battery Ventures
2884 Sand Hill Rd. Suite 101
Menlo Park, CA 94025 (Directions)
Register

If you're interested, you need to register by 3pm on the 21st (Dealmaker events almost always sell out).

If you want your Lessons Learned completely unfiltered, but need food to help you stomach it, February office hours may be for you. This time, the event is courtesy of Bootstrappers Breakfast. We'll be at Hobbee's in Palo Alto on February 6. This event is for entrepreneurs only, and they ask that you register in advance.

Location
Hobee's 4224 El Camino Real, Palo Alto, CA 94306

Date and Time
Friday, Feb 06, 2009
07:30 AM - 09:00 AM

RSVP

For those of you in the Bay Area, I hope you'll come and bring tough questions. If you do come, be sure to say hello and identify yourself as a reader. For those of you not nearby, feel free to ask your questions in the comments. I'll try and do a virtual office hours sometime if you're interested.

Sharding for startups

The most important aspect of a scalable web architecture is data partitioning. Most components in a modern data center are completely stateless, meaning they just do batches of work that is handed to them, but don't store any data long-term. This is true of most web application servers, caches like memcached, and all of the network infrastructure that connects them. Data storage is becoming a specialized function, delegated most often to relational databases. This makes sense, because stateless servers are easiest to scale - you just keep adding more. Since they don't store anything, failures are easy to handle too - just take it out of rotation.

Stateful servers require more careful attention. If you are storing all of your data in a relational database, and the load on that database exceeds its capacity, there is no automatic solution that allows you to simply add more hardware and scale up. (One day, there will be, but that's for another post). In the meantime, most websites are building their own scalable clusters using sharding.

The idea behind sharding is simple: split the data between multiple machines, and have a way to make sure that you always access data from the right place. For example, consider this simple scheme. Imagine you want to store data about customers, each of whom has "last name" field in your database. You could create 26 identical databases, and assign each of them a letter of the alphabet. Then, whenever you want to look up data about John Smith, you first connect to the "S" database, and then fetch the data you want. All of the sudden, your single database solution just got twenty-six times more capacity added to it.

Devotees of sharding will already be groaning, because this is a bad idea. The problem should be familiar to anyone who plays Scrabble: not all letter are equally likely to be used as the first letter of someone's name. The shards for S and R will be heavily loaded, but the shards for Q, X, and Z will probably be idle.

A big part of sharding is picking a good partitioning scheme. Many have been proposed, and I won't discuss them in detail here. I really enjoy reading stories of how people have tackled this problem. Two of my favorites are from Flixster and LiveJournal. In the end, they fall into three broad categories:

1. Vertcal partitioning. In this scheme, all of the data related to a specific feature of a product are stored on the same machines. For example, Friendster was famously vertically partitioned at one time in its growth curve. A different cluster of machines served each of Profiles, Messages, Testimonials, etc.
   


2. Key-based partitioning. In this scheme, you use part of the data itself to do the partitioning. The "letters of the alphabet" scheme I presented earlier is one (bad) example of this type. More common is to use a one-way hashing algorithm to map the data to be accessed to one of the shards that store it. Certain data types present natural hashes as well: for numeric keys, you can take the key mod N, where N is the number of shards; dates are easily partitioned by time interval, etc.
   


3. Directory-based partitioning. This scheme maintains a lookup table somewhere in the cluster, that simply keeps track of which data is stored on which shard. This has two major drawbacks: the central directory can become a single point of failure, and there is a performance cost for having to consult the directory ever time you want to access data anywhere in the cluster.
   

So far, this is just a summary of what all of us who have attempted to build web-scale architectures considers obvious. For a good in-depth look at each partitioning scheme, I recommend Scalability Strategies Primer: Database Sharding.

Sharding for startups

To support a single partitioning scheme is easy, especially if you design for it from the start. But startups rarely have either luxury. It's almost impossible to know ahead of time what kind of data partition makes the most sense for any particular part of your application. It also doesn't make sense to design your app for scalability from the start, since you may have to create many iterations before you find a version that requires any scalability at all. Even worse, it's a rare app that requires only one kind of data partitioning.

The solution is to build an architecture that works for the startup condition. To me, the optimal architecture is one that serves four goals simultaneously. I'll talk about those goals, and then propose a system that I believe serves them well.

1. The ability to handle increased capacity incrementally, by adding proportional amounts of new hardware. This is the main reason most people undertake sharding in the first place. Note that most key-based partitioning systems do badly on this criteria. Most key-hashing schemes have some breaking point where, if you add just one more shard, you have to re-hash all of the data (imagine having to add a 27th database server to the alphabet-hashing scheme discussed above).
   
   
2. Incremental and proportional software complexity. As you increase the amount of data partitioning, application code gets more complex. Just to cite one common example, when all of your data resides on a single shard, you can run complex joins against it. If that same data is partitioned, you need to process the data in other ways. Just as we want to be able to make proportional investments in hardware to handle increased load, a good architecture allows us to make proportionally-sized changes to the code. We want to avoid anything that requires a massive rewrite just to change how data is partitioned.
   
   
3. Support multiple sharding schemes. Log-based data is often optimally stored in date-based partitions. Customer data is normally partitioned according to customer id. What about log data about customers? A good architecture allows us to use the right tool for the job, or even combine tools as necessary.
   
   
4. Easy to understand. Some sharding schemes are themselves hard to follow, or require you to write code that is hard to read. Especially as teams grow, a good scheme can be undone by programmers who use it wrong. It's unrealistic to expect everyone on the team to understand scalability in detail. Instead, a good architecture has clear rules that allow them to do the right thing without having to think too hard about it.
   

With those criteria in mind, let me propose a solution that has served me well in several startups. First, I'll describe how it works, and then I'll lay out how to build it incrementally.

A URL-based sharding system

Start by thinking of every kind of data in your application as associated with some entity. Give each entity a name. Then, whenever you want to have your system find the shard for a given entity, compose this name into a URL. For example, let's say you have data about customers. A given customer would have an entity URL that looked like this:

customer://1234

Somewhere in your API, you have a way to access data about a given customer. I'm not talking about high-level abstractions (like an object-oriented view of a customer's data). I'm talking about the actual data-fetching operation. At some level in your system, there's probably code that looks something like this:

$connection = new_db_connection("host", port, "dbname");
$statement = $connection->prepare($sql_statement, $params);
$result = $statement->execute();

What I propose is that you make your sharding decisions directly at the "new_db_connection" seam in the code. The new code would look like this:

$connection = new_db_connection("customer://1234");
$statement = $connection->prepare($sql_statement, $params);
$result = $statement->execute();

All of the complexity of determining which shard you need to use (which we'll get to in a moment) is hidden behind that simple API. Once a shard-lookup decision is made, there's no additional overhead; your application is talking directly to the right database.

In some cases, you can make it even simpler. If you're of the type of organization that doesn't use db access objects directly, but simply passes SQL statements through a central API, you can try this: consider the entity identifier as part of the statement itself. I've done this before using a structured comment added to each SQL statement, like this:

/*entity customer://1234 */ SELECT name FROM customer WHERE id = 1234

At the expense of some additional parsing overhead in your data-access layer, you get additional metadata associated with every query in your application. You may start out using this data only for sharding, but you may find it has other benefits as well. For example, you might notice that caching gets a lot easier if you have good metadat about which queries are associated with the same entity. It's a very readable approach, and it makes it easy to change the sharding for a query without having to change a lot of intervening function signatures (since the shard info is carried by the statement that's already passed through).

Whichever API you use, URL-based sharding is very easy to understand. I've taught everyone from hard-core programmers to scripters to HTML-designers to use it properly. And implementing it is easy, too. Every language already has a facility for parsing URLs. I normally also use URLs to specify the shards themselves, too. So, at its heart, the sharding system is really a way of mapping entity URLs to shard URLs, which look like this:

customer://1234 -> mysql://hostname:port/dbname

The scheme part of the shard URL can allow you to use multiple shard-storage types. For example, in one system I built using this technique, the scheme specified the name of the ADODB driver to use for accessing that database.

No system is entirely without drawbacks, however, and URL sharding is not an exception. In this case, the big drawback is that it works best if you store the mappings in a central directory. This gives you the big payoff of being able to support all kinds of sharding with just one system. Each kind of shard just maps to an appropriate kind of URL:

customer://1234 (id)
forums://master (vertical)
log://January/2008 (date)

You can even do more complex schemes, like having groups of IDs all map to the same URL in a bucket system, although I recommend keeping it simple whenever possible.

If you have a solid caching system in your cluster, like memcached, the overhead of a system like this is really quite small. The mappings never change (unless you build a data migration tool, in which case they change rarely) and so are easy to cache. Using memcached's multiget, which allows the fetching of many keys in parallel, I have written code to aggregate all the shard lookups for a given page and prefetch them, reducing the overhead even further.

The actual directory itself is straightforward. You need to store two kinds of information. One is a list of all the available shard types, and the machines available to store those types. This information is normally maintained by your operations team. Whenever they want to brng new capacity online, they simply add a row to the appropriate table. The second is the mapping itself. The sharding API follows a simple rule: whenever asked to find the shard for a URL, it either pulls it from the directory or, if it's not there, assigns it to one of the available shards for that entity type. (There's some added complexity as shards get "weighted" for more or less capacity, and for cases where data needs to be moved between shards, but these are not too serious to deal with). I normally recommend you just store this directory on your master database, but you could use a standalone vertical shard (or even a key-based partition!) to store it.

Incremental build-out

The most important attribute of URL-based sharding, though, is that it easily supports an incremental build-out that is perfect for startups. We can start, as most startups do, with a single, central master DB. We can take advantage of the many open source libraries out there that don't scale especially well, and write as much non-scalable code as we want. When we finally have a product that starts to get traction, we can practice Just-in-Time Scalability, as follows:

1. When load on the master DB gets above some agreed-upon threshold, we measure which tables and queries are causing the most read/write traffic.
2. We pick one such query, and decide to shard it. We always try to find queries that are causng load but are also relatively self-contained. For example, most applications usually have a number of highly-trafficked tables that are rarely joined with any others; these are excellent candidates for early sharding.
   
3. Once we've picked a query, we have to identify all of the tables it touches, and all of the queries that access those tables. The better your internal API's and data architecture, the easier this step will be. But even for total spaghetti code, this doesn't take more than a few hours.
4. We change the code to have all of the queries identified in step 3 tagged with the apporpriate entity URL that they affect. If we've uncovered an entity type that's not previously been sharded, we'll have to add some new shards and shard-types to the central directory.
5. Since there is already some data about these entities on the master DB, we need to migrate it to the new, sharded, locations we picked in step 4. I won't go into detail about how to do this data migration in this post; coordinating it with the code changes in step 4 is the hard part. You can read a little about it in my post on Just-in-Time Scalability.
   

Each time we need to add capacity to our application, we go back to step 1. At IMVU, during our steepest growth curves, we'd have to do this exercise every week. But we could do the whole thing in a week, and be ready to do it again the next week. Each time, our application got a little more complex, and a little more scalable. But at not time did we have to take the site down for a rewrite.

If your experience is anything like mine, you may be surprised at how much cruft remains on your master database. One advantage of this system is that it doesn't require you to add scalability to any piece of data that is seldom accessed. All that random junk sitting in your master databsase's scheme may make you feel bad, but it's actually not causing any harm. Only the parts of your app that need to scale will get the sharding treatment.

What do you think?

I've used this setup successfully, and have recommended it to many of the startups I advise. I much prefer it to the more complex solutions I've seen and worked with over the years, even though I recognize it has some drawbacks compared to those other systems. I'd really like to hear your feedback. Go ahead, poke holes in the design. Let me know the use cases that I've overlooked. And most of all, share with all of us what's worked for you.

Lessons Learned on Mashable today

I had the good fortune to be asked to write a guest post over at Mashable, which is running today, called HOW TO: Raise Money in a Down Economy. The major thesis will be no surprise to those of you who've been following this blog for a few months. Here's an excerpt:

Our experience was different from the traditional VC model, in which funding is reserved for true domain experts, those with deep background in their field solving a problem they personally have experienced. For the rest of us, there is an alternative: to create credibility by building a lean startup ...

At IMVU, we were terrified by how early we shipped (and charged for) our first product. Frankly, we were embarrassed by how many bugs, crashes, and defects it had. We were even more embarrassed by the pathetically small number of customers we had, and the pathetically low amount of revenue we had earned so far. It wasn’t uncommon for advisors and even potential investors to ask, when we would show our results, “are there some 000’s missing from this graph?” We’d always cringe as we admitted that, no, we really only had a few thousand customers and a few thousand dollars in monthly revenue.

Despite that, we were able to establish credibility, because we used those early numbers to demonstrate that our customers were getting real value from our product and, more importantly, that we understood how to build more of it.

Rather than just focus on the right kind of pitch to use, I wanted to write about the kinds of startups that create lasting value, and therefore are great companies to fund even in down times. The core of the article is my first attempt to articulate the key metrics (in graph form) that I believe demonstrate customer value. When startups ask me what to measure, I always come back to these three as a starting point:

1. Revenue per customer.
2. Retention cohort analysis.
3. Funnel averages over time.

You can read the article to find out why.

Happy new year

When I started writing this blog a few months ago, I wasn't sure if anyone would find it interesting. I certainly never expected the overwhelmingly positive feedback you've given me. Thanks to everyone who's taken the time to read, subscribe, and comment. I've been really moved by your support and suggestions. Here's hoping your 2009 is inspiring and a time of growth.

What challenges are you grappling with in the new year? What do you expect for 2009? What topics would you like to see here on Lessons Learned?

Thanks, as always, for stopping by.