No departments

Big companies have departments. Startups are companies. Startups aspire to become big companies. Therefore, startups should have departments. Right?

Why do companies have departments? There are a lot of reasons: ladder of advancement, sharing of best practices, functional specialization. Each of these benefits also exists in startups, which is why most startups are also organized in departments. But I have come to believe that because of the unique context of startup land, the payoff is a lot smaller than it is for larger companies. Meanwhile the drawbacks of functional departments can cause real and lasting harm.

I once worked at a startup with an exceptional functional department system. The leaders of each department were world-class experts in their respective fields. The team hired only the best and the brightest. Looking back after a few years, it’s evident that many of the people who worked in these departments have gone on to do incredible things in industry. They are leaders, visionaries, founders and managers having tremendous success.

Yet talent organized improperly can lead to failure. I was an engineer on the engineering team. We had to work closely with artists on an art team. We sat in different parts of the building, ate lunch separately, spoke a different specialized jargon, and generally didn’t understand each other. According to the Waterfall methodology in which we worked, this shouldn’t have been a problem. After all, we rarely had to work on the same project at the same time. The art team would often be involved in the specification phase of a new feature, since they were responsible for the look-and-feel of the product. Then we’d build the feature, which would often include tools that were intended for the art team to use to build the parts of the product that they were responsible for (in video game parlance, this is the “art path” that allows artists to get their work into the production product).

Sure, some communication was necessary, especially as artists had to be trained on new tools periodically. But according to the theory, this should have been covered by the various specs and documentation we were rigorous about producing.

If anyone has ever worked in an environment like this, you’ll probably be able to imagine the things that can go wrong. For one, the engineers consider the artists stupid; the artists consider the engineers arrogant. Not a lot of trust builds up that can be used when real disagreements emerge. Instead, there’s a positive feedback loop of bad feelings. And like feedback on a simple microphone sound system, this would occasionally boil over into screeching.

I remember one such meeting vividly. I was the junior guy on a project team; I was called in to do some technical due diligence for reasons that were obscure to me, because the team already had much more senior engineers assigned to it. I was invited to a feature decision meeting, where the team was closing in on a detailed spec. The meeting was tense. The artists on the team had called in the big guns, and VP-level folks were there to explain the importance of certain aspects of the visual design that threatened to be cut. I eventually realized that I was there as part of the same plan – the art team has specifically requested someone technical but unimportant to be able to render opinions that might undermine their more senior opposition. Not to be outdone, the technologists on the team had also brought their big guns, and the meeting was packed with employees of every level – from VP’s all the way down to me.

As the meeting progressed, the temperature kept rising. At first, I couldn’t even follow the recriminations back-and-forth. Eventually, though, I realized what was at issue: the art team was insisting that the UI for this feature have rounded corners. Incredibly, they were willing to bring the company to a standstill to protest that this was an absolutely essential feature. Even more surprisingly, the engineering team was equally vocal about their contention that adding rounded corners would add weeks of development time to the project, which would have pushed it out way past its hard deadline, effectively killing it. On the surface, this was a ludicrous dispute – both sides were willing to kill the project rather than proceed with (or without) a minor UI tweak. Were they just crazy?

I don’t think so. This meeting was just the latest in a series of escalating skirmishes that had taken place over many months. The feedback loop looked something like this. The art team would create a spec for a feature, detailing the UI as best they could. The engineering team would then build that feature, mimicking the UI as close as they could using the current primitives supported by the system. When the art team would review the final product, they were inevitably outraged – it deviated from the spec in ways they considered major. So there would be a lot of scope negotiation at the very end, when it is most expensive. Sometimes, the art team would win the argument, and the engineers would pull a few all-nighters to make them happy, but feeling betrayed at the new additions to the spec. Sometimes the engineering team would win, and the art team would have to accept (and be held responsible) for a feature that didn’t really work the way they wanted, feeling betrayed at the violation of their agreement.

As an isolated incident, this wouldn’t be a big deal. But scope negotiations between departments are an example of an “iterated prisoner’s dilemma” situation, where the same parties repeatedly negotiate, and rely on their previous experience to inform their choices in the current round. Unfortunately, the equilibrium in this particular setup has one overriding outcome: longer and more detailed specs. Here’s why.

The art team feels burned that they didn’t get what they asked for. Last time, the engineers weaseled out of their commitments by point out areas where the spec didn’t specify what was really important to the artists. So this time, they are going to spell out what’s important in even greater detail, to leave less wiggle room.

The engineering team feels burned too, and feels that they were blamed for deficiencies in the spec as if it was their fault that the technology doesn’t really support what the artists want to do. So they react in two ways. First, they actively encourage a more detailed spec, and are more aggressive about pointing out possible inconsistencies. This forces the art team to make concrete decisions about stuff they don’t really care about. Second, the engineering team starts to pad their estimates, knowing that each feature in the spec is not really done when they think it’s done – there’s going to be inevitable scope creep from the art team when they finally see the final result.

What are the consequences of this more detailed spec? For one, it takes a lot longer to create, meaning that the projects themselves get larger in order to rationalize the increased investment in planning. Second, the extra detail obscures the artists’ original intent in specifying the feature, so the engineers are even more likely to miss the big picture and build the wrong things. And lastly, it removes the engineering team’s ability to find breakthrough solutions that might deliver most of the value at a fraction of the cost. They can’t use any discretion for fear of breaking the spec’s contract, even if the changes would probably go unnoticed or even be in the company’s best interests. The lack of trust (and the procedure of the Waterfall methodology) makes it very difficult to ask for clarification or changes in the spec while the implementation is underway.

This feedback is a nasty trap, and it’s just how this room full of otherwise rational adults wound up in a screaming match about rounded corners. It was painful to watch. Now, part of the reason I remember this particular meeting so well is that I wound up doing something considered really radical at the time. I suggested that we change the underlying architecture of our UI system so that the artists would be able to build their own UI pieces themselves and then integrate them into the product without requiring new code every time. It took an incredible amount of politicking and arm-twisted, but I did eventually get the teams to agree to that solution. I’m proud of that contribution, but the reason I tell that part of the story is not to show off, but rather to be able to tell you what happened next. Although both teams got something valuable about the new system, neither was very happy. I had successfully defused the situation, and by reducing the feedback loop between the artists spec and its implementation, I was able to help them realize their goals better. As a technical fix, it was brilliant. As a solution to the underlying problem, it was useless.

Neither side liked me very much for having "fixed" their problem. In particular, the artists felt like I had created a lot more work for them – they were used to having other people grapple with implementation details for them, and now they had to do it themselves. They either had to hire a developer onto the art team itself (unthinkable) or learn those development skills themselves (which was, to be fair, really hard). The engineering team wasn’t happy either. Creating this new architecture was a fair bit of work, and they couldn’t shake the feeling that I had basically sided with the enemy, giving them tools that would require a lot of engineering support but basically deprive the engineering team of any credit for the resulting features.

I’ve now come to believe that this confrontation was a direct result of the partition between the departments in this company, and that rather than seek technical solutions to the disagreement, I would have been better off working to break down those walls.

Let me tell one more story from that same company. I mentioned before the art path, the set of tools the engineering team was required to build and maintain for the art team to be able to create content. Because the art team was considered an internal customer (and “friendly” to boot), we didn’t waste a lot of time making the tools easy to use. Instead, we spent time making sure the exact behavior of the tools were well documented – by engineers, naturally.

This led to some pretty bizarre situations. One time I remember in particular, an engineer fixed a bug that was causing artists’ creations to render in our 3D environment with a skewed rotation. The details aren’t important, but it turned out that under certain relatively common conditions, the objects would come out completely upside-down.

I’m sure this engineer was expecting to be treated as a hero by the art team, since he had just fixed a major bug. But instead he was reviled. Why? The artists had known about this bug for ages, but had just assumed it was a natural part of the system. They had learned that the best way to solve problems is through trial-and-error, doing whatever it takes to get the object to look right. So sometimes the object would come out upside-down, sometimes not. When it did, they’d just invert their original work so that after the upside-down transformation, it looked right. Sure, there was a lot of randomness in whether they would need to take this extra step, but this was nothing unusual. From their point of view, the tools were full of meaningless jargon and bizarre incantations that resulted in nearly random behavior. This bug was not even considered a major one, since at least it had a deterministic fix.

But now you can see what happened when the bug was fixed. A large, but mostly random, sample of objects started rendering upside-down! This required either that we leave the bug in place, or that the art team go back and rework all those thousands of upside-down objects. Neither solution is particularly alluring.

Again, this led to lots of mutual recriminations. Why didn’t the art team come to the engineers and tell them of this bug as soon as it manifested? Why aren’t they smart enough to figure out what the tools actually do? On the other side, why don’t the engineers just deliver tools that work like they’re supposed to? Other more established companies have tools that are intuitive to use, why can’t we?

The solution to this problem is actually really simple. Just create an art path team, composed of some artists and engineers. Force them to live and work in the same physical space, force the engineers to actually do some art production, and force the artists to actually learn what the technical limits of the tools are. As the team gets traction, simply rotate members from both departments through this team, so that the knowledge they gain is eventually diffused through both organizations. And hold the leaders of that team – artists and engineers alike – accountable for a set of clear goals for the tools that are important to the company.

There’s nothing intrinsically difficult about this problem, just as their was nothing intrinsically difficult with the “rounded corners” problem I discussed earlier. The barrier to doing the right thing is the entrenched ideas originating from departments. Who would lead this team? Who would they report to? How would we ensure that the team was faithful to the best practices of both the art team and the engineering team? And plus, do we really want to be cross-training engineers in art and artists in engineering? Isn’t that a waste of time? After all, both teams are already too busy, how can we afford to pull people off and waste time?

This is another variation of the time/quality/money fallacy – the very quality problems that a team like this would address are currently wasting time and causing the team to be “too busy” to invest in the solution. So: enough with functional departments at startups. Let's start holding people accountable solely for their contribution to the only thing that matters: validated learning about customers.