Measure way fewer things

Good metrics galvanize teams around the most important problems. But it’s so easy to measure the wrong things, measure too many things, or make metrics complex or confusing. Then you’ve turned this power tool in the wrong direction, where it can inflict as much harm as good. As a developer, you might not have direct control over what your company wants to measure. That's fine. But you can be conscious of what you and your team decide to pay the most attention to and look at day to day.

As always, I suggest radical simplicity. Don't over-engineer your code, your processes, or in this case, your metrics. Tell a room of developers, "We need to make this one number go up," and be amazed at what they can do. Using metrics like this gives the team focus, and also connects your work to the company goals.

We wanted to “measure what matters.” We got into long debates about the merits and meanings of various metrics. We proposed a lot of ideas and said yes to most of them. We built complex dashboards showing dozens of metrics. Was each metric accurate and meaningful? Who knows. Did anyone look at these with any regularity? I sure didn’t. Was I responsible to make any of these metrics move? Nope.

We had long meetings about how to measure a particularly elusive metric. We believed this metric told us whether our product solved problems at a deep, nuanced level. We called it an “impact metric” (a term of such immense vagueness that it means nothing at all). At long last we decided on a sophisticated measurement borrowed from academia. We built a complex query, and put a number on a very official-looking dashboard. The number had decimals, so it looked precise. The metric beat industry standards by a wide margin! Just one problem — the data we fed into this formula was incomplete and unreliable. We had precision without accuracy, so the number meant nothing. It appeared to be scientific, but it wasn’t.

The greater complexity and nuance we tried to measure, the easier it was to misinterpret data. The complexity of this metric also masked the fact that it represented shoddy data. A casual consumer of this information would have no idea it was bullshit. Bullshit that appears to be scientific can lure smart people into delusions. We stared at this supposedly important number and wondered how to make it move. No one had any convincing ideas.

Someone asked me a basic question: how many people use my product? To my great embarrassment, I did not have a good answer off the top of my head.

To summarize our problems:

• We measured too much, so we lost focus on the most important stuff.
• We measured confusing, second-order effects of our product. These effects were interesting, but difficult to affect. So they weren’t particularly helpful, and were a bit demotivating. On top of that, they gave us false confidence that we knew things we didn’t know.
• Our metrics were not linked to company / team goals. It was not clear which mattered most. No one was responsible to make any of our metrics move.

The company got serious about setting good goals. The executive team put forward 4 simple (but challenging) goals. We measured a small number of metrics that showed progress toward those goals. I was responsible for 3 things:

• Work with the customer success team to get 15 specific clients way more active on our product. The metric: number of times these clients completed basic tasks in our product.
• Deliver a specific feature, satisfying 3 pending contracts, resulting in new revenue. The metric: new revenue from these 3 contracts.
• Increase usage of our most useful feature. The metric: number of times all users used this feature.

We had tremendous focus. We discussed these metrics weekly with executives. My development team talked about them constantly. Everyone contributed ideas of how to make them move in the right direction. We blew one of our metrics out of the water. We got to brag about our accomplishments. One of our other metrics lagged behind. No big deal. Everyone knew, it surprised nobody, and we had a plan to fix it.

The keys to using metrics as a powerful, productive force are:

• Align metrics with company goals.
• Keep it simple, and measure a small number of things.
• Look at these metrics often.
• Measure things your team can directly affect.

Here are suggestions of things you should measure. If this information is not available to everyone on your team, go fix that first. If you’re measuring lots of other stuff, consider whether that stuff matters.

Are new people signing up?

• Number of new accounts
• Revenue growth

Are customers using the product?

• Logins
• Number of times users successfully do the thing they’re supposed to do on your app

Are customers happy?

• Net promoter score
• App reviews

Do customers stick around?

• Churn

Beyond this, make a few metrics related to specific team goals or company goals. Voila, you're done. You could get so much more complex than this, but you probably don’t need to. If everyone in the company knows of those 4 things, you’re in an incredible place. Keep it simple. Focus on company goals. Understand your basic vital signs. Beyond that? Shut out the noise. Your team will be better and smarter for it.