Measured the wrong thing

Description

  • Why are you still just tracking MAU?

Disproof

  • Numbers that are connected all the way through user value to your continued existence

Consequences

  • Your solution ends up looking like it’s working when it’s really not
  • After a great amount of effort and money to acquire them, you start bleeding customers.
  • You probably throw a bunch of money at re-acquiring these “warm inactive leads” (generally they cost 3x to reacquire) instead of fixing the problem in the first place
  • Each attempt to move the wrong number gets more and more wrong. Eventually, the music stops and you never realised it was a funeral dirge - you were too busy trying to polka

Causes

  • Some smartass got the “you can’t improve what you don’t measure” part right, and entirely missed the part where measuring the wrong thing leads to improving the wrong thing
  • A new leadership hire will assert dominance by demanding quantified r-e-s-u-l-t-s and find themselves getting d-e-f-l-e-c-t-e-d by big numbers that look good
  • Some teams just want to feel good about a number they can make go up quickly instead of a number that matters, and in a culture that’s not critical about numbers everyone else buys the hype
  • And sometimes, nobody realises that metrics themselves abstract away important information - "the numbers made me do it" (see Sutherland’s talk on this)
  • Otherwise it can just be that you failed to create joined-up thinking across the organisation

Approaches

  • Ask people what the point of the product is and see if anyone says “solve it so good that each successive customer gets cheaper to acquire”
  • Try a strategy different from “throw more VC money at it”
  • Try to understand why you're measuring something