Over the many years, I’ve worked on a bunch of products that follow this pattern:
• PM proposes a new feature, it’s one that tons and tons of customers request and is seen as obvious and fundamental. “I want to back up my data” is a very typical example of this, as is “I want to automate this action”.
• The PM, after talking to a bunch of people and asking them things like whether they want the feature, whether they’d pay for it, how much, etc.. makes a strong pitch and everyone gets on board.
• I’m pulled in as the data person and we work out a success metrics plan so that when the feature launches we can see what sort of impact it has.
• During the metrics planning I usually ask what they expect success to look like. PM often confidently says they expect > 90% adoption. So many people say backups are best practice, that automation is required for scaling up, etc,. it is blindingly obvious everyone will use it.
• The feature launches. Metrics say 10% of all users adopt the feature.
• Oh, [some excuses here]. It’ll surely pick up in a bit. Let’s wait for more data.
• Things do not pick up.
• Cue a few weeks of me proving that our metrics telemetry is not broken.
• Things still do not really pick up much, just hangs in the 10-20% range. Some product changes and experiments are proposed and worked on to try to boost the numbers. They help to varying degrees.
• PM stops trying to hype it up, moves on to work on another feature. (View Highlight)
The pattern of behavior we see here is the PM essentially letting a data point, “everyone I ever talk to asks for this exact same feature” make their decisions for them without ever considering in what contexts that people will or will not adopt the feature they’re considering. (View Highlight)
The only way to avoid having such over-inflated expectations would be to have a very deep understanding of what users are actually trying to accomplish, and how they’re currently accomplishing it. This is the kind of knowledge that deep user research attempts to get to. The problem is that talking to a bunch of customers over drinks at an industry conference and asking very leading questions doesn’t equal actual user research. (View Highlight)
If you’re a data person like me, the chances are pretty good that you feel a certain amount of anxiety at even the mere thought of having to interview customers to understand them. That’s what qualitative researchers are for, right? But putting aside who does the customer interviewing part of the research, there are still many things that use data folk can do to try to steer teams away from such a giant pitfall. (View Highlight)
First, we might have access to historical data of these scenarios that we can use to remind people to temper their expectations. Maybe you can find users who are engaging in a behavior that is related to what will be built, and you can show it’s a relatively uncommon occurrence. Nowadays, if I’m working on a product feature that looks like its headed towards a low adoption rate, then I’m pretty vocal about warning teams up front to expect low numbers. To the extent that teams are willing to listen, I can steer them into better understanding their customers and figuring out if the feature they’re planning on building is actually something users will likely adopt. (View Highlight)
Then, when the time comes for me to help a team set up their success metrics, I’m always pushing to make sure we have telemetry “high up in the funnel”. That is to say, I make sure we have stats on how many people are exposed to and likely know the feature existed, and what fraction actually show some interest in the feature. I know that I’m going to have to prove that lots of users simply aren’t showing interest about the new feature, so I might as well make sure I can show that up front. (View Highlight)
Finally, when teams are in the flailing around in denial phase of the launch, what we can do as the data person helping them make sense of their metrics is to help them face the reality that people aren’t going to adopt the thing en-masse. There’s usually not going to be some little tweak or flow redesign that is going to make a feature explode in popularity. (View Highlight)
(View Highlight)