🔠Unmeasurable Metrics
Hello! Adam Thornhill here. ‘The Podcast Guy’ saving you 10 hours a week.
Enjoy the 117th Podup, with special thanks to ChatGPT Consulting.
Today, we’ll dive into the best insights and ideas from Unsolicited Feedback.
When is data good enough? Sure, there are some cases when you need 100% accuracy (think regulated industries like healthcare and aviation). But for most of us, the pursuit of perfect data can be a fool's errand.
In many cases, imperfect or proxy data is all you need. Particularly if it's for a two-way door decision. This episode breaks down Jason Cohen's post ‘Metrics That Cannot Be Measured’, helping us avoid the time-suck of unmeasurable metrics.
Metrics to avoid
The crux of Jason's post is that some of the most important metrics that people love to talk and think about are impossible to measure. As a result, the effort that's put into them is a complete waste of time. This includes:
The impact a single feature has on revenue
The impact a single activity has on churn
The probabilities of risk
The ROI of company culture and values
The value of brand
Brian Balfour
How to assess features
Product management books tell you that every feature needs to be measured back to its impact on acquisition, retention, and monetization. In practice, it's impossible to do a holdout test on a specific feature. They're not lego blocks that you stack on top of each other to increase your revenue.
Instead, come up with a suite of proxy metrics. At Reforge, we use a framework called TARS. It stands for:
T - Target audience. Not every feature is built for every user of your product. What percentage of your users is the feature built for?
A - Adoption. How many people from your target audience used this once? What does it mean to be activated on this feature?
R - Retention. Of the people who used it, how many came back? Most companies and most teams never measure this.
S -Â Â Satisfaction. For many features this isn't practical. For the rest, try to get some qualitative feedback i.e. Did this achieve what the user wanted?
Fareed Mosavat
Why it matters
Our obsession with data-driven decision-making can sometimes lead us down a rabbit hole of unmeasurable metrics.
While data is undeniably important, the quest for perfect metrics can be a distraction, especially when dealing with complex variables like company culture, brand value, or the impact of a single feature on revenue.
The key takeaway here is to focus on what can be realistically measured and understood. This doesn't mean disregarding complex variables but rather using proxy metrics that give us a good-enough understanding to make informed decisions.
Next steps
Audit your metrics. Take a hard look at the metrics you're currently tracking. Are they directly impacting your decision-making, or are they vanity metrics that distract you from what’s really going on?
Adopt the TARS Framework. Use this when assessing what good looks like for new features. It's not perfect, but it's actionable and beats misleading indicators like NPS.
Identify two-way doors. If decisions are reversible, you can get away with using proxy or imperfect data. Save the rigorous data analysis for one-way door decisions that are hard to undo.
Combine qual with quant. For metrics like company culture and brand value, consider qualitative assessments. Employee surveys or customer interviews can provide valuable insights that numbers cannot alone.
Timebox your analysis. Set a specific amount of time for data analysis to avoid the time-suck. Once the time is up, make the best decision you can with the information you have.
Your thoughts?
Thanks to ChatGPT Consulting for making this post possible…
Struggling with ChatGPT adoption? Then you’re like most businesses:
✖ Employees are resistant to change
✖ Just 19% have tried ChatGPT
✖ Only 4% use ChatGPT daily
Avoid seeing your company being left behind - lead the way instead.
Transform your business productivity in 4 weeks. The accelerator program is designed to make ChatGPT an integral part of your business DNA.
The best part? It's not a one-size-fits-all approach. Get bespoke 1-on-1 ChatGPT training tailored to every person within your business.
P.S. This program has proven results, with 100% of staff reporting over 5.7% efficiency gains, and some staff boosting productivity by 21.3% 🚀💼
86% of readers have decision-making authority or influence. Reach C-Suite Execs, Directors/VPs, Senior Professionals, and Business Owners/Founders with Podup.
Quotes were pulled at different points of the episode. Sentences were left out to make the narrative more concise. Podup is not associated or affiliated with any podcast.