Define Success Metrics for a Social Feed Product

Why This Is Asked

Defining metrics is a core skill for data analysts working on product teams. This question tests whether you can translate a business goal into a rigorous measurement framework — and whether you anticipate the ways metrics can be misleading.

Key Concepts to Cover

• North Star metric — the single metric that best captures long-term product health
• Leading indicators — metrics that move before the North Star, useful for short-term experiments
• Counter-metrics — metrics that should NOT get worse (even if the primary metric improves)
• Guardrail metrics — hard limits on user experience (e.g., p99 load time must stay under 3s)
• Goodhart's Law — when a measure becomes a target, it ceases to be a good measure

Framework: Metric Hierarchy

North Star: Long-term user value metric (e.g., weekly active users who view 5+ posts)
├── Primary: Engagement depth (posts viewed per session, comments per post)
├── Leading indicators: Scroll depth, return visit rate
├── Counter-metrics: Time spent (can increase even as satisfaction drops)
└── Guardrails: Ad revenue per user (ranking shouldn't tank monetization)

How to Approach This

1. Clarify the Business Goal

• Is the ranking change optimizing for engagement, satisfaction, creator growth, or revenue?
• What problem is it solving? (Users not seeing relevant content? Too much low-quality content?)
• What does "success" mean to the PM and leadership?

2. Define the North Star

For a feed: "percentage of users who complete a session with at least 3 meaningful interactions per week." Meaningful = comment, share, or 30-second+ video view — not a passive scroll-past.

Avoid raw time-on-site: it conflates doomscrolling (low value) with deep engagement (high value).

3. Add Counter-Metrics

A ranking change that increases engagement but surfaces more inflammatory content is a failure even if the primary metric goes up. Add:

• Negative feedback rate (hide post, unfollow, report)
• Satisfaction survey scores (sampled weekly)
• Creator posting frequency (if ranking deprioritizes creators, they churn)

Common Follow-ups

1. "DAU went up after the ranking change. Is that a success?" DAU is a lagging, noisy metric — it's influenced by seasonality, marketing, and many things outside the feed. Look at engagement depth and satisfaction within the cohort exposed to the change.

2. "How do you measure satisfaction quantitatively?" In-product surveys (1-5 rating, sampled post-session), implicit signals (return visit rate, share rate), and third-party sentiment analysis.

3. "The PM wants to ship after 2 weeks because DAU is up. What do you say?" Two weeks is often insufficient to see churn effects — users who had a bad experience may not show up in DAU for 3-4 weeks. Request a holdback group and extend to 4 weeks minimum.