Analyze the Impact of a New Notification Strategy

Why This Is Asked

This is a classic "metrics can mislead" scenario. Short-term DAU increases from notifications often mask long-term retention damage. This question tests whether you think beyond the headline number.

Key Concepts to Cover

• Short-term vs long-term effects — notifications can inflate DAU while eroding satisfaction
• Opt-out rate — the leading indicator of notification fatigue
• Notification-driven vs organic DAU — are we creating real engagement or session inflation?
• Cohort analysis — do users who got more notifications retain better at 30/60/90 days?
• Counter-metrics — uninstall rate, push permission revocation

How to Approach This

1. Reframe the Question

"DAU went up 8%" is not the same as "this was a good decision." Ask:

• Did quality of sessions increase, decrease, or stay the same?
• What happened to notification opt-out rate and uninstall rate?
• Were users who received more notifications more likely to churn at 60/90 days?

2. The Right Metrics

| Metric | Why it matters | |--------|---------------| | Notification-driven session rate | What % of sessions came directly from a notification click | | Organic DAU (excluding notification-driven sessions) | Did we grow real engagement or just inflate DAU | | Opt-out / permission revocation rate | Leading indicator of user annoyance | | 30/60/90 day retention by notification frequency cohort | Did heavy notification users churn faster? | | Uninstall rate | Extreme dissatisfaction signal |

3. Cohort Analysis Approach

-- Cohort users by notification frequency bucket
-- Compare 90-day retention across buckets
SELECT
  notification_frequency_bucket,  -- low/medium/high
  DATE_TRUNC('week', first_notification_date) AS cohort_week,
  COUNT(DISTINCT user_id) AS users,
  COUNT(DISTINCT CASE WHEN days_since_first >= 90 AND is_active_day_90 THEN user_id END) AS retained_90d,
  retained_90d::float / users AS retention_rate_90d
FROM user_notification_cohorts
GROUP BY 1, 2
ORDER BY 1, 2;

Common Follow-ups

1. "The opt-out rate went up 15% but DAU is still up. What do you recommend?" A 15% opt-out increase is a serious signal — those users will have permanently degraded experience. Recommend pulling back frequency for the segment with high opt-out rates and measuring whether they retain better.

2. "How would you design a proper experiment to test notification frequency?" Randomly assign users to frequency tiers (control = current, treatment = 2x). Run for 90 days minimum. Primary metric: 90-day retention. Guardrails: opt-out rate, uninstall rate, satisfaction score.