Metric anomaly detection explainer with root cause framework

# Anomaly Investigation: Conversion Rate Drop **Metric:** Website-to-trial conversion rate **Normal range:** 3.8% - 4.5% (past 90 days) **Current value:** 2.1% (42% below normal) **Time period:** Last 7 days (April 5-12, 2026) --- ## Step 1: Is This Real or a Data Issue? Before panicking, rule out data problems. Run these checks: ### Tracking Verification ```sql -- Check if tracking events are firing correctly SELECT DATE(created_at) AS day, COUNT(*) AS page_views, COUNT(DISTINCT session_id) AS sessions FROM events WHERE event_name = 'page_view' AND created_at >= '2026-04-01' GROUP BY day ORDER BY day; ``` **Check:** Are total page views consistent with prior weeks? If views dropped proportionally with conversions, the traffic source changed — it's not a conversion problem. ### Funnel Integrity ```sql -- Check each funnel step for data gaps SELECT COUNT(*) FILTER (WHERE step = 'landing') AS landing, COUNT(*) FILTER (WHERE step = 'pricing_click') AS pricing, COUNT(*) FILTER (WHERE step = 'signup_start') AS signup_start, COUNT(*) FILTER (WHERE step = 'signup_complete') AS signup_complete FROM funnel_events WHERE created_at >= '2026-04-05'; ``` **Check:** Is any funnel step showing zero or near-zero? That suggests a broken tracking pixel or a 500 error on a page, not a real conversion drop. ### Deploy Check ```bash # Check if anything was deployed around April 5 git log --since="2026-04-04" --until="2026-04-06" --oneline ``` **Result of our checks:** Tracking is intact. Page views are actually UP 12% (more traffic, fewer conversions). No deploys on April 4-5. **This is a real conversion drop.** --- ## Step 2: When Exactly Did It Change? Narrow the window from "last 7 days" to the exact hour. ```sql SELECT DATE_TRUNC('hour', created_at) AS hour, COUNT(*) FILTER (WHERE event = 'landing') AS visitors, COUNT(*) FILTER (WHERE event = 'signup_complete') AS conversions, ROUND( COUNT(*) FILTER (WHERE event = 'signup_complete') * 100.0 / NULLIF(COUNT(*) FILTER (WHERE event = 'landing'), 0), 2 ) AS conversion_rate FROM funnel_events WHERE created_at >= '2026-04-04' GROUP BY hour ORDER BY hour; ``` **Finding:** Conversion rate was 4.2% on April 4. It dropped to 2.8% at 2pm UTC on April 5 and has been declining since, reaching 1.9% on April 12. **The decline is gradual, not sudden.** This rules out a single broken page or deploy. It suggests a change in traffic quality or a competitor action. --- ## Step 3: What Segment Is Driving It? Break the metric down by every available dimension. ### By Traffic Source | Source | Before (3/29-4/4) | After (4/5-4/12) | Change | |--------|-------------------|-------------------|--------| | Organic search | 4.8% | 4.6% | -0.2pp ✅ stable | | Direct | 5.1% | 4.9% | -0.2pp ✅ stable | | Google Ads | 3.2% | 1.1% | **-2.1pp** 🔴 | | LinkedIn Ads | 2.9% | 2.7% | -0.2pp ✅ stable | | Referral | 4.5% | 4.2% | -0.3pp ✅ stable | **Finding:** Google Ads conversion rate collapsed from 3.2% to 1.1%. All other channels are stable. Google Ads also saw a 40% traffic increase — so we're getting MORE traffic of WORSE quality. ### By Device (Google Ads only) | Device | Before | After | Change | |--------|--------|-------|--------| | Desktop | 3.8% | 3.5% | -0.3pp ✅ | | Mobile | 2.4% | 0.4% | **-2.0pp** 🔴 | **Finding:** The problem is specifically Google Ads mobile traffic. ### By Campaign | Campaign | Before CVR | After CVR | Spend Change | |----------|-----------|-----------|-------------| | Brand terms | 8.2% | 7.9% | — | | Competitor terms | 3.1% | 2.8% | — | | Generic terms | 2.5% | 0.3% | **+180%** 🔴 | **Found it.** The "generic terms" campaign had its budget tripled (from $2K/day to $5.6K/day) on April 5. The expanded targeting is driving low-quality mobile traffic. --- ## Step 4: External Factors | Factor | Relevant? | |--------|-----------| | Seasonality | No — April has no known seasonal effects for B2B SaaS | | Competitor launch | Checked — no major competitor announcements this week | | Google algorithm update | No core update reported in April 2026 | | Ad policy change | No — our ads are still approved and running | | **Budget change** | **YES — marketing increased generic keyword budget 180% on April 5** | --- ## Step 5: Root Cause Hypothesis **Root cause:** On April 5, the marketing team increased the Google Ads "generic terms" campaign budget from $2K/day to $5.6K/day to hit a monthly spend target. The additional budget exhausted high-intent keywords and began serving ads for broader, lower-intent queries — primarily on mobile devices. **Evidence:** 1. Conversion drop timing matches the budget increase exactly 2. Only Google Ads channel is affected — organic and direct are stable 3. Within Google Ads, only the generic campaign shows the drop 4. Mobile traffic tripled while mobile conversion rate dropped 83% 5. The search terms report shows new queries like "free project tools" and "how to manage tasks" — informational intent, not purchase intent **Confidence: 95%.** The timing, isolation to one channel/campaign, and search term data all converge. --- ## Step 6: Recommended Response ### Immediate (today) 1. **Reduce generic campaign budget back to $2K/day.** This instantly stops the low-quality traffic bleed. 2. **Add negative keywords** for informational terms: "free," "how to," "what is," "tutorial," "template" (these are driving clicks but not conversions). 3. **Set mobile bid adjustment to -40%** on the generic campaign until quality improves. ### This week 4. **Review search terms report** and add all irrelevant terms to the negative keyword list. 5. **Segment the generic campaign** into two: "high-intent generic" (terms with commercial intent) and "awareness generic" (measured by different KPIs — CTR, not conversion). 6. **Set conversion rate alerts** on all campaigns: alert if any campaign drops below 1.5% for 48+ hours. ### This month 7. **Reallocate the $3.6K/day budget increase** to higher-performing campaigns: - +$1K to competitor terms (3.1% CVR, room to grow) - +$1.5K to LinkedIn Ads (2.9% CVR, untapped scale) - +$1.1K to retargeting (typically 6-8% CVR) --- ## Step 7: Monitoring Setup ### Automated Alerts | Alert | Condition | Check Frequency | Notify | |-------|-----------|----------------|--------| | Conversion rate drop | >20% below 7-day average | Hourly | #marketing Slack | | Campaign CVR anomaly | Any campaign <1.5% for 48hrs | Daily | Marketing lead email | | CPA spike | CAC >$500 for any campaign | Daily | #marketing Slack | | Traffic mix shift | Any source >30% change in share | Daily | Marketing lead | | Mobile CVR divergence | Mobile CVR <50% of desktop CVR | Daily | #marketing Slack | ### Dashboard Addition Add a "Traffic Quality Score" panel to the marketing dashboard: - Conversion rate by source (daily trend) - Conversion rate by device (daily trend) - Search terms word cloud (weekly refresh) - Budget vs. conversion rate scatter plot per campaign