4 days ago•

A/B/n experiment designer with proper power analysis baked in

Views

5.0K

Copies

1.6K

Likes

452

Comments

Copy rate

31.0%

Prompt

You are a senior experimentation scientist at a top product company. I'm planning an [experiment_type] test with [num_variants] variants on our [surface] to improve [primary_metric]. The baseline conversion rate is [baseline_rate] and we see roughly [daily_traffic] eligible users per day.

Design the experiment end to end:
1. Hypothesis statement in the form "We believe X because Y, and we'll know we're right if Z"
2. Primary metric, guardrail metrics, and secondary metrics with definitions
3. Power analysis — required sample size per arm for MDE of [minimum_detectable_effect]
4. Duration estimate given daily traffic and multi-arm splitting
5. Randomization unit and stratification strategy
6. Pre-registered analysis plan including how you'll handle peeking, SRM checks, and novelty effects
7. Decision rules — what ships, what rolls back, what gets iterated

Be rigorous. Call out any assumption that looks shaky.

Customise this prompt

Fill in 7 variables to personalise this prompt

Experiment Type

Num Variants

Surface

Primary Metric

Baseline Rate

Daily Traffic

Minimum Detectable Effect

Preview

You are a senior experimentation scientist at a top product company. I'm planning an [experiment_type] test with [num_variants] variants on our [surface] to improve [primary_metric]. The baseline conversion rate is [baseline_rate] and we see roughly [daily_traffic] eligible users per day. Design the experiment end to end: 1. Hypothesis statement in the form "We believe X because Y, and we'll know we're right if Z" 2. Primary metric, guardrail metrics, and secondary metrics with definitions 3. Power analysis — required sample size per arm for MDE of [minimum_detectable_effect] 4. Duration estimate given daily traffic and multi-arm splitting 5. Randomization unit and stratification strategy 6. Pre-registered analysis plan including how you'll handle peeking, SRM checks, and novelty effects 7. Decision rules — what ships, what rolls back, what gets iterated Be rigorous. Call out any assumption that looks shaky.

Example output

# Experiment Design: Checkout Page Redesign (A/B/C) ## 1. Hypothesis We believe redesigning the checkout page to surface trust signals earlier and collapse the shipping/payment steps into a single view will increase completed-purchase rate, because qualitative research (n=18 session replays) shows drop-off clusters at the shipping-to-payment transition. We'll know we're right if completed-purchase rate increases by at least 2.5% (relative) over control, with no degradation in AOV or refund rate. ## 2. Metrics **Primary:** Completed-purchase rate = completed_purchases / checkout_starts, measured per assigned user across a 7-day attribution window. **Guardrails (directional harm must not exceed -1% relative):** - Average order value (AOV) - 30-day refund rate - Payment-error rate - Page load time (p95) - Customer support contacts per 1,000 orders **Secondary (directional, not for ship decisions):** - Time-to-purchase (median) - Coupon-application rate - Guest-checkout share ## 3. Power Analysis Assumptions: baseline rate 12.4%, MDE = 2.5% relative lift (absolute 0.31pp), alpha = 0.05 two-sided, power = 0.80, three arms (control + 2 treatments) with equal allocation. - Single-comparison sample size: ~71,400 per arm - Bonferroni adjustment for 2 pairwise comparisons (each treatment vs control): alpha = 0.025 → ~84,600 per arm - **Total required: ~253,800 users** Reality check: with Bonferroni we leave some power on the table. If sequential testing is available (e.g., mSPRT), we can reduce required N by ~15-20%. Recommend sequential. ## 4. Duration At 42,000 eligible users/day / 3 arms = 14,000 per arm per day. Required duration: ~6 days. **Round to 14 days** to cover two full weekly cycles and absorb novelty effects. Do not ship earlier even if significant. ## 5. Randomization - Unit: hashed user_id (logged-in) or anonymous_cookie (logged-out), sticky for 30 days - Stratify on: device class (mobile/desktop/tablet), new-vs-returning, and geo region (top 5 markets) - Use CUPED with prior-30-day order count as covariate to reduce variance ~20-30% ## 6. Pre-Registered Analysis Plan - **Frozen metrics and comparisons** logged in experiment registry before launch - **SRM check** daily; any p < 0.001 on chi-squared sample ratio triggers investigation and pause - **No peeking** for ship decisions; sequential testing thresholds pre-committed if used - **Novelty effects** monitored by comparing week 1 vs week 2 treatment effects; if delta > 30%, extend 7 days - **Segment analysis** pre-specified for mobile-vs-desktop and new-vs-returning only; all other segment claims flagged as exploratory ## 7. Decision Rules - **Ship treatment** if: primary metric lift > 0 with p < 0.025, all guardrails within ±1%, segment effects directionally consistent - **Rollback and iterate** if: any guardrail breached OR mobile lift negative while desktop positive (heterogeneity concern) - **Inconclusive** if: primary p > 0.05 after full duration → deprioritize, do not extend - **Shaky assumption flag:** baseline rate came from Q3 data; if Q4 seasonal shift pushes baseline above 15%, recompute power before reading results.

Prompt

Customise this prompt

Example output

Related prompts