How do I use the "Perform A/B Test Analysis" prompt?

Copy the prompt template, fill in the variables (TEST_NAME, HYPOTHESIS, PRIMARY_METRIC, TEST_DURATION, SAMPLE_SIZE, TEST_RESULTS_DATA), and paste it into ChatGPT, Claude, or Gemini.

What AI tools work with this data prompt?

This prompt works with chatgpt, claude, gemini. Simply copy and paste the template into any of these AI assistants.

Tips for getting the best results?

Always include your raw data and sample sizes for accurate statistical calculations Specify your significance level (α) and desired power upfront for proper interpretation Consider both statistical and practical significance - a tiny but significant difference may not be worth implementing Include information about external factors during the test period that might influence results Ask for visualizations to be described when presenting results to stakeholders

data intermediate

Perform A/B Test Analysis

Analyze A/B test results with statistical significance testing. Get actionable insights and recommendations from your experiment data.

Works with: chatgptclaudegemini

Prompt Template

You are a data analyst specializing in A/B test analysis. I need you to analyze the results of an A/B test and provide comprehensive insights. Test Details: - Test Name: [TEST_NAME] - Hypothesis: [HYPOTHESIS] - Primary Metric: [PRIMARY_METRIC] - Test Duration: [TEST_DURATION] - Sample Size: [SAMPLE_SIZE] Test Results: [TEST_RESULTS_DATA] Please provide a complete analysis including: 1. **Statistical Significance Test** - Calculate p-value and confidence interval - Determine if results are statistically significant (α = 0.05) - Check for adequate sample size and power 2. **Effect Size Analysis** - Calculate the practical significance of the difference - Provide lift/improvement percentage - Assess business impact 3. **Data Quality Assessment** - Check for any data anomalies or outliers - Validate test setup and randomization - Identify potential confounding factors 4. **Actionable Recommendations** - Clear decision: implement, reject, or continue testing - Next steps based on results - Suggestions for follow-up tests if applicable 5. **Executive Summary** - Key findings in non-technical language - Business implications - ROI estimation if applicable Present your analysis in a clear, structured format with tables and visualizations described where helpful. Highlight any limitations or caveats in the interpretation.

Variables to Customize

[TEST_NAME]

The name or identifier of your A/B test

Example: Homepage CTA Button Color Test

[HYPOTHESIS]

Your original test hypothesis

Example: Changing the CTA button from blue to green will increase click-through rate by 15%

[PRIMARY_METRIC]

The main metric you're measuring

Example: Click-through rate (CTR)

[TEST_DURATION]

How long the test ran

Example: 14 days (March 1-14, 2024)

[SAMPLE_SIZE]

Number of users in each variant

Example: Control: 5,247 users, Variant: 5,189 users

[TEST_RESULTS_DATA]

Your test results data including key metrics

Example: Control (Blue Button): 5,247 visitors, 367 clicks, CTR: 7.0% Variant (Green Button): 5,189 visitors, 425 clicks, CTR: 8.2%

Example Output

# A/B Test Analysis: Homepage CTA Button Color Test ## Statistical Significance Test **P-value: 0.023** ✅ Statistically significant (p < 0.05) **95% Confidence Interval:** 0.2% to 2.1% improvement **Sample size:** Adequate for detecting the observed effect ## Effect Size Analysis **Relative Lift:** +17.1% improvement in CTR **Absolute Lift:** +1.2 percentage points **Practical Significance:** Moderate effect size with meaningful business impact ## Data Quality Assessment - ✅ Proper randomization confirmed - ✅ No significant outliers detected - ⚠️ Slight imbalance in traffic distribution (1% difference) - ✅ External factors controlled for ## Recommendations **Decision: IMPLEMENT** the green button variant - Expected monthly impact: +58 additional conversions - Estimated revenue increase: $2,900/month - Implementation risk: Low **Next Steps:** 1. Roll out green button to 100% of traffic 2. Monitor for 2 weeks to confirm sustained lift 3. Test additional CTA variations to optimize further ## Executive Summary The green CTA button significantly outperformed the blue version, increasing click-through rates by 17.1%. This improvement is both statistically significant and practically meaningful, with low implementation risk and clear positive ROI.

Pro Tips for Best Results

Always include your raw data and sample sizes for accurate statistical calculations
Specify your significance level (α) and desired power upfront for proper interpretation
Consider both statistical and practical significance - a tiny but significant difference may not be worth implementing
Include information about external factors during the test period that might influence results
Ask for visualizations to be described when presenting results to stakeholders