A/B Test Statistics Calculator

ab-test-stats

Calculate A/B test statistical significance. Use when: determining if test results are significant; calculating required sample size; estimating test duration; analyzing conversion experiments; making data-driven decisions

guia-matthieu/clawfu-skills·0installs149stars

SKILL.md

Full skill instructions

A/B Test Statistics Calculator

Calculate statistical significance for A/B tests - know when your results are real, not random chance.

When to Use This Skill

Test analysis - Determine if results are statistically significant
Sample planning - Calculate required sample size before testing
Duration estimation - Know how long to run experiments
Power analysis - Ensure tests can detect meaningful differences

What Claude Does vs What You Decide

Claude Does	You Decide
Structures analysis frameworks	Metric definitions
Identifies patterns in data	Business interpretation
Creates visualization templates	Dashboard design
Suggests optimization areas	Action priorities
Calculates statistical measures	Decision thresholds

Dependencies

pip install scipy numpy click

Commands

Check Significance

python scripts/main.py significance --control 1000,50 --variant 1000,65
python scripts/main.py significance --control 5000,250 --variant 5000,300 --confidence 0.99

Calculate Sample Size

python scripts/main.py sample-size --baseline 0.05 --mde 0.02
python scripts/main.py sample-size --baseline 0.10 --mde 0.01 --power 0.90

Estimate Duration

python scripts/main.py duration --traffic 1000 --baseline 0.05 --mde 0.02

Examples

Example 1: Analyze Test Results

# Control: 1000 visitors, 50 conversions (5%)
# Variant: 1000 visitors, 65 conversions (6.5%)
python scripts/main.py significance --control 1000,50 --variant 1000,65

# Output:
# A/B Test Results
# ─────────────────────────
# Control:  5.00% (50/1000)
# Variant:  6.50% (65/1000)
# Lift:     +30.0%
#
# Statistical Analysis
# ─────────────────────────
# p-value:      0.089
# Confidence:   91.1%
# Result:       NOT SIGNIFICANT (need 95%)
#
# Recommendation: Continue test for more data

Example 2: Plan Sample Size

# Baseline 5% conversion, want to detect 20% relative lift (1% absolute)
python scripts/main.py sample-size --baseline 0.05 --mde 0.01

# Output:
# Sample Size Calculator
# ──────────────────────────────
# Baseline conversion: 5.0%
# Minimum detectable effect: 1.0% (20% relative)
# Target conversion: 6.0%
#
# Required per variant: 3,842 visitors
# Total required: 7,684 visitors
#
# At 1000 daily visitors: ~8 days

Key Concepts

Term	Definition
p-value	Probability result is due to chance
Confidence	1 - p-value (usually want 95%+)
Power	Probability of detecting real effect (usually 80%)
MDE	Minimum Detectable Effect - smallest lift worth detecting
Lift	Relative improvement (variant - control) / control

When Results Are Significant

p-value	Confidence	Verdict
< 0.01	> 99%	Highly Significant ✓
< 0.05	> 95%	Significant ✓
< 0.10	> 90%	Marginally Significant
≥ 0.10	< 90%	Not Significant ✗

Skill Boundaries

What This Skill Does Well

Structuring data analysis
Identifying patterns and trends
Creating visualization frameworks
Calculating statistical measures

What This Skill Cannot Do

Access your actual data
Replace statistical expertise
Make business decisions
Guarantee prediction accuracy

Related Skills

cohort-analysis - Analyze user cohorts
funnel-analyzer - Analyze conversion funnels

Skill Metadata

Mode: centaur

category: analytics
subcategory: statistics
dependencies: [scipy, numpy]
difficulty: intermediate
time_saved: 3+ hours/week

Install skill

Copy and run in your terminal

npx -y skills add https://github.com/guia-matthieu/clawfu-skills --skill ab-test-stats

View on GitHub

skills/analytics/ab-test-stats/SKILL.md

landing-page-optimizer
Optimize landing pages for maximum conversions using proven frameworks from Unbounce and Oli Gardner—apply the \"one goal, one message, one action\" principle with data-driven design, copy, and CTA best practices. Use when: **Create a new landing page** for a campaign; **Optimize an existing land...
0 installs
nurture-sequences
Transformez l'attention en conviction avec des séquences email evergreen qui font progresser les leads de curiosité → confiance → call booké, basé sur la méthodologie A5 du Marketing Swarm de Lasse Flagstad. Use when: **Créer une séquence evergreen** - Emails automatiques qui tournent indéfinimen...
0 installs
competitor-monitor
Monitor competitor websites for changes. Use when: tracking competitor pricing changes; monitoring new features; watching for content updates; alerting on website changes; competitive intelligence
0 installs
rlm
Process large codebases (>100 files) using the Recursive Language Model pattern. Orchestrates parallel sub-agents to map-reduce across files without context rot. Use when: analyzing large repositories; auditing security or auth across many files; finding patterns across 50+ files; processing larg...
0 installs

More skills from guia-matthieu

guia-matthieu/clawfu-skills

lead-qualification-meddic

Qualify enterprise opportunities using MEDDIC/MEDDPICC framework for complex B2B sales cycles

149 0

View

guia-matthieu/clawfu-skills

expansion-signals

Identify upsell and cross-sell opportunities through usage patterns, growth signals, and account behavior analysis

149 0

View

guia-matthieu/clawfu-skills

content-writing

Create ridiculously good content using Ann Handley's \"Everybody Writes\" methodology—practical, reader-focused writing that serves your audience and achieves your goals. Use when: **Write blog posts** that engage and convert readers; **Create email content** that gets opened and clicked; **Craft...

149 0

View

guia-matthieu/clawfu-skills

podcast-seo

Optimize podcast content for search engines using transcripts, structured show notes, and strategic keyword targeting to drive organic discovery. Use when: Launching a new podcast and planning for discoverability; Adding transcripts to existing episodes for SEO benefit; Optimizing show notes for ...

149 0

View

guia-matthieu/clawfu-skills

brand-strategy

Create distinctive brands that customers choose because they believe there's no substitute, using Marty Neumeier's Brand Gap and Zag frameworks Use when: **Building a new brand** from scratch (startup, product, service); **Repositioning an existing brand** that's become commoditized; **Defining b...

149 0

View

guia-matthieu/clawfu-skills

lead-scoring

Build and apply lead scoring models combining firmographic fit, behavioral signals, and intent data for sales prioritization

149 0

View

guia-matthieu/clawfu-skills

lead-qualification-bant

Qualify leads using the BANT framework (Budget, Authority, Need, Timeline) with discovery questions and scoring criteria

149 0

View

guia-matthieu/clawfu-skills

pricing-strategy

Design products around price using Madhavan Ramanujam's \"Monetizing Innovation\" methodology—determine willingness to pay before you build, not after. Use when: **Set pricing for a new product** before or during development; **Validate willingness to pay** before investing in features; **Structu...

149 0

View

guia-matthieu/clawfu-skills

product-discovery

Build products customers actually want. Apply Marty Cagan's Silicon Valley-tested framework to discover solutions that are valuable, usable, feasible, and viable. Use when: **New product development** when validating what to build; **Feature prioritization** to ensure you're solving real problems...

149 0

View

guia-matthieu/clawfu-skills

cognitive-biases

Apply Kahneman and Thaler's behavioral economics principles to marketing—understand how customers actually make decisions and design experiences that work with human psychology, not against it. Use when: **Improve conversion rates** by removing friction and applying behavioral nudges; **Design pr...

149 0

View

guia-matthieu/clawfu-skills