Skill Profile
A/B Testing & Experimentation
"The observable action of designing, running, and interpreting controlled experiments that compare two or more variants of a product or experience in order to make evidence-based decisions about what to change."
YOUR SKILLS
Problems This Skill Solves
- A team disagrees about whether a new checkout flow will increase conversions — an A/B test provides objective evidence
- A marketing email subject line needs optimising — multivariate testing identifies which version drives the highest open rate
- A feature is shipping to a new market — a phased rollout experiment measures its effect on retention before full release
- A pricing change is being considered — an experiment isolates its true causal effect on revenue from seasonal noise
Roles That Use This Skill
1 total · 1 industryThis skill is concentrated in one industry.
Technology / Business / Finance
"A/B testing is mainly about statistical rigour — run the test long enough to reach significance, and the result is reliable."
Statistical significance reached with an adequate sample confirms the observed difference is not noise. It does not confirm the test was run during a representative period, that the metric being tested reflects user value, or that the winning variant does not have unintended effects on other metrics. An A/B test can be statistically valid and commercially misleading.
"A/B testing tells you why something works."
Experiments measure what happened; understanding why requires qualitative research. A/B testing is causal, not explanatory.
Research & Outlook
Experimentation is becoming a core operating model for digital product teams. AI is accelerating experiment design and analysis, but human judgement on what to test and how to act on results remains essential.
See This Skill In Action
Watch a professional demonstrate A/B Testing & Experimentation in a real working environment — what it looks like, how it's applied, and why it matters.
Analytics / Research
A/B Testing & Experimentation
Also Known As
Growth Path
Can set up a basic two-variant test using a third-party tool, understands the concept of statistical significance, and can read a results dashboard.
Designs experiments with correct sample sizes and power calculations, identifies common pitfalls (multiple testing, novelty effects, peeking), and presents results to stakeholders.
Builds or improves experimentation infrastructure, designs complex sequential and multi-armed bandit tests, defines organisational experimentation standards, and mentors junior analysts.
How to Practise
- 1.Run a simple A/B test on a personal project or open-source site — vary one element (headline, CTA colour) and measure click-through
- 2.Work through a sample size calculator and understand the relationship between effect size, power, and significance threshold
- 3.Read case studies from companies like Booking.com, Netflix, and Airbnb on their experimentation cultures
- 4.Analyse a publicly available experiment dataset (e.g. Udacity A/B testing course dataset) end-to-end
How to Prove
- ·Portfolio of past experiments with hypothesis, design, results, and decisions documented
- ·Contribution to an experimentation platform or internal tooling (pull requests, documentation)
- ·Certification: Google Analytics, Optimizely, or Udacity A/B Testing course completion
- ·Evidence of a product decision that was reversed or confirmed based on your experiment results