Kohavi

ZZZ - OLD - Sphere Alumni except cohort 8

5 sessions x 2 hours over two weeks, starting 8 Apr 2024

8 Apr 2024

5 sessions x 2 hours over two weeks, starting 5 Feb 2024

5 Feb 2024

2 Dec 2024

5 sessions x 2 hours over two weeks, starting 4 Dec 2023

4 Dec 2023

5 sessions x 2 hours over two weeks, starting 28 Aug 2023

28 Aug 2023

5 sessions x 2 hours, starting June 12, 2023

12 June 2023

ZZZ - OLD - Sphere cohort 8

Sept 23 2024

June 24 2024

Nov 2023

3 March 2025

8 Sept 2025

17 March 2025 - Advanced AB

12 May 2025

7 July 2025

2 June 2025 - Advanced AB

21 July 2025 - Advanced A/B

16 Dec 2024 - Advanced A/B

15 Dec 2025 - Advanced

1 Dec 2025

22 Sept 2025 Advanced

23 March 2026

9 March 2026

26 Jan 2026 - noon PST

1 June 2026

15 June 2026 - Advanced

9 Feb 2026 - Advanced - noon PST

Many myths are based on some real limitation. The lesson reviews the claims and why they are usually myths in practice. Recording is 35 minutes + 15 minutes of Q&amp;A. Example mythsOverstated effect, concurrent experiments are invalid, use pre-post analysis with few users, 95% confidence has 5% error.

A/B Testing Myths Lightning Lesson

Full text of Chapter 1 and table of contents for this best-selling book by the course instructor.

Chapter 1 of Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing

The podcast has been viewed over 50,000 times.

The ultimate guide to A/B testing: Podcast with Lenny Rachitsky

Full text of Chapter 1 and table of contents for this best-selling book by the course instructor.

The podcast has been viewed over 50,000 times.

Interested in reading chapter 1 of my book: Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing?

A/B testing book Chapter 1

Learn from a world-leading expert the essentials of accelerating innovation with trustworthy A/B tests - private class

Dr. Ronny Kohavi

Innovation & AB Testing - the Essentials

Founder & Chief optimizer

Learn advanced A/B-testing topics from a world-leading expert. Ten topics in three days of 2 hours each.

⚠️ Warning: Advanced course 

Author of best-selling A/B testing book

Technical fellow and VP at Microsoft & Airbnb, co-author of A/B testing book

Advanced Topics in Practical A/B Testing

Data Scientist

Growth Marketer | SEM

Principal Software Engineer

Engineering Manager

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

Accelerating Innovation with A/B Testing

Technical PMM

Fullstack Developer

Senior Product Manager

Growth Lead (Product team)

Microsoft

Airbnb

Amazon

Through multiple real examples of well-run experiments and stories from Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the value of ideas.Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products (MVPs) and features, and make the organization more evidence-based and humble as it adopts and learns to use evidence from the gold standard in science—the controlled experiment.You will understand the challenges in designing and running trustworthy randomized controlled experiments (A/B tests), including the importance of the Overall Evaluation Criterion (OEC), scaling, common pitfalls, and Twyman's Law. We cover cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluation, AI/Machine learning and triggering, build-vs-buy trade-offs, challenges, and requested topics. The Advanced Topics course is designed as a technical follow-on: <a target="_blank" rel="noopener noreferrer nofollow" href="https://bit.ly/AdvancedABFS">https://bit.ly/AdvancedABFS</a>

Empower your organization to be data-driven and innovative

Internalize the humbling reality: we are poor at assessing the value of ideas

Understand the key advantages and limitations of A/B testing

Learn how to design metrics and the Overall Evaluation Criterion

Learn how to design trustworthy A/B tests

Learn about the cultural challenges

Learn about the relationship to AI and Machine Learning Modeling

Data science managers and scientists will be able to design and interpret the experimental results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make organizations more data-driven and efficient with fewer severe incidents through A/B tests

We cover advanced topics not typically addressed in courses and books and leave significant time for interactions and Q&amp;A.Classes are small: typically ~15 live attendees. The course assumes you already took <a target="_blank" rel="noopener noreferrer nofollow" href="https://bit.ly/ABCourseRonKohavi">https://bit.ly/ABCourseRonKohavi</a> or have equivalent knowledge.

Improve your A/B testing capabilities and build trust

Improve your A/B Testing capabilities and trust

Discuss your own questions

Designed as a follow-on to the flagship course: https://maven.com/kohavi/abtesting. 

Take that course first, or have equivalent knowledge

Follow-on to flagship course

Good understanding of p-value misconceptions, statistical power, OEC.

Statistics of A/B Testing

Data science managers and scientists who understand the basics of A/B testing and want to better understand advanced topics

Product and program managers who want to advance their understanding of A/B testing

NOTE: this is designed for private classes of at least 40 people. You pick the dates. Sign-up for the waitlist and reply to the email if you are interested in scheduling a class. This is about 6 hours of content taught over an 8-hour day (with breaks). The costs are detailed in https://bit.ly/ABCourseBulkDiscounts under "Essentials" class. For the flagship live class, visit https://bit.ly/ABClassRKFlagship (Maven doesn't allow making this link clickable, so please copy-paste).--- Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas. Trivial changes can be surprisingly useful, whereas large efforts often fail. A/B testing allows you to evaluate changes and accelerate innovation by building Minimum Viable Products and Features (MVPs). The organization becomes evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment. You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law. Learn the essentials of A/B testing by watching easy-to-consume videos at your own pace, ask questions on the Maven questions channel, and attend two interactive one-hour sessions for fun polls and Q&amp;A.

Course syllabus

6 hours of content

Course schedule

Organization leaders (VPs, directors, managers) interested in understanding the essentials of A/B testing to accelerate innovation

Program managers focused on growth, revenue, conversions, and prioritization who will understand how to better design robust clear metrics

Data science managers and scientists who do not need or want the 2nd part of https://bit.ly/ABClassRKFlagship (or prefer videos)

Anyone interested in making the organizations more data-driven and efficient with fewer severe incidents

Who is this Course For?

What you will learn from this course

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn how to designing trustworthy A/B tests

You'll understand the key tradeoff between the cost and fidelity of different methods. You'll learn about the key concept of EVI: the Expected Value of Information. You'll learn about different prioritization frameworks.

Learn how to generate and prioritize ideas

A/B Testing Replication Crisis: Lessons from 10 A/B tests

Ronny Kohavi

How we designed the 10 A/B tests to evaluate four patterns: choosing the metrics and duration for proper power.

How to design a trustworthy A/B tests

While we expected to see exaggerations, our results showed surprisingly small effects.

Most claims of large improvements are highly exaggerated

Key lessons from the Trustworthy A/B Patterns community project.

Lessons

A/B Testing Myths

Many myths are based on some real limitation.   We will review the claim and why it is usually a myth in practice

A/B testing myths

Overstated effect, concurrent experiments are invalid, use pre-post analysis with few users, 95% confidence has 5% error

Example myths

Suggest and vote on myths

Quick Introduction to A/B Testing

The pros and cons of A/B testing

How many users do you need (power)?  What are you optimizing for (the OEC)?

Necessary ingredients

Most experiments fail to improve the metrics they were designed to improve.  Plan to fail fast and pivot often.

Humbling statistics

That A/B test that improved purchases by 20% (e.g., by rounding the buttons) is bogus. Be skeptical.

Learn directly from Kohavi

Kohavi

Courses

Accelerating Innovation with A/B Testing

Advanced Topics in Practical A/B Testing

Innovation & AB Testing - the Essentials

Lightning Lessons

A/B Testing Replication Crisis: Lessons from 10 A/B tests

A/B Testing Myths

Quick Introduction to A/B Testing

Resources

A/B Testing Myths Lightning Lesson

Chapter 1 of Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing

The ultimate guide to A/B testing: Podcast with Lenny Rachitsky

Chapter 1 of Trustworthy Online Controlled Experiments: a Practical Guide to A/B Testing