Addressing Valid Pushback to A/B Testing

Sometimes people have really stupid reasons to push back on A/B testing. Other times.... they actually have a pretty good reason.

Author

Shiva Manjunath
March 20, 2025

Not every pushback to A/B testing deserves a thesis on why they’re wrong. In some cases, there is VERY legitimate pushback. Ton “The Don” Wesseling hopped on the pod to discuss this - to provide “legitimate” reasons on why some people don’t want to A/B test, then how to address them.

Plus, in this issue we have some funny skits, some bonus fun content I’m working on (CRO After Dark), and some thoughts on frameworks.

Latest Skit

Getting tests approved in 2025 is getting wild…. maybe we should simplify the testing approval process?

(Gerda KILLED it in this skit!)

Go follower her on LinkedIn: https://www.linkedin.com/in/gerda-vogt-thomas/ 

Latest Podcast Episode

Addressing Legitimate Pushback to Experimentation ft. Ton Wesseling by From A to B

Do people have legitimate pushback to A/B testing? I mean, some don't. They just hate being proven wrong. But there are others who DO have legitimate pushback. How do we address this? Ton "The Don" Wesseling offered…

Bonus: New Series called “CRO After Dark” with Sani!

What is it? Basically discussing topics in Experimentation / CRO that don’t necessarily give you tactical things to do, but it’s shit that’s probably on ALL of our minds!

Vibes are always immaculate when you get two unhinged hosts on the same podcast

Latest Meme

When the ‘we don’t need to test it’ idea loses…

There are MANY things we don’t know. That is the literal reason why this magic tool to “test” things to know if they work or nah exist, and are beautiful….

www.linkedin.com/posts/shiva-manjunath_experimentation-optimization-activity-7295071353727041536-SjJY

Thing on my mind

Frameworks.

They can be amazing starting points, but in other cases they may not be relevant to you. In fact, in some cases, they can actually be detrimental. It’s always important when approaching a framework to understand the reason for the way things are laid out - this is FAR more important than the past success of the frameworks.

There are frameworks for everything. CRO, product, even just frameworks for raising kids (I can’t attest to their success). But the crucial thing with all of them is making sure you take those baselines, understand it, and adopt it to your needs!

Check out the homie Paul Randall’s framework newsletter - he aggregates amazing frameworks all across the board to help you start thinking differently 🙂 

The SAVE FOR LATERNewsletter

The homie Paul Randall’s newsletter!

saveforlater.kit.com/newsletter

Things to go subscribe to:

Reply

or to participate.