Scaling experimentation programs using first principles thinking
First principles is a superpower.
First-principles thinking is a problem-solving method where you break a system, a process, or a situation down into its fundamental parts and then arrive at a solution by reconstructing them.
A model is usually the outcome of putting pieces together, a more methodic way of thinking about the variables involved in the system.
I see first-principles thinking as a reverse-engineering exercise.
Being able to think deeply about the problem instead of jumping into solutions, in my experience, is perceived as a seniority trait.
To experimenters and product leaders, flexing first principles is a superpower.
Focusing on the quantity of experiments early on
Last month I recommended Bhav's First Principles thinking, which focuses on the number of experiments as a program objective.
Quantity of experiments across the network of products was the metric we worked on the most during the maturity phase of iTech Media's experimentation program.
We learned from Ronny K. and Lukas V. that total number of experiments was the metric they focused and succeeded early on.
Metric driver tree is a first-principles tool
First, we’ll need to break down your outcome metric into its full list of input metrics. We’ll do this via qualitative analysis, using a tool called a driver tree.
This will help us ensure we have a list of actionable metrics to populate our lever dashboard with.
— From Reforge Mastering Product Management
Reforge expects Product Managers to build their own metric trees qualitatively as the first step to building actionable dashboards.
Under this frame, Bhavik's contribution is clear as a breakdown metric tree for the total number of experiments across the business as an outcome metric.
Is this enough to scale up your experimentation program?
On one hand, I believe Bhavik delivered the best strategic starting point for new conversion programs.
On the other, after organizations move past 250 experiments per year, prioritizing the number of experiments has diminishing returns.
The case against funnels
The number one blindspot of funnels is the feeling of "to keep this growing, we must add more and more on top of the funnel!"
This goes against compounding effects, which to me and to Einstein are the 8th wonder of the modern world.
Self-reinforcing loop models are superior for that reason.
When I flip Bhav's tree, I can hear "that's why we need more developers! More ideas per person! More more more!"
Floops™️
Paraphrasing Socrates, “the only absolute truth is that there are no absolute truths”.
This fight between the “absolute rights and wrongs” has always existed and has always been a problem: depending on who you decide to listen to and trust, you’ll always be discarding a big chunk of the truth.
Social Media has only made this worse, and we in the Marketing and Business world are probably amongst the most obnoxious with our “this is THE RIGHT WAY to do X”.
Well… chances are it ain’t.
— David Sanchez, the contrarian experimenter.
—
After Growth Series, David had strong opinions about modeling everything as loops. For him, every loop has a funnel, and I must say he's right.
How do those ideas connect? What is the loop behind the funnel?
Back to the experimentation flywheel
I reference Aleksander's work so much that it feels like a broken record.
This is the starting point I use for socializing the vision for maturing experimentation in every business I support.
If every loop has a funnel, Bhav's funnel is the breakdown of step one, running more A/B tests to support more decisions.
When I compare his metric tree to the core metrics suggested for each step it's clear to me how the loop and the funnel connect.
Floops™️ 😂
Three maturity stages
I tend to see experimentation programs evolving in three maturity stages:
Focus on getting more tests out of the gate — step 1
Socialize the value and get people interested — steps 2 and 3
Reduce the barrier to testing — steps 4 and 5
Structuring stages as a timeline is vital for storytelling with leadership because it clarifies how Experimentation fits the company vision.
—
In iTech Media, I see their program moving towards the last stage designing automation and CMS integrations to minimize the cost of testing.
Interesting statistic
I heard from Craig Sullivan during last Experimentation Elite conference that Booking.com, the experimentation giant, had around 80% of its total number of experiments automated via their CMS. Food for thought.
Contributions
First principles thinking is a reverse-engineering superpower developed by seasoned product leaders and experimenters.
Metric driver trees are helpful to visualize the fundamental parts of a problem and how variables are connected to yield the desired outcome.
Funnel modeling helps identify short-term action, but fails to incentivize compounding effects.
Loop models add another layer of abstraction that supports long-term vision.
Every loop has a funnel, that said, the same first-principles thinking can be applied to each step of the experimentation flywheel to identify the right metrics to move.
Shameless plug
"Mr. Growth Mindset"
I first met Dennis van der Heijden, Convert's Head of Global Happiness, at the first Accelerating Innovation with AB testing cohort at ScholarSite.
I remember sharing ideas around scaling experimentation programs, as most of my questions to Ronny K were around that.
At some point he brought the idea of Holocracy and how he was growing the organization with multiple flexible workplace awards, I was fascinated.
If you haven't crossed the concept before, here's a loom he put together.
Out of my content enthusiasm, Dennis connected me with Trina Moitra, Convert Head of Marketing.
Apart from common interests in the industry, we share similar outlooks on what it takes for nurturing a Growth Mindset.
I'm happy about this collaboration because it's great to be connected with positive and like-minded people.
Four-minute read where I share more about my journey, go-to resources, and some rules of thumb.
Thank you, Trina and the Convert team.
Handpicked from other brilliant minds
Andy Johns on innovation vs optimization
I've been working with corporate innovation, I believe it’s a career track for experimenters. I used many references from this post on my PRFAQ to raise awareness and secure budget with senior leadership.
Vassilena V. on having a side gig
A side project will teach you new skills, show off your drive to others, and help you earn confidence in everything you do.
Starting a side project is easy. Sticking to one is as hard as hell.
I love Vassyiena V's thoughts on the subject, I read it and kept nodding in confirmation.
Time — May 2022
13 hours reading or listening to audiobooks;
12 spent studying the financial market;
16 spent producing content;
11 on Reforge;
20 days traveling.
Life waves 🌊
Life and work alternate with sustainable harmony.
Striving to maintain Sleep between 45% and 48%.
Time, the most precious resource.
Rovinj, Croatia 🇭🇷
The thumbnail this time is from the beautiful sunset in Rovinj, Croatia, the Istrian paradise.