Hello from windy Wroclaw today ☕️
Quote from David Allen author of Getting Things Done.
Notes from Lukas Vermeer's participation in 1000 experiments club.
Quote
From David Allen
Connecting the same core idea from another angle:
An expert is someone who, over many years, manages to remain confident enough to keep trying and humble enough to keep learning.
So what?
Everyone's making it up as they go along — and it's ok.
Keep confident. Keep learning.
Make it up. Make it happen.
Note
From Lukas Vermeer
Marylin Montoya and the ABTasty team closing 2021 with an outstanding episode of 1000 Experiments Club ft. Lukas Vermeer.
We have to distinguish between the teams that are responsible for running the experiments and using that to inform their product development and an experimentation team which is more on the meta level thinking about how is our experimentation scaling?
What are currently the friction points that our people are having?
What is stopping them from running better experiments?
What are the risks that we have that we see to our experimentation strategy?
In Positive Experiments #5, I used this quote to represent my biggest learning in the world of experimentation for 2021.
I’m using those four questions as the “types of problems” our Center of Excellence deals with in iTech Media.
There comes the point where you've tried all the different button colors, all the size of like all the optimization and sort of tweaking the UI shouldn't be an important part in the validation program.
I was always much more interested in those experiments that really changed the company direction.
Good confirmation of how we're evolving CRO at iTech.
From Kevin Andreson’s presentation during Conversion Hotel, I like how he describes what it takes for evolution.
For him, getting strategic means applying the scientific method to the wider business, validating strategic programs with pragmatic questions.
On the flip side, embedded experimentation means enabling Product to test on their initiatives without maintaining a parallel lane of tests that compete for resources nor generate roadmap friction.
In the book leadership is language, the author talks about escalation of commitment, the more time and energy you have invested into committing to something, the more difficult it is to change course.
Bottomline, the prettier the roadmap, the bigger the commitment to the roadmap, the more difficult it becomes to change direction.
Commitment bias was something outside my radar until this episode.
"There was one key assumption that he (Johnny Longden) made that I very strongly disagree with.
He made a distinction between product development and experimentation. And I don't think that distinction should exist, okay.
The product development should be the teams that are running experiments because they are the ones that are making changes.
They are the ones that should understand the customers, they are the ones where the experimentation and the results will drive their decisions.
So this distinction gives me the idea that there are product teams who are not experimenting. I don't think those teams should exist. I don't think you should have product teams are not testing.
In iTech, this is a key distinction we are making as part of our agreement with Product that came out of our Experimentation Framework.
Product is ultimately responsible for the number of tests because they are the ones with developers changing things.
Experimentation is responsible for the quality and trustworthiness of results.
Marylin M.: Any last comment? How would you summarize our chat in a sentence?
Lukas V.: Start small. Rome, wasn't built in a day. [...] If you think about the scientific method to build measure loop. It doesn't have to be complicated, right?
Starting small and empathizing with Product maturity.
AB test is the gold standard for decision-making, however, some data is better than no data. There will be times where pre-post analysis or usertesting.com will be your best friend.
Been there.
Stay true to the evidence hierarchy and, if you’re doing business, not science, move on!
Here you have the LinkedIn version of the notes I took:
Amazon is doing this, we got to do it as well!
FOMO seems to have been a driver for Booking to go from walking to flying with experimentation. I loved how that journey started after Ronny's "how we do it at Amazon" presentation.
I would argue that the role of the central team is not to run experiments, but to look at that flywheel and to figure out where's currently the most friction and drive investments so that the flywheel can spin faster.
As we continue to hand over ownership of the number of tests to the Product organization, our Q4 in iTech Media was all about identifying the constraints in our flywheel and writing down initiatives.
"For your first experiment, you want what we call a counterintuitive result. You want something that teaches people that it's sometimes difficult to predict what the next round is going to do because that starts the mindset of the scientific method, much more than the optimization."
Bet the Test was a successful initiative to get the entire org reflecting on counterintuitive results! 🎉
Haven't checked what we've done?
𝐖𝐡𝐚𝐭 𝐈 𝐦𝐢𝐬𝐬𝐞𝐝:
Experimentation = innovation?
In your world, what does it mean to have a growth mindset? Can you give an example?
How do you train someone on experimentation? Are there frameworks that you recommend to product managers or processes that they should follow to structure the experimentation within a roadmap or their general guidelines? — You started blaming roadmaps and didn't answer the question! 😂