Positive Experiments #2
Validation strategy when AB testing is not possible, a financial milestone, and the Think Week practice.
Validation strategy when AB testing is not possible
AB tests are the best tool I have in my toolbox for decision-making. When I can’t use them I ask what pieces of evidence could we gather to increase our confidence before making a decision?
This is a summary case study of how I use Behzod Sirjani's decision-first approach to research from Reforge User Insights for Product Decisions program.
I cover steps one to three:
How we defined a decision with the product manager.
How we mapped the evidence gaps we had to cover to make a better decision.
How we planned the approach between product analytics and UX research.
The complete process is lengthier than what I’m presenting here and makes up to a very cool Miro board that I work on in a series of workshops. I’m keeping it concise to the most important steps while keeping a coherent story.
Setting the stage with our product development challenge
We have a template going through a redesign and we don’t have enough traffic to run a test even if we decided to keep it running for 6 to 8 weeks.
Our analyst suggests this template is the most frequent second page visited for journeys with more than one pageview.
The template supports the conversion journey and suggests high intent.
I can’t reveal specifics, however, to paint a picture think about a B2B product where most traffic lands on the homepage and the pricing page is receiving the redesign.
The strategy side of decision-first in three steps
1. Define the decision
Objective: We want to answer “Is this design solution ready to be launched? How do we know if it worked?”
Stage of development: We are at the design stage, we are not sure if the solution is usable nor if it’s ready to be launched.
Splash zone (risk): We defined it as a medium to high risk after considering the three dimensions below.
Consider user impact: At least 30% of all users will be affected.
Consider workload: The workload for designers and developers is quite significant (more than two sprints).
Consider stakeholders: The product owner and experimenter are the main stakeholders for the decision.
Validation: No red flag, this is part of an OKR Product initiative.
2. Map out the evidence
Define evidence needs: As a team, we brainstormed all questions we had that would support a better decision.
Characterize evidence needs: We considered how stakeholders prefer to consume data for decision-making.
Identify evidence gaps: We didn’t have answers for any of the questions, therefore they were all gaps to be covered by some sort of research.
3. Plan the approach
Instead of choosing research methods based on convenience or experience, we should evaluate the evidence we need and build up to a research design that will help us gather that evidence.
The methodology archetype map is what I use to define research type. There are four quadrants in the archetype matrix. Each quadrant signifies specific characteristics of evidence, and as a result, different archetypes of user research.
Conclusion and strategy outcome
Because this template change was considered medium to high risk and an AB test wasn’t the right tool for decision-making, we dedicated extra energy to plan a proper validation strategy.
We ended up with:
Unmoderated testing via usertesting.com to answer some of the qualitative open questions we needed. We knew the outcome would please our product owner.
Usability test reports and pre-post analysis to add the quantitative layer that would keep our experimenter confident with the decision.
We also decided to keep a Hotjar pool running for a longitudinal voice of the customer to contrast the “one-off” characteristic of the unmoderated test.
Is there anything I could clarify from the decision-first process so you can get more value?
A twist of stock market
The individual investor should act consistently as an investor and not as a speculator. They should be able to justify every purchase they make and each price they pay by impersonal, objective reasoning that satisfies them that they are getting more than their money's worth for the purchase.
— Benjamin Graham [adapted]
A financial milestone — $100K in the market
This is a special moment in my retail investor journey and I’m sharing it with you.
Studying the stock market is my #1 hobby and I translate a lot from what I do in the experimentation world to my investment hypotheses.
How do I play the game? A brief overview
I maintain three portfolios, Growth (40%), Dividends (57%), and Crypto (3%). Allocation strategy is something I started caring about after the $50K mark and I keep it simple.
In green (Relative % 2.0) is the ideal allocation for an individual stock. It is defined based on my conviction and hypotheses of future growth based on multiple data points I collect.
In blue (% Value) the current value of the stock in relation to the portfolio.
The game is to make blue match green.
The #1 learning — Time in the market, not timing the market.
The portfolio time series tells the story that keeping money in the market with patience pays off.
I didn’t get to “meme stocks” nor adventured with day trading. I’m a boring investor.
I bought into companies with a mission I trust, a strong economic moat, and straightforward revenue engines that I can explain to a 9-year-old.
This is by no means financial advice.
I’m just a fool dreaming about passive income.
Do and share your own research.
Sprinkled with positivity — Think Week.
"Life is easier when you know what you want—but most people don't take the time to figure out what they want.
It's not that we are completely lost, but our efforts are often slightly misdirected. People will work for years and ultimately achieve a lifestyle that isn't quite what they were hoping for—often, simply, because they never clearly defined what they wanted.
An hour of thinking can save you a decade of work."
This is an excerpt from James Clear's 3-2-1 newsletter and it’s pretty much why I made Think Week an official routine in my life.
Inspired by how Bill Gates does it, every end of the quarter I take a week off in a warm place to exercise future thinking.
I don’t take it to the extreme of “cabin in the woods”, but I do step out of daily operation as much as possible to focus on planning the next objectives and work on a set of relevant questions that I captured during the previous quarter.
The thinking theme this time was about defining life's mission and reviewing my investments in happiness.
I thought a lot about this video from Ben Felix. I believe you might like it :)
Handpicked from other brilliant minds
Why you should take a think week
Piggybacking the Think Week discussion, in this episode from My First Million they discuss briefly the benefits and how to go about this routine. Listen at minute 34:05.
I have more curiosity now than I have time.
The Think Week is a potential solution for curiosity but there are better solutions to unlocking time.
Also, I’ve been binge listening to this podcast lately. It feels like a hustler’s gold mine.
How to x10 increase the ROI you get from your web development spend
As a visual learner, I tend to get a lot of value from process diagrams.
I came across this post from Jonny Longden and I see many correlations to how we implemented our Experimentation operating system in iTech Media.
No test > Safety Net
Simple test > CRO
Complex test > Product OKR Initiatives
What can we learn from creative testing frameworks?
I participate in a couple of mastermind groups, one of my favorites is Growth Engineers hosted by Michael Taylor. The theme of the last session was all about creative testing at scale, from naming conventions to operational frameworks.
This post from QuantMar got my attention as it suggests a strategic approach to testing that I tend to use with Product Managers and teams I work with.
I read this as first validate the design solution (vertical thinking), then optimize the existing solution (horizontal thinking).
Self-promoting and #shamelessplug
700+ experimenters ready to go through 33 free and on-demand sessions from brilliant minds in the experimentation community.
🚀 Andrea Corvi and I open up the growth pains of scaling tests from start-up to multiple product teams.
Still from the conference, I’m looking forward to watching:
➤ Seven lessons from running 22,000 AB tests from Ayat Shukairy
➤ The scientific method in business from Jonny Longden
➤ Change management for Experimentation Culture Ruben de Boer
Meta-analysis — September 2021
2 experiments launched;
26 hours spent studying financial market;
33 hours producing content;
19.5 hours networking.
The #1 learning of Q3-2021 — Network works
I reached this realisation after chatting with Lorenzo Carreri.
After some point in our careers we tend to learn a lot more from others’ experiences than from courses or books alone. For me, this proved to be true based on how much I learned from intentional networking with new connections during Q3.
Was it worth your time reading? If so, here’s my last CTA.