Positive Experiments #1
Navigation experiment, Hotjar for experimentation, Chinese market dip, rooftops and landscapes.
The newsletter experiment
As of today, I'm writing this to 27 people.
I will prioritize information depth instead of breadth.
Meaning I'm counting on your replies and comments to make this a reading worth your time. Of course, I might email you directly as well.
Experiments I'm running. Projects I'm leading.
Experiment question: Have you ever analyzed an experiment using a Bayesian approach?
Sitewide navigation experiment
I use Craig Sullivan’s hypothesis toolkit. Check here for his latest iteration:
BECAUSE WE SAW... that current usage of the mega menu is marginal and recent wins from a parent product navigation with wide mega menu design
WE BELIEVE THAT BY... updating the navigation content, taxonomy, look and feel our users will be better served due to us making it easier for them to complete their task.
FOR... all users landing in the product (sitewide experiment)
WE WILL KNOW WE`RE SUCCESSFUL... when we see an increase in navigation usage and more users visiting more than one page, ultimately increasing onsite engagement.
Metrics and analysis strategy
My primary metric is engagement. Unfortunately, I can't disclose the metric definition, so let's use bounce rate to keep it simple. I expect a decrease in bounce rate.
My secondary metrics are:
conversion to menu usage;
Avg. Pageview per unique user — which I will control but not run any statistical test because I will not have enough sample size to measure significance on continuous metrics.
Again, this product has limited traffic for a frequentist approach, so I'll be leveraging more from the "Simplistic Bayesian" analysis I've been developing with my product teams.
I wrote a case study using an experiment I closed last month as an example. Click to check.
If you find time, check it out and bring questions. Javid from Postman helped me cover some blind spots in the methodology.
The bottom line is.
Experiment design specifics
I'm currently running a site-wide navigation test using Adobe Analytics and SiteSpect as my testing tool.
Page performance is essential to our business, and SiteSpect was the best performing testing tool we trialed.
On SiteSpect, we are using a Feature Flag setup that makes the build more robust than client-side search and replace methods. This is important because of the number of moving parts involved in our nav experiment.
Hotjar setup for experimentation
Two quick things about Hotjar for experiments: tagging variants and hotjar events.
Tagging Variants
As part of our SiteSpect implementation, Hotjar receives a user scope custom dimension with the variant ID.
Why does it matter? It allows product designers and UX researchers to watch recordings after the experiment goes live for research and QA purposes.
I usually recommend delivery leads and project managers to have a checkpoint two days after the test goes live to watch some recordings and make sure analytics are being collected as expected.
Hotjar events
I asked our front-end developer to fire the snippet below together with the first menu-related event.
hj('event', 'navigation_interaction');
Why does it matter? It allows product designers and UX researchers to filter recordings and narrow down the time spent in this activity.
Not everyone enjoys the "Hotjar recordings and chill” session — I do 🍿
Answering the right questions
With both Hotjar configurations set, my team can answer, "what is the average experience and journey like for users that land in the variant and interact with the new menu?"
What's the growth case?
For a business where SEO is a crucial growth lever — a.k.a content growth loops — we believe that improving the internal linking hierarchy and reducing bounce rate will deliver quality content signals to Google, leading to better avg. page ranking in the mid-term and better conversions in the short-term.
As a visual learner, let's visualize this using Reforge's Company Generated Content (CGC) growth loops as a frame.
If a team at Hubspot were running this particular experiment, they would be optimizing the loop at Content Distributed (3) and the conversion rate between New Visitors (4) and New Lead/Follower (1).
A twist of stock market
Stocks question: Are you invested in the stock market? If so, what are you bullish on? If not, what holds you back today? Can I help?
I'm a young individual investor, and I can see myself growing the portfolio for the next 40 years. So I'm fine with the long game. The long game allows me to take more risks.
Why? Because I have no clue about the market behavior next week, I'm 95% sure that in 10 years, it will be stronger. In Buffet's words (at 2:50):
you just don't know what's going to happen, you know that America's tailwinds are not exhausted, you're going to get a final result if you own equities over a long period of time.
That means I'm now excited about the dip in the Chinese market amidst the uncertainty of their regulatory environment. So BABA and JD are my picks.
From the analysts I follow:
You can follow my portfolio strategy with this DataStudio report here. I posted here the explanation and rationale behind this dashboard on LinkedIn.
Have questions or opinions to share? Let’s help each other and invest better.
Sprinkled with positivity
I enjoy rooftops because of the views.
I appreciate landscape paintings because of the views.
I came back from London last Monday, and I love the fact museums are free in the UK. This one is from The National Gallery.
An imaginary harbor is swathed in the evening sunshine. Fishermen engage in gentle activity in the foreground while a ship is towed into port. Vernet paints the tiny figures furling the sails of the ship with painstaking precision.
Content worth sharing if you have extra time
A Pirate's take on Strategy vs. Tactics
https://diogomonica.com/2018/10/07/a-pirates-take-on-strategy-vs-tactics/#none
The Gist: "If we are defining a set of end-states leading to the desired outcome, we are designing a strategy; if we are discussing how to better reach those end-states, we are discussing tactics." This post was recommended to me by David Sanches. He's fond of the idea "Someone's strategy is someone else's tactics."
First Principles Thinking - If Elon Musk Did CRO - CRAP Talks - Metric trees
https://craptalks.com/blog/2020/02/first-principles-thinking-if-elon-musk-did-cro/
This one came from a chat with Bhavik Patel from CRAP talks earlier this month, as we were both involved in projects for experimentation insights and documentation centralization.
How to read A Profit & Loss Statement — Brian Feroldi on Youtube
If you invest, you need to learn the language of business, and the language of business is accounting.
I’ve been trying to step up my accounting game to read balance sheets and this video helped me A LOT. Their lemon stand vs. Apple example is super cool.
Self-promoting and #shamelessplug
Growth Mentor workshop — Make better product decisions with growth experiments
35 min of me being an experimentation fanboy + 20 min of edgy Q&A:
Why experiments in the first place? How does it support decision-making
Different experimentation workflows — Product Initiatives, CRO, and Safety Net
How to understand experiment as a process with a real (but simplified) example
Experiment Nation Conference and talks I’m looking forward to
Together with Andrea Corvi, Sr. Experimentation Manager at iTech, we expanded the topics I covered in the GrowthMentor workshop with hiring recommendations to Product and Hiring Managers.
If you’re planning to grow either a CRO or a Product team with an innovation flavor, I’m positive you’ll find value in the last part of our chat.
Still from the conference, I’m looking forward to watching:
Meta-analysis — August 2021
2 experiments launched;
18 blog posts read;
21 hours spent studying financial market;
40 hours producing content;
Was it worth your time reading? If so, here’s my last CTA.
I’m curious how word of mouth can be impactful leveraging this newsletter. I will share word-of-mouth calculations in future Meta-analysis.