The Thing about A/B tests — To Learn or To Ship by Anna Marie
A/B (split testing) — statistically valid way of seeing how good or bad the feature idea is
2 types:
- test to ship (suppose to fix smth or for product completeness).
Answers the question: will not solving this make us feel bad or be embarrassed about the product?
- test to learn (because we want to figure smth out).
Answers the question: will not solving this make us doubt the results of the test?
Things to know:
- perform at the same time (not in sequel)
- half of the tests will fail / half succeed
- you cna skip session tracking
New feature testing:
- Plan the test
- Run the test
- Analyst checks what metrics (top line metrics such as how many users came to the product, check if any interactions) moved on the global level. These move very little. Then Analyst checks local metrics i.e. did they do the new flow we created. Did it drive the top level metric.
Example of learning: this feature did what we expected from it on a local level, but it didn’t drive the usage, the growth or any other top level metric we looked for.
Then the team (Analysts, PMs etc) discusses their narratives: I think this is the right narrative given the fact I saw in my project… it’s because of this fundamental aspect that you haven’t thought about…
Choosing the focus group.
As per Lean customer development [book] the 80–90% of qualitative learnings (about any feature) comes from talking to 5 people.
When to A/B test
Always, except:
- when high code complexity and clean up
- often appears in Test to ship
Pro tip: for PM if you are not using the features that drive growth and usage - it doesn’t how perfect the UX you won’t be able to keep the lights on.
Telling a narrative about a product is crucial. Start by figuring out who you are communicating with and try to understand what is it matters to them. Spend half of 30 min to understand the client, then consult them the following 15 min.
