Agile Product Development — The power of CRO or how to make / lose millions
Why you are playing with fire if your optimization team is lacking expertise and how this can cost you millions:
The agile product development approach (or conversion rate optimization as many people call it) allows you to directly measure the impact of any
change you made on your website. While it is more complicated to measure the financial benefits of brand building campaings or tv commercials, agile product development (CRO) gives you precise and isolated results which is why the most successful companies in the market follow that approach.
A short introduction to testing
You run version A of your website/landing page against version B during the same period of time. Visitors are rolled randomly into each version. Collect enough data and you will see a valid result which variation is better and by how much. You will need to collect a sample size big enough to interpret the result and see if is statistically significant (meaning you can trust that the effect is really there and didn’t happen by chance).
Now the last part is the tricky part: the interpretation of data. Not every test that looks positive is a winner and not every test that seemed negative is a loser. Why? This is the concept of false positives and false negatives. Maybe your test showed -6% after 2 days so you become nervous. Letting this test run will hurt the business over time so it should be stopped, right? To know if it should continue to run or not you will need people with knowledge of statistics. They will have to decide if the current -6% are pure variance. It could very well be that a few days later the test is at +4% and you would have regretted stopping it too early. The same goes for tests that show positive tendencies right from the start. People tend to stop these too early because of their own confirmation bias (they knew from the beginning it was a good idea). But what might show +5% in the beginning can easily turn into negative effects and end up with -5%.
Imagine you had changed something without testing: you would need to rely on your gut feeling entirely and still you would never know if this change was welcomed by users. Sure you can compare year over year numbers but how accurate would that be? The change in numbers you might see could be caused by a million other factors — marketing spendings, weather, general economic situation, a global pandemic, new competitors in the market…
Scenario 1: Tests are real winners because the team has great expertise
Imagine you run 12 tests per year:
- 4 of these tests are positive and will be rolled out (implemented)
- the average uplift for these tests will be 5%
- the total uplift of all tests combined will be 21,5% (1.05⁴)
Check your yearly revenue and imagine you increased it by 21,5% because you were running these 12 tests. For a lot of companies this would be huge.
You will thank your product manager / conversion manager for his good work and the team will have to continue doing so. That’s why you will want someone in that position who knows what he/she is doing — that means knowing enough about statistics and validity of data and how to influence user behavior by knowing psychology.
Scenario 2: Tests are false positives because the team is lacking experience
Imagine another outcome — your team is not experienced enough and stops the tests too early before the data is valid:
- 4 of 12 tests are believed to be positive but in reality each of them is -3%
- the total downcast of these 4 tests is -12,5%
- your team believed these were winners and implemented them all
Your revenues will drop by 12,5% this year and you won’t even know why — you had such good test results, right?
Conclusion
So why do so many companies put such huge responsability in the hands of juniors and intermediates? Would you hire a junior as your CFO?
A lot of companies haven’t understood that the team responsible for agile product development (testing) is steering the revenue of the business.
If you don’t fill these positions with lots of expertise you are playing with fire.
Don’t we all wish for that hockey stick curve when it comes to growth? Sure agile product development can provide this but it can also inverse the effect so beware!
