Iterating Quickly with A/B Tests
Infrastructure for Stacking Wins
You start with a single test. This leads to two possible states in your code.
But say you add just two more:
The number of possible code states begins to expand exponentially, 2^n to be precise, where n is the number of tests. That’s 2^n code states to manage, test, and monitor. And within each of these categories unique challenges are presented:
Management — A/B tests encoded with feature flags are often ramped up manually, over long periods of time. Your organization has to remember to track them, and you’ll have to remember to clean them up when you’re done.
Testing — Unit and Smoke tests by default usually test the “control” side of the test—the side with no experimental changes. You have to make sure to code in variations for each test, and that undermines a key value in automated testing: testing regressions on the periphery.
Monitoring — With similar challenges to unit testing, monitoring also comes with a scale challenge. Monitors are typically built to detect drastic changes in traffic or behavior. If you’re testing on a small portion of traffic, drastic effects in your variations may not be visible to your monitors immediately.
So what can we do about this?
If you have continuous deployment, site traffic is likely already directed to a load balancer. When you make a new deploy, your load balancer switches from your old build to your new build.
We can use the load balancer to instead move traffic gradually from the old version of the site onto the new version.
This is called a “canary deploy,” and it can be done for every continuous deployment. This offers several advantages compared to feature flag A/B tests:
They come with every deploy without extra effort for developers
They are ramped up quickly and cleaned up just as fast
Because divergent states aren’t allowed to compound, they are simpler to test and monitor
Of course, there are some times when you’ll want a more traditional A/B test. It’s likely that a few commits will go out with each deploy; if you require a high level of specificity, you’re better off using a feature flag. This often coincides with when you are testing a longer running feature, and are not just checking for big changes but for specific, incremental effects.
Nevertheless, there’s huge advantage to integrating canary deploy A/B tests into your CI/CD pipeline. As much as possible, we shouldn’t add special testing and monitoring infrastructure with every new feature—it should come baked in with every new feature automatically. In this way, we can iterate as quickly as possible without compromising quality or integrity.