Parellel Tests

Some people know them as A/B tests and i think thats really a shame. The name A/B testing implies your intent is to use the parallel testing suite of your choice to gather performance analytics usually on content based differences and variances in the UI. There is a huge value add to using A/B testing just for collecting performance analytics so i'm not claiming that it's a misuse of parallel test suite implementations. However, the value value adds with parallel testing doesn't just end with performance testing as it can be used to enhance your product release process, your QA process and even your development process.

Improving the release process

I've worked on web products where we've done production code releases at midnight once a week and on ones that have one or more releases a day during peak hours. Depending on your past experiences you may think that releasing during peak hours that often is just asking for the fail whale to strike on a daily basis. However, my experience with both was that the midnight releases during hours of really light load where the most chaotic and error prone by an order of magnitude. So coming from that background wouldn't releasing every day during peak hours be one of the most counter intuitive things ever?

There are no silver bullets and the best way to slay a beast is with lots of lead bullets. I can talk about the value of hiring competent developers, using continuous integration and various release strategies to improve the release process. However, the lead bullet of this blog post is parallel testing.

When you're developing a new feature you can start it off by wrapping both the new and old implementation in a parallel test and setting the exposure of the old implementation to 100% with the new implementation set to 0% with a manual override on your test user id so that your test account will be presented with the functionality of the new implementation. This new feature can then be committed up the chain and even to prod with the same 100% to 0% test ratio and public facing end users will not have access to the new feature. At this point the only people with access to the new feature would be typically developers, Q/A and various business owners inside the company all of which would be able to test the new functionality in a production environment.

Once the new functionality has been pushed to production, sufficiently tested internally and vetted by all product owners. The release of the product is simply a toggle of the percentage in the parallel test suite from 100% old 0% new to 0% old 100% new. The concurrent test wrapper can even be left in place for undefined amounts of time and you can roll it back and push it back out all from a GUI/WebUI.

Improving the Q/A process

The Q/A process at the places i've worked at is always tightly coupled with the release process so i've already inherently discussed some of the benefits on how parallel test suites can be used to improve the Q/A process by talking about how it can be used to improve the release process. IE: being able to test new features in production in a manner not publicly visible to users is a huge improvement to the Q/A process alone if you couldn't already do that before.

Another way parallel tests can be used to improve the Q/A process is they allow you to do selective releases. It's not uncommon for certain sites to have select users who are highly active, vocal or are just in good contact with the business. You can do selective releases where only employees and those selected users will get initial "exclusive" test access to the feature before it gets released with 100% exposure. This is an easy and almost free way of crowd-sourcing a part of your Q/A department to users you can trust and have past relationships with.

Improve the development process

It first occurred to me that parallel tests can be used to improve the development process when i encountered an article by John Carmack about parallel implementations. Basically whenever he wants to test an alternative implementation of a feature, instead of refactoring the old implementation in place or scrapping it he wraps the old and the new implementation in parallel tests that he can toggle in real time from a console.

This is great for if you want to run benchmarks against alternate implementations because theres a minimal impact on the rest of the code and if the new implementation turns out to be worse you can simply throw out the concurrent test and restore the original implementation as it was without having to dig through branches, tags or commits in version control.


  • Parallel tests for releases
    • Enable/Disable features from a control panel instead of only source control
    • Incremental releases - bucket testing
  • Parallel tests for Q/A
    • Selective releases - crowd sourcing Q/A
    • Private beta tests in production before public release
  • Parallel test for the development environment
    • Enables you to develop alternative implementations and test them back and foward with a toggle

See Also