Shadow deployment for test in production
testing
Solution
We did this at Twitter using Diffy. Not only do you want to capture V-next, you also want to compare it V-current. Further, you want to aggreagate and denoise this comparison over thousands of samples to gain confidence.
Problem
As well known, the most important parts of Test In Production (TiP) are: - Leverage real customer behavior/data - Mitigate the customer risk. I found the Shadow Deployment a promising approach. With this approach, the V-next product is deployed in production side by side with the V-current. When customer request arrives, a clone of the request is routed to V-next for testing purpose. But the result is never returned to customer. Thus NO risk is exposed to customer. Once the V-next is ready, we can simply switch to it and tear down the V-current. So it seems the critical parts are: - How to implement the routing logic for various kinds of projects, such as Web service, Web site, etc. - How to isolate the data generated during the test because it could have side effect for upstream and downstream systems in production. My questions are: - Is there anything else we need to care? - And is there any best practice/tools/design/solution to address these concerns? For the specific technology details, please check here: https://stackoverflow.com/questions/14599167/how-to-clone-iis-request-and-redirect-to-a-test-version-site (Unfortunately, SO community deleted above post...) Thanks!