Recently, I gave a talk on how “A Locust Swarm Can be a Good Thing!” at Stir Trek in Columbus. The talk covered our experience of load testing and preparing for hundreds of thousands of users on the first day of the .Realtor site launch.
This was a challenging environment with lots of external dependencies of varying capacity and failure tolerance. We knew from the beginning that we’d never be able to load test all of our dependencies at once, so we had to figure out ways to test in isolation without spending all our time rewriting our test infrastructure. Additionally, our load testing started late enough that we would not be able to coordinate an independent test with all of our dependencies in time. We also needed to figure out what our users would do without having ever observed user behavior on the real site. We built a model user funnel to capture our expectations and continually tweaked it as we discovered new wrinkles. This funnel formed the basis for our load test script and allowed us to prioritize our integration concerns.
In the end, we learned a lot about making our workflows asynchronous, linux kernel optimization, decoding performance metrics, and building giant DDoS clouds of load test slaves. We also learned that load testing should start “earlier”. Conversations about load and user behavior drive new requirements and testing can uncover fundamental infrastructure problems. Decoding and isolating performance problems can require a lot of guessing and experimentation; things that are difficult to do thoroughly with an unmovable launch date. It’s also difficult to make large scale changes to an application with confidence under time pressure. One of my key takeaways is to be nicer to external partners. The point of load testing is to find the breaking points of a system and most people don’t like when their toys get broken. Building trust and safety into that relationship is very important before trying to figure out where and how something went wrong.
Check out the slides to the original talk here.
Stir Trek is an excellent conference with extremely thoughtful organizers and friendly people. 20 people came to talk to me in person after my talk, which was great! Tickets sell out very quickly, but I recommend getting in next year if you can!