Stress tests are normally used in combination with larger system changes or campaigns. The purpose is usually to find overall bottle-necks in the system, and when needed increase for example band width, or number of servers. We are interested in how performance is affected by incremental changes, for example in a sprint release in an agile development project, and the purpose is to find performance issues, and improvement potential in the code. This can be done by comparing measures from the load test for the different sprints.
Perform Fair Comparisons
As for all types of comparisons, it is important to avoid comparing different things. A basic rule is that you should change as few parameters as possible when comparing two tests. This can be hard when testing an e-commerce site, where both content, layout, and functionality is constantly developed and updated, but to keep the number of variables as low as possible, there are a few things to have in mind.
First of all you should make sure that the user scenarios that are used are general enough to be used during a long period of time without needing to be changed. This means that you for example should avoid visiting pages that are connected to specific campaigns, or seasonal products, and therefor risking to be excluded from the product range. If you don’t want to rely on that a specific product is available, an alternative is letting the test script selecting a product randomly by the displayed ones, but you should have in mind that you need to handle the result with statistical methods to ensure that the randomness itself doesn’t affect the result.
If you have a few stable user scenarios, it is important to isolate the variable that you want to test. If we want to test how a new version of the code performs in comparison to previous versions, it might not be enough to compare to the test results from the previous sprint. The risk is that differences that are measured are connected to content related changes at the site. To avoid this, you need to make sure to run the tests you compare directly before and after a certain change is implemented.
Maximize the Information
In this article, we will not go into detail how to run the test, since this procedure in basically the same as a stress test, which is described in the second part of this series of articles. One practical difference is that you don’t need to load the system more than it can handle, you should instead put the load in line as what can be considered a normal amount of users. The test can therefor run during the day without affecting the visitors. It can run in a separate test system since we are not looking for absolute measures, but differences compared to previous tests.
Just as we have written in the previous parts of the series of article, the load test tool can normally give information about load times for the different pages that are visited during the test, and also detailed information about which type of content that is represented for the different parts. In some cases it is also offered to download diagnostic systems that supervise the server during the test, and provide information about for example CPU, memory and bandwidth usage.
If you want to collect even more information about what really takes time or eat resources during the test, the best way is often to build a tool for performance measures directly in the system that should be tested. At 3bits, we sometimes work with profiling to measure execution times to call certain functions, databases, and/or external resources. We also work with systems as Microsoft Application Insights, which makes is possible to collect and present more performance related data. We might have to come back to these tools in a separate article, but we can conclude that you are not limited to the measures that your load tool offer, but you can gather even more information about your system by making sure yourself to gather exactly the information that you are interested in.
With this we conclude this series of articles about load testing. If you have any questions or comments about the articles or tests of programs in general, you are more than welcome to get in touch.
