Web Application Performance Testing
In this blog, I wish to discuss on step by step guide to Performance Testing of a Web application, particularly that has a high amount of traffic on it.
Have you ever wondered how would you perform performance and load testing of a web application with say 10K or 30K users? Also, those virtual users should behave like regular users rather than robots. If you are looking for an answer to the above questions, then you are in the right place. If not, you would still be surprised by reading how we did it very easily!
Recently we had a customer who wanted to carry out web application performance testing with varying user load. Starting from 10K to 30K, uploading more than 100K videos (or ~16 TB) in an hour! This blog is all about how we did that.
Before we begin, I’m hoping that you have enough knowledge about Visual Studio Test Rig, if not, I encourage you to visit a blog written by my colleague – “Performance & Load Testing Using Visual Studio Test Rig.”
Now that you know “what is Test Rig” (Remember I’m counting on you ), let us first understand how to test rig works-
The above image shows the overall architecture of a test environment:
- The left part – Consists of the “Test Controller” & SQL Database, and is primarily responsible for governing your test. Microsoft Visual Studio will be used for executing a test or rather an application interface. You may install Visual Studio on the same machine where you have a different controller or on a different machine.
- The middle part – consists of “Agents” and is responsible for generating load on your system. So, a ‘Controller’ behaves as a master and sends commands to an ‘Agent(s)’ who behaves as a slave(s) and generates load on behalf.
- The right part – the web application which will be tested.
The left & middle parts together are called “The Test Rig”.
Coming to the main part of our discussion,
How did we achieve such a High Load Requirement?
Well, remember this thumb rule in the test rig, the more load you want to generate, the more resources (CPU/Memory/IO) a client (Agent) requires. Scaling a machine vertically has a limit but horizontally doesn’t. So, with an increase in load requirement, we can add more agents to our rig network i.e. more machines.
How many Agents do you Need?
Honestly, this depends on different requirements or rather a type of user personas and scenario you are executing:
For example, in my case there were 3 different scenarios (s):
- Test the web application – This requires quite a lot of RAM because the agent renders the HTML in memory and then also runs extraction rules (code).
- Test the APIs consumed by the Mobile App – This may not require as much of RAM as the Web application scenario does.
- Test the Storage services (Video upload services) – This will require more IO and throughput at the network level.
Now if you try to apply similar math to these different scenarios then it will fail!
We were using Azure Standard D4 VM for #1 & #2 whereas Azure Standard D15 VM for #3.
Ideally speaking, if you have a very normal (not complex) script to execute then 1 VM with 4 cores and 8GB RAM can generate 1K users.
My suggestion is to do a small round of load test first with 1VM (of a specific size) and then see what its capacity in terms of User Load is (when CPU reaches 85-90%). (Hint – You may want to try a step load pattern for this in the load test profile).
From there you are on simple math. Say a VM of a specific size handles 450 users and then if you want to go for a 5K user load then you will need 11-12 VMs.
Do you do this On-premise?
Yes, if it is a very small test and won’t require many resources.
No, if this is a large user load test then it’s better to go with some cloud provider (we went with Microsoft Azure, I really liked the idea of cloud which is scale as you need, pay as you use!).
Since we specifically used the cloud for this purpose, I’ll be talking about it more from here forward.
Remember I said earlier that with high load requirements, you do horizontal scaling of agents by adding more VMs…? But the louder question is what if this test will run multiple times over the period? (In fact, we were doing such web application performance testing for almost 6 months for a different part of an application with each release).
So that leads to the next question, “how one maintain those large numbers of VMs and save cost also?”. The obvious answer is to delete the resources when they are not in use. The next challenge is, if we delete resources then it takes time to spin up the entire environment… (almost 10-15 minutes for 1 VM) and that too manually! That’s a waste of time… right? So in the next section, I’ll explain, how we did that with automation scripts.
Pre-requisites:
- A bit of prior knowledge around VM & Azure Cloud (because I’m going to do it for Azure).
- Bit of knowledge about PowerShell
- Azure Command Line tools
How is it done:
- Setup the rig environment with one Controller VM and one Agent VM: While you set up these two VMs,
- Make sure they both have the same Username & Password. Remember, this is top most important and necessary. Otherwise, the controller won’t be able to communicate with the agent. The trust between a controller and an agent can be set up by adding both VMs in the same workgroup and that’s harder to set up in the cloud! A simple approach is to keep the same Username & Password for both VMs, isn’t it?
- Make sure that you add these VMs to a Virtual Network. This also means that you should create a Virtual Network first prior to creating any VM.
- Make sure that they both are connected, ideal they should be. But I suggest that you verify and make 100% sure that they are indeed connected and communicate with each other. This you can verify from VS.
- Now that you know, the Controller can communicate with the Agent, we can convert the Agent VM into a VM image. Later this image can be used for generating new VMs automatically. I suggest that if you have any static test data, keep them copied in the VM before moving ahead. Specifically, in our case, it was Video files (Large video files of around ~7GBs). If you don’t do this, your test execution will take time to copy these files onto each agent and that’s again a waste of time.
Follow these steps explained in “Create unmanaged VM image from Azure VM” for agent VM. Remember, once you mark your agent VM as generalized, they won’t be accessible, i.e. you cannot do RDP into that VM and will become useless.
- Hopefully, by now, you should have converted Agent VM into VM image (I’m sure you did) if so, let’s begin writing automation scripts for VM Image. See the script below image:
The script has:
- Configuration parameters
- A loop that decides how many VMs should be created
- A line of code in PS that creates a VM
- A line of code that restarts the VM. (We had to add this code because in our case, the Controller was not able to connect with the Agent right after creating the VM).
And that’s it. You run that PS script and that will create 10, 20, or whatever number of VMs you want.
Reference links:
Test Controller and Agent installers are available at https://docs.microsoft.com/en-us/visualstudio/test/lab-management/install-configure-test-agents?view=vs-2017
Management of Controller and Agent from VS is available at https://msdn.microsoft.com/en-us/library/dd695837.aspx