Putting an application under stress and comparing the results to previous tests can be useful to ensure that new features and additions do not have a negative impact. In addition, using load testing to find the breaking point can also help identify the week parts of the application early before they fail at a critical time in a production environment.

I have run many load tests before using JMeter on virtual machines and more recently explored K6. This month Azure Load Test became generally available so I thought it a good time to take look and see how we can run tests in the cloud.

In this blog post we’re going to take the URL shortener project that I previously built and perform a number of load tests on it. Will first take a baseline of the application running under dotnet 6, upgrade it to dotnet 7 and load test again before finally making a more complex JMeter script and seeing the impact of using that under Azure Load Test.

Getting Started with Quick Start

Azure load testing allows us to run simulated load through our application, report on it inside the Azure portal and compare the results to previous runs. Although it is based on JMeter scripts, to get started we can start with a simple web address.

Before we can begin, we first must select to create an Azure Load Test resource.

Once created we can get started with a quick test. This takes a single url rather than a full JMeter script and allows us to do a simple test quickly.

In the Url Shortner I have setup a redirect from https://goto42.dev/001-test back to the home page which I can use for the test.

The preview helpfully explains how much this test will cost in virtual user hours. Azure load test is charged for the Azure Load Test resource then a small cost per virtual user per hour, beyond an allowance that comes included with the resource.

This test will run for 2 minutes with 50 users after the first 30 seconds. As the test is running you can see the data in real time.

Once complete you can see a summary of the results. It is possible to also add in server infrastructure such as databases and storage to this view.

Behind the scenes this Quick test has used a JMeter script, which you can download to review if you wish.

Comparing Results

The previous load test has been saved upon creation so it can now be re-run at any time.

Now that we have our first baseline results I have upgraded the application to dotnet 7 triggering a deployment and then reran the load test. We can use the compare feature to compare this new run with the previous one.

Building a JMeter Script

Hitting a single URL is not a very substantial test, what ideally I would like to do is hit a range of URLs some once some multiple times in a repeatable way especially when I test out adding caching. To begin I have set up one thousand urls in the Url Shortner that point back to the home page, simply named 001-test through to 999-test.

Apache JMeter is an open source desktop Java application that we can use for both building JMeter test scripts an running load tests. For this more complex scenario will first build out our test in JMeter before saving the test file and uploading it to Azure load test.

To configure the test I could have used JMeters random number generator however this would mean that each test run would have a different set of URLs making it hard to compare between test runs. Therefore, instead I’ve decided to create a CSV file of URLs and drive the test from that.

The test is run inside a Thread Group that sets the number of virtual users and number of times to repeat the test run. In this test I have configured this with 100 users (so it reads 100 Urls form the CSV file) and to loop 10 times, so we can test the caching.

I have then added a CSV Data Set Config referencing a file with one line per Url. Before we save the file to be used in Azure we will need to modify the file location to be only the file name as explained on MS Learn.

Finally I added a Http Request that references the url variable provided by the CSV Data Set Config.

JMeter is very feature rich and there are lots of great resources to learn more. In this post I will quickly run over the elements used for this test. The full test file can be downloaded from GitHub if you’re interested and running it yourself.

Running the New Test

We first need to create a New test by Uploading a JMeter script. In the first screen we should upload both the JMeter test file (JSX) and our CSV file.

We don’t need any parameters or secrets for this run. You have the option to select engines to use. Engines are an abstraction over the infrastructure and there are limits to the number of virtual users per engine. For now we will keep to a single engine.

Once this is all setup the test can run and as before you can see the result. This time you will see the users at 100 as this is what had been set int eh JMeter script file.

Debugging Load Tests

Not everything goes to plan and JMeter in particular can be quite fiddly, a single mistake can just result in a test not running. To debug locally it’s worth adding in an Aggregate Report Listener to see what has been run. For more in depth look a View Results Tree Listener can be used to see diagnostics on each of the samplers.

When running with Azure Load Test after each run, you are able to download 3 different file types;

1. The input file is the JMeter file that has been used. You can download this and run it locally.

2. The Logs will show you diagnostic information generated while running.

3. The Results will show you the same CSV output of as and JMeter Aggregate Report Listener highlighting the https calls, status, response times and error rate.

Limitations

As with many Azure service it is not going to be suitable for every situation and a number of the limitations are outlined on the Microsoft learn website. A limitation worth noting if you already have an existing test suite is that the test can only take a maximum of three hours.

The cost is another factor. As with several Azure services there is a low initial cost then you pay by the hour, multiplied by the number of users you are simulating. This can start out relatively inexpensive, however large numbers of users over many hours can quickly add up. It is worth remembering that in some cases you may reach a point in which it would be cheaper to run a virtual machine with JMeter. If you did go this route you need to manage the machine and reporting yourself. For full details review the pricing page.

This is a new service and no doubt over time will be extended. There is a great Q&A that was held that shows some insights such as a possible plan to add K6 support.

Final Thoughts

Simulating load on your application is an important part of a development process and Azure load testing makes this easy to do with JMeter scripts.

Overall I have found it to be quick and easy to setup, it is missing a few features but they has an active road map and I look forward to seeing what comes next.

There are many other features we have not covered in this post. It is possible to configure Azure load testing in your CI/CD setup for regular testing, provide secrets via key vault along with using client certificates or private link. For a more in depth look I can recommend Matt Allford’s video and the Microsoft learn resources.