Make Microsoft Run Benchmark Tests for You
Benchmarking is important! Problem is running those tests can make your machine unusable. Let’s make Microsoft and GitHub feel that pain instead.
We’ll cover a few other details before we get to the Microsoft and Github fun and games. Firstly, what is benchmarking anyway? Then set up a simple benchmarking example that anyone can run and finally down to how to offload those tests.
For the TL;DR; crowd, this isn’t magic. We’re using GitHub Actions, a public repository and unlimited build minutes. Now that you know, why not read on for the rest of the journey.
Benchmarking
In software engineering terms, Benchmarking is the process of testing code during execution to determine how well it is performing. The most common metrics used to understand the performance of the code are time and memory. There are more, although these are probably the most important.
How long does the code take to execute? How much memory does a program consume while running the code? The longer the code takes to complete and the more memory it consumes, the worse the performance.
This is important when running your code in the cloud. Compute power and memory cost real money.
For software engineers benchmarking is commonly done by comparing two methods that achieve the same result. This helps the engineer understand which implementation offers the optimal performance for the application they are working on. When an engineer applies this knowledge to sections of code that are used and reused thousands of times, it can impact the performance of an application in a colossal way.
Benchmarking in .NET
For the purpose of this demonstration, we’ll use .NET and C# to execute some simple benchmark tests. We’ll use the BenchmarkDotNet library; it is the best way to run benchmark tests in .NET. It tracks the benchmark tests performance and turns your methods into reproducible experiments.
BenchmarkDotNet has tons of features that are essential in comprehensive performance investigations. Four aspects define the design of these features: simplicity, automation, reliability, and friendliness.
It is a robust tool and makes it simple and easy to set up a benchmark test in a .NET console app. For more in-depth information BenchmarkDotNet and its features, please review their documentation.
A Simple Benchmark Test
To set us off on our benchmarking journey, we will set up a simple app in .NET to test the fastest way to generate random numbers. We want to make this thread-safe, so we don’t break the random number generator and return all zeros.
The first step is to create a new console app in .NET. The CLI takes care of this for us via a simple one-line command. Thanks, Microsoft, that’s too easy.
dotnet new console --framework net6.0Next, we will create a new class to hold our generators. We have three different thread-safe random number generators all in one file. They will form the base for our benchmark tests. The sample repository has them separated as they probably should be.
Next another new class, Benchmark.cs. This will host our tests. This is where BenchmarkDotNet comes in to shine. The [MemoryDiagnoser] attribute ensures our output captures how much memory our tests consume. The [Params] attribute sets the value of the Iterations property. For the simple test, we have set it to 100000, another option would be including multiple values here to run more than one set of tests. Finally each method with the Benchmark attribute will be included as a test when we run the application. These benchmark tests will have three test methods, which means we will end up with three sets of results.
Finally, we will set up the Program.cs to run our benchmark tests. This is as simple as setting your Program file to match the following code.
This is a plain and simple setup to get a test ready to run.
Running the Benchmarks
Want to test this out on your machine? It’s easy. If you have the .NET SDK installed, just run the following command. Be warned, you will probably hear your computer’s fan kick into gear. If you want to kill your machine, update the parameters on that test to higher and higher numbers.
dotnet run -c Release --project ./BenchmarkApp.csproj
The sheer amount of output lets you know that you’ve done something right. Give it a few seconds then your fan will kick in. The fun part is you’ll find yourself waiting on the results. Watching the screen like somehow it’s going to make it go faster because it’s being eyeballed. This is lost time in your day. You could be doing better things with your compute power, and that’s why I’m proposing that you run your benchmark tests away from your machine.
At the bottom of the output you’ll find a results table, much like the example below. It indicates that the Shared method is clearly the better performer here.
| Method | Iterations | Mean | Allocated |
|------- |----------- |---------:|----------:|
| Locked | 100 | 7.495 us | 1,152 B |
| Shared | 100 | 2.235 us | 848 B |
| Thread | 100 | 9.223 us | 1,288 B |Running Benchmarks Remotely
In order to do this, we will use a public GitHub repository because GitHub Actions are easy to use, but they offer a fantastic bonus feature. If your repo is public, your build minutes are free!!! Theoretically you can do this on other providers but using GitHub is the only way to make Microsoft do it for free.
In order to do this, you need a GitHub Account and you need to Create a New Repository.
Make that repository public and then clone your new repository to your machine, drop the example code into your new folder and make your initial commit.
The last step is Creating a GitHub Action Workflow. Follow the instructions in the documentation, but it is as simple as navigating to Actions, clicking the New Workflow button, and select the set up a workflow yourself option just under the title.
Drop the following code into your action and commit it. That’s it! Once you’ve done this, your new Action will have kicked off in the background.
Now that we have an Action running, we can get on with our other tasks and check back on the results when we are ready. To see the results, let’s do the following steps.
- Navigate back to our Actions tab.
- Find our most recent workflow run and select it
- Select the build job
- Select the Benchmark step in the build job
- Scroll to the bottom, and you will see the results table
It should look like the following picture.
Finally
At this point, we have created a new .NET benchmark test and made Microsoft run it for us. The beauty of this method is that you are comparing all the test methods against each other on the same machine, so the performance metrics are all still relevant. If you want to test on a specific operating system and system configuration, GitHub allows you to configure Self Hosted Runners as well.
This method should be adaptable to other benchmarking tools. You will have to adapt the action YAML to run your test instead of a .NET application.
My machine no longer crashes and burns while running benchmark tests, and using this method, neither should yours. Hope you find this method helpful! If you do, share it around, no one should have to suffer an unusable machine because of benchmark tests!
Example Repository
If you want to see a real life example of the application detailed in this post, you can find it on GitHub. The action triggers new benchmark tests, but the code is a little more separated into files and projects. There’s a sample console app included too which runs the random number generators to validate that they are indeed thread-safe.
Connect or Support?If you like this, or want to checkout my other work, please connect with me on LinkedIn, Twitter or GitHub, and consider supporting me at Buy Me a Coffee.
