Creating a Micro-Benchmark Framework

When we are trying to make a particular piece of code run faster because we see that it is
taking more time compared to everything else in our application, we will need to
compare different implementations of completing the same task. This is where microbenchmark
testing can help.
Micro-benchmark testing allows us to take a close look at how fast small bits of code are
performing. There are a couple of items to keep in mind as we use micro-benchmark
testing, though. The first is that we cannot exactly determine best practices from a microbenchmark
test because it is such an isolated case. The second is that although a piece of
code might perform faster in a micro-benchmark test, it might very well take up a lot
more memory and therefore cause the garbage collector to collect its garbage more often.
Optimization Suggestions 43
3
The point here is that although micro-benchmark testing is good and we should do it
(which is why we are going to build a framework for it) we need to be careful of any
assumptions we make solely on what we find out from our micro-benchmark tests.
XNA Game Studio Express not only lets us create game projects, but also allows us to
create library projects. When the library projects are compiled they can be used by other
applications—a game, a Windows form, or even a console application.
We are going to create another application, but this time it is going to be a normal
Windows Game Library project. This class should be called CheckPerformance. We will be
utilizing this library from a console application. The code for the CheckPerformance can
be found in Listing 3.1. The purpose of this class is to create methods that perform the
same tasks different ways. We will then create a console application that calls the different
methods and measure the amount of time it takes to process each method.
LISTING 3.1 The CheckPerformance class has methods that produce the same results
through different means.
public class CheckPerformance
{
private Vector3 cameraReference = new Vector3(0, 0, -1.0f);
private Vector3 cameraPosition = new Vector3(0, 0, 3.0f);
private Vector3 cameraTarget = Vector3.Zero;
private Vector3 vectorUp = Vector3.Up;
private Matrix projection;
private Matrix view;
private float cameraYaw = 0.0f;
public CheckPerformance() { }
public void TransformVectorByValue()
{
Matrix rotationMatrix = Matrix.CreateRotationY(
MathHelper.ToRadians(45.0f));
// Create a vector pointing the direction the camera is facing.
Vector3 transformedReference = Vector3.Transform(cameraReference,
rotationMatrix);
// Calculate the position the camera is looking at.
cameraTarget = cameraPosition + transformedReference;
}
public void TransformVectorByReference()
{
Matrix rotationMatrix = Matrix.CreateRotationY(
MathHelper.ToRadians(45.0f));
// Create a vector pointing the direction the camera is facing.
Vector3 transformedReference;
CHAPTER 3 44 Performance Considerations
Vector3.Transform(ref cameraReference, ref rotationMatrix,
out transformedReference);
// Calculate the position the camera is looking at.
Vector3.Add(ref cameraPosition, ref transformedReference,
out cameraTarget);
}
public void TransformVectorByReferenceAndOut()
{
Matrix rotationMatrix = Matrix.CreateRotationY(
MathHelper.ToRadians(45.0f));
// Create a vector pointing the direction the camera is facing.
Vector3 transformedReference;
Vector3.Transform(ref cameraReference, ref rotationMatrix,
out transformedReference);
// Calculate the position the camera is looking at.
Vector3.Add(ref cameraPosition, ref transformedReference,
out cameraTarget);
}
public void TransformVectorByReferenceAndOutVectorAdd()
{
Matrix rotationMatrix;
Matrix.CreateRotationY(MathHelper.ToRadians(45.0f),
out rotationMatrix);
// Create a vector pointing the direction the camera is facing.
Vector3 transformedReference;
Vector3.Transform(ref cameraReference, ref rotationMatrix,
out transformedReference);
// Calculate the position the camera is looking at.
Vector3.Add(ref cameraPosition, ref transformedReference,
out cameraTarget);
}
public void InitializeTransformWithCalculation()
{
float aspectRatio = (float)640 / (float)480;
projection = Matrix.CreatePerspectiveFieldOfView(
MathHelper.ToRadians(45.0f), aspectRatio, 0.0001f, 1000.0f);
view = Matrix.CreateLookAt(cameraPosition, cameraTarget, Vector3.Up);
}
public void InitializeTransformWithConstant()
{
Optimization Suggestions 45
3
LISTING 3.1 Continued
float aspectRatio = (float)640 / (float)480;
projection = Matrix.CreatePerspectiveFieldOfView(
MathHelper.PiOver4, aspectRatio, 0.0001f, 1000.0f);
view = Matrix.CreateLookAt(cameraPosition, cameraTarget, Vector3.Up);
}
public void InitializeTransformWithDivision()
{
float aspectRatio = (float)640 / (float)480;
projection = Matrix.CreatePerspectiveFieldOfView(
MathHelper.Pi / 4, aspectRatio, 0.0001f, 1000.0f);
view = Matrix.CreateLookAt(cameraPosition, cameraTarget, Vector3.Up);
}
public void InitializeTransformWithConstantReferenceOut()
{
float aspectRatio = (float)640 / (float)480;
Matrix.CreatePerspectiveFieldOfView(
MathHelper.ToRadians(45.0f), aspectRatio, 0.0001f, 1000.0f,
out projection);
Matrix.CreateLookAt(
ref cameraPosition, ref cameraTarget, ref vectorUp, out view);
}
public void InitializeTransformWithPreDeterminedAspectRatio()
{
Matrix.CreatePerspectiveFieldOfView(
MathHelper.ToRadians(45.0f), 1.33333f, 0.0001f, 1000.0f,
out projection);
Matrix.CreateLookAt(
ref cameraPosition, ref cameraTarget, ref vectorUp, out view);
}
public void CreateCameraReferenceWithProperty()
{
Vector3 cameraReference = Vector3.Forward;
Matrix rotationMatrix;
Matrix.CreateRotationY(
MathHelper.ToRadians(45.0f), out rotationMatrix);
// Create a vector pointing the direction the camera is facing.
Vector3 transformedReference;
Vector3.Transform(ref cameraReference, ref rotationMatrix,
out transformedReference);
// Calculate the position the camera is looking at.
CHAPTER 3 46 Performance Considerations
LISTING 3.1 Continued
cameraTarget = cameraPosition + transformedReference;
}
public void CreateCameraReferenceWithValue()
{
Vector3 cameraReference = new Vector3(0, 0, -1.0f);
Matrix rotationMatrix;
Matrix.CreateRotationY(
MathHelper.ToRadians(45.0f), out rotationMatrix);
// Create a vector pointing the direction the camera is facing.
Vector3 transformedReference;
Vector3.Transform(ref cameraReference, ref rotationMatrix,
out transformedReference);
// Calculate the position the camera is looking at.
cameraTarget = cameraPosition + transformedReference;
}
public void RotateWithoutMod()
{
cameraYaw += 2.0f;
if (cameraYaw > 360)
cameraYaw -= 360;
if (cameraYaw < 0)
cameraYaw += 360;
float tmp = cameraYaw;
}
public void RotateWithMod()
{
cameraYaw += 2.0f;
cameraYaw %= 30;
float tmp = cameraYaw;
}
public void RotateElseIf()
{
cameraYaw += 2.0f;
if (cameraYaw > 360)
cameraYaw -= 360;
Optimization Suggestions 47
3
LISTING 3.1 Continued
else if (cameraYaw < 0)
cameraYaw += 360;
float tmp = cameraYaw;
}
}
We do not need to be concerned with the actual contents of the different methods. The
main concept we need to understand at this point is we have different groups of methods
that do the same task but are executed in different ways. We discuss the details of the
Matrix in Chapter 4, “Creating 3D Objects.” For now we can take a look at the last three
methods in the listing and see they are all doing the same thing. All three methods are
adding 2 to a variable cameraYaw and then making sure that the value is between 0 and
360. The idea is that this code would be inside of the game loop reading input from a
device and updating the cameraYaw variable appropriately.
Now we can create the console application that will actually call that class. We need to
add a new Console Application project to the solution and can call this project
XNAPerfStarter. We can name this project XNAPerformanceChecker. The code for
Program.cs is given in Listing 3.2.
LISTING 3.2 The program measures the amount of time it takes to execute the different
CheckPerformance methods.
class Program
{
static int timesToLoop = 10000;
static void Main(string[] args)
{
while (true)
{
XNAPerformanceChecker.CheckPerformance cp =
new XNAPerformanceChecker.CheckPerformance();
Stopwatch sw = new Stopwatch();
//Call all methods once for any JIT-ing that needs to be done
sw.Start();
cp.InitializeTransformWithCalculation();
cp.InitializeTransformWithConstant();
cp.InitializeTransformWithDivision();
cp.InitializeTransformWithConstantReferenceOut();
cp.TransformVectorByReference();
cp.TransformVectorByValue();
CHAPTER 3 48 Performance Considerations
LISTING 3.1 Continued
cp.TransformVectorByReferenceAndOut();
cp.TransformVectorByReferenceAndOutVectorAdd();
cp.CreateCameraReferenceWithProperty();
cp.CreateCameraReferenceWithValue();
sw.Stop();
sw.Reset();
int i;
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.InitializeTransformWithCalculation();
sw.Stop();
PrintPerformance(“ Calculation”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.InitializeTransformWithConstant();
sw.Stop();
PrintPerformance(“ Constant”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.InitializeTransformWithDivision();
sw.Stop();
PrintPerformance(“ Division”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.InitializeTransformWithConstantReferenceOut();
sw.Stop();
PrintPerformance(“ConstantReferenceOut”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.InitializeTransformWithPreDeterminedAspectRatio();
sw.Stop();
Optimization Suggestions 49
3
LISTING 3.2 Continued
PrintPerformance(“ AspectRatio”, ref sw);
sw.Reset();
Console.WriteLine();
Console.WriteLine(“——————————————————————-”);
Console.WriteLine();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.TransformVectorByReference();
sw.Stop();
PrintPerformance(“ Reference”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.TransformVectorByValue();
sw.Stop();
PrintPerformance(“ Value”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.TransformVectorByReferenceAndOut();
sw.Stop();
PrintPerformance(“ReferenceAndOut”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.TransformVectorByReferenceAndOutVectorAdd();
sw.Stop();
PrintPerformance(“RefOutVectorAdd”, ref sw);
sw.Reset();
Console.WriteLine();
Console.WriteLine(“——————————————————————-”);
Console.WriteLine();
sw.Start();
CHAPTER 3 50 Performance Considerations
LISTING 3.2 Continued
for (i = 0; i < timesToLoop; i++)
cp.CreateCameraReferenceWithProperty();
sw.Stop();
PrintPerformance(“Property”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.CreateCameraReferenceWithValue();
sw.Stop();
PrintPerformance(“ Value”, ref sw);
sw.Reset();
Console.WriteLine();
Console.WriteLine(“——————————————————————-”);
Console.WriteLine();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.RotateWithMod();
sw.Stop();
PrintPerformance(“ RotateWithMod”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.RotateWithoutMod();
sw.Stop();
PrintPerformance(“RotateWithoutMod”, ref sw);
sw.Reset();
sw.Start();
for (i = 0; i < timesToLoop; i++)
cp.RotateElseIf();
sw.Stop();
PrintPerformance(“ RotateElseIf”, ref sw);
sw.Reset();
string command = Console.ReadLine();
Optimization Suggestions 51
3
LISTING 3.2 Continued
if (command.ToUpper().StartsWith(“E”) ||
command.ToUpper().StartsWith(“Q”))
break;
}
}
static void PrintPerformance(string label, ref Stopwatch sw)
{
Console.WriteLine(label + “ – Avg: “ +
((float)((float)(sw.Elapsed.Ticks * 100) /
(float)timesToLoop)).ToString(“F”) +
“ Total: “ + sw.Elapsed.TotalMilliseconds.ToString());
}
}
We need to also add the following using clause to the top of our Program.cs file:
using System.Diagnostics;
The System.Diagnostics class gives us access to the Stopwatch we are using to keep track
of the time it takes to process the different methods. After starting the timer we then loop
100,000 times and call a method in the CheckPerformance class we created earlier. Once
the loop finishes executing the method the specified number of times, we stop the stopwatch
and print out our results to the console. When using the Stopwatch object we need
to make sure that if we want to start another test that we first call the Reset method. This
isn’t built into the Stop method in case we just want to pause the timer and start it back
up for some reason. We could also use the static StartNew method instead of the instance
Start method. The StartNew method effectively resets the timer as it returns a new
instance of the Stopwatch.
When we are trying to measure performance on pieces of code that perform very fast
(even inside of a large loop) it is important to be able to track exactly how much time it
took, even down to the nanosecond level.
Typically timing things in seconds does not give us the granularity we need to see how
long something is really taking, so the next unit of time we can measure against is
milliseconds. There are 1,000 milliseconds in a second. Next comes the microsecond, and
there are 1,000 microseconds in a millisecond. Next is the tick, which is what the
TimeSpan object uses. There are 10 ticks in a microsecond. Finally, we come to the
nanosecond. There are 100 nanoseconds in each tick. A nanosecond is 1/1,000,000,000
(one billionth) of a second. Table 3.1 shows the relationships between the different
measurements of time.
CHAPTER 3 52 Performance Considerations
LISTING 3.2 Continued
TABLE 3.1 Time Measurement Relationships
Nanoseconds Ticks Microseconds Milliseconds Seconds
100 1 0.1 0.0001 0.0000001
10,000 100 10.0 0.0100 0.0000100
100,000 1,000 100.0 0.1000 0.0001000
1,000,000 10,000 1,000.0 1.0000 0.0010000
10,000,000 100,000 10,000.0 10.0000 0.0100000
100,000,000 1,000,000 100,000.0 100.0000 0.1000000
1,000,000,000 10,000,000 1,000,000.0 1,000.0000 1.0000000
PrintPerformance, our method to print out the performance measurements, takes in the
label that describes what we are measuring along with a reference to Stopwatch object.
TimeSpan is the type the .NET Framework uses to measure time. The smallest unit of time
measurement this type allows is a tick.
Because we want to measure our time in nanoseconds, we multiply the number of elapsed
ticks by 100. Then we take the total number of nanoseconds and divide by the number of
times we executed the method. Finally, we display the average number of nanoseconds
along with total number of milliseconds the task took to execute.
When we print out our measurements we want the average time it took to execute a
method for the number of iterations we told it to loop. The reason we do this instead of
just running it once is so that we can easily determine a more accurate number. In fact,
we even run an outer loop (that we exit out of by entering text that starts with “E” or
“Q”) to account for anomalies in performance on the machine in general.
An additional item to note about this code is that we call each method of the
CheckPerformance object once before we start measuring execution times. The reason we
do this is so the compiler can do any just-in-time compiling for the methods we will be
calling so we are not taking that time into account.