I’ve been writing a game using Unity, and I’ve just realised how rare it is to use floating point (decimal) numbers in my day job as a software developer. I remember my Computing teacher back in College stating how computers are rubbish at calculating decimal numbers, but I never really understood. Calculators work just fine. They may only have several decimal places but that is usually more decimal places than you need.

In game development, decimal numbers are frequently used for sizes and positions in a 3D space and therefore calculations with these objects involve lots of decimal maths.

After writing some logic to position my characters on the map, I tried writing a unit test. Unit testing this method could be beneficial when I add extra features later.

I wanted to compare a Vector which is composed of 3 floating point numbers. Since I already had written the logic and was happy with the character’s positioning, I just grabbed the output of the method and specified it as the expected output, but my test failed!

Expected: (-34.28, 0.00, 20.63)
But was:  (-34.28, 0.00, 20.63)

so then I inspected the Vectors more closely

"(-34.28, 0.00, 20.63)"
	magnitude: 40.00208
	normalized: "(-0.86, 0.00, 0.52)"
	sqrMagnitude: 1600.166
	x: -34.275
	y: 0
	z: 20.625
 
new Vector3(-34.28f, 0.00f, 20.63f)
"(-34.28, 0.00, 20.63)"
	magnitude: 40.00894
	normalized: "(-0.86, 0.00, 0.52)"
	sqrMagnitude: 1600.715
	x: -34.28
	y: 0
	z: 20.63

I think the numbers it shows you is a text representation which involves rounding the numbers to 2 decimal places. But if you look at the actual X, one is -34.275 and the other is -34.28. So there’s a 0.005 difference. A similar problem with Z.

What also threw me, is when I looked at the documentation, https://docs.unity3d.com/ScriptReference/Vector3.Equals.html  I saw this:

“Due to floating point inaccuracies, this might return false for vectors which are essentially (but not exactly) equal. Use the == operator to test two vectors for approximate equality.”

I was originally using Assert.Equals(expected, actual) so I switched to Assert.True(expected == actual) but it still wasn’t working.

I looked at the Unity source code and they are providing their own == operator for vectors that does some “square magnitude” calculation then compares if it is less than a certain number – a margin of error. 

this is Unity’s version

[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static bool operator ==(Vector3 lhs, Vector3 rhs)
{
   float num = lhs.x - rhs.x;
   float num2 = lhs.y - rhs.y;
   float num3 = lhs.z - rhs.z;
   float num4 = num * num + num2 * num2 + num3 * num3;
   return num4 < 9.99999944E-11f;
}

So I did something similar and increased the number. I came up with this, using Unity’s provided square magnitude function, but wasn’t sure if it made mathematical sense: It worked for my scenarios though

private static void AssertAreApproximatelyEqualDueToFloatingPointImprecision(Vector3 expected, Vector3 actual)
{
   var difference = Vector3.SqrMagnitude(actual - expected);
   Assert.IsTrue(difference <= 5.0E-04);

/*
eg
0.0004996415
<=
0.00050
*/
}

Looking back at this, I probably over complicated it. I suppose I should just change the expected values to have more decimal places because it’s the rounding to 2 decimal places that will be the actual problem in this case. In other cases with more decimal points, you will encounter issues due to the way computers do handle large decimal numbers which the Unity docs are alluding to.

In other situations, you also need some leeway in your calculations. If you want to move a character to a certain position, you may find that they could be 0.1 away from their destination, then on the next frame, they move their usual distance based on their movement speed – and then overshoot it. So you end up needing to write code like “if distanceRemaining < 0.2f then stop”

So floating point numbers, especially when it comes to game development, really does add extra complications to think about.