A while ago, I wrote a blog about the Merge Ready Checklist, which was a process we have to prove we have followed to be able to complete and release our software project.
The process was created by a some experienced former Software Testers, now basically Quality Assurance Managers.
As part of the checklist, they then insist on having Test Coverage of 80% which I always think is an unreasonable ask. When I joined Development, we had Visual Studio Enterprise licences which have a Test Coverage tool available. However, we have since downgraded to Professional. So I asked what Test Coverage tools we can use because it needs to be something that IT have approved to download, and that we have a licence for. We were told we could use AxoCover, but I found it wasn’t compatible with Visual Studio 2019 or above, which was an inconvenience.
Ok, so let’s run it and see what happens. Firstly, you are greeted with a phallic symbol.
Test execution started.
___
/ \
| |
\ /
/ \
/ \
___/ \___
/ \
| _______ |
\___/ \___/
AxoCover Test Runner Console
It’s supposed to look like their logo, which looks more like a fidget-spinner.
Here are the metrics it produces, before; and after.
| Coverage | Uncovered | Total | |
| Classes | 11.4% | 3040 | 3433 |
| Methods | 14.4% | 17784 | 20770 |
| Branches | 12.7% | 29085 | 33308 |
| Lines | 13.4% | 98844 | 114114 |
| Coverage | Uncovered | Total | |
| Classes | 11.9% | 3034 | 3443 |
| Methods | 13.3% | 17747 | 20473 |
| Branches | 12.0% | 29020 | 32985 |
| Lines | 13.4% | 98786 | 114118 |
Then I can’t even make sense of the statistics? Is that saying that I have removed methods (total has gone down!)? I have added a few classes with several methods each (these obviously contain lines of code and conditional statements so I expect all values to be higher (but despite more Classes, the number of methods and lines has decreased). I had also added some Unit Tests but maybe would have expected 30% on new code.
I asked Tech QA to explain the figures to me, and they were like “we dunno, we aren’t developers. We just look for the 80% number“. Then I point out that they were supposed to be judging the 80% coverage on NEW CODE only. This is for the entire solution file. So this doesn’t give them the evidence they want, and it’s not accurate either and cannot be trusted.
After running it several times and adding/removing code to see how the numbers changed, I then was suddenly low on disc space. Turns out Axo Cover reports are 252MB each! Yikes.
Testing AxoCover
Since the numbers were nonsensical. I decided to create a simple test project and run it on simple examples. Let’s see how it judges what is a line/branch/method/class.
namespace Axo
{
public class ExampleClass
{
}
}
0 method 0 branches 0 lines
So a class definition with no methods and actual lines of code results in zeroes all around. It must ignore boilerplate class code.
namespace Axo
{
public class ExampleClass
{
public ExampleClass()
{
}
}
}
1 method 1 branches 3 lines
So now I have a method but it is empty. Seems the idea of ignoring boilerplate code doesn’t apply to methods. It must count the method definition plus braces inside the method for the line count, but it doesn’t make sense to count braces since that’s just to group related code. 1 branch is weird too, that should be for IF statements which we will test soon.
namespace Axo
{
public class ExampleClass
{
public ExampleClass()
{
var a = 3;
var b = 4;
var result = a * b;
}
}
}
1 method 1 branches 6 lines
So now I have added 3 lines to the method. The line count has increased by 3 so it seems like it make sense.
namespace Axo
{
public class ExampleClass
{
public ExampleClass()
{
var a = 3;
var b = 4;
var result = a * b;
if (result > 10)
result = 0;
}
}
}
1 method 3 branches 8 lines
I’ve added 2 lines, where 1 is an If Statement. So now we have increased the branches but it has increased by 2. This must be the “implicit else” where the “result” is either greater than ten or it is less than 10, so there’s 2 paths. I’d still say that is 1 branch though.
namespace Axo
{
public class ExampleClass
{
public ExampleClass()
{
var a = 3;
var b = 4;
var result = a * b;
if (result > 10)
MethodA();
}
private void MethodA()
{
}
}
}
2 method 4 branches 10 lines
I’ve replaced one line with a method call to a new method. Method count increasing by 1 makes sense. Given the previous examples, adding a new method adds 1 to the branch count for some reason. In the empty method example, we got +3 to the line count, but now we only get +2, so that seems wrong. I don’t even think an empty method should increase the line count or the branch count, so the figures are becoming increasingly nonsensical.
namespace Axo
{
public class ExampleClass
{
public ExampleClass()
{
var a = 3;
var b = 4;
var result = a * b;
if (result > 10)
MethodA();
else
MethodB();
}
private void MethodB()
{
}
private void MethodA()
{
}
}
}
3 methods 5 branches 13 lines
So now instead of an implicit else, I’ve made it explicit, and created another Method. Method count makes sense. Branch count has increased by 1 which I think will be for the new method and not the else. We have +3 to the line count but should we have 2 for the else, then up to 3 for the new method.
namespace Axo
{
public class ExampleClass
{
public ExampleClass()
{
var a = 3;
var b = 4;
var result = a * b;
if (result > 10)
MethodA();
else
MethodB();
}
private void MethodB()
{
}
private void MethodA()
{
{ }
}
}
}
3 methods 5 branches 15 lines
I was intrigued if it really was including braces. Some random braces gives +2.
public class ExampleClass
{
public ExampleClass()
{
}
public ExampleClass(string test)
{
}
}
2 methods 2 branches 6 lines
I thought I’d reset and try again. So we have 2 methods, which as we have discovered means 2 branches with AxoCover’s metrics. It seems to count both methods as +3 lines.
namespace Axo
{
public class ExampleClass
{
public void ExampleClassMethodA()
{
}
public void ExampleClassMethodB(string test)
{
}
}
}
2 methods 2 branches 4 lines
Looking back through the examples, I wondered if it is actually counting an empty Constructor as +3, but an empty method is +2. So this example has actual methods rather than constructors, and it seems to confirm my theory.
Discussion
When it comes to counting metrics in code, I think there is some degree of subjectivity to it. What even is a line of code? You could technically add many “statements” and put them all on one physical line. So you could look at the line numbers and see 1 line, but when actually reading the line of code, you can read multiple instructions. The analogy could be that you expect a list to be one item per line but someone could write a list on paper as a comma-separated list on one line. One is more readable than the other but it’s still valid either way. If someone asked you how many items were on the list, you could count them either way and end up with the same number. Therefore, I think the line count should actually be “statement” count.
I do think AxoCover’s definition of a branch seems wrong, and what they interpret as lines seems inconsistent and a possible bug in their logic.
On a larger and more complex codebase, the statistics it produces seems really nonsensical. So I think I have proved this tool isn’t worth using, and we definitely shouldn’t be using it to gatekeep on our Merge Ready Checklist.
Stranger Coding Tings 