As we head towards the release of our project, we have a massive suite of manual test cases that need running. The managers have looked at the numbers and worked out the average we need to run per day. We haven’t been meeting the targets.

The managers have decided to try and incentivise us with more rewards, be it money, extra holiday, or treats. We just have to hit the daily target of test cases ran, with a new reward available each day.

In the meeting, our Lead Developer exclaimed “We will hit those targets within 10 minutes”. What he meant was, he was just going to mark the tests as run without actually running them. As explained in “Increasing Productivity #1“; “you get what you measure”: If all you want to see is the tests have ran, then we simply mark the test as ran.

In the blog, I also explained that each test takes a completely different amount of time to run, and some are more difficult to run. So the way I see it; if we carry on just running whatever test case is next, then we will fail to meet the target every day, since nothing has changed. Instead, we could spend time looking at them, sorting them out in levels of difficulty, then all running the easy tests. This means we will meet our target and then claim a reward. The next day we can just focus on the hard ones. Sure we will massively fall short of our target, but at least we bagged a reward the previous day. The net result over the two days will be the same as it was without the rewards. So it will look like this:

When I was discussing this with the Lead Developer, he said that: had he been the one running the test cases, he would have kicked off in the meeting because the system is insulting. The way he saw it, people are trying to do their jobs, and managers are stating it isn’t good enough, and are taking the approach that they are only motivated by money, and not by anything else.

When he put it that way, I realised he has a point there. It reminded me of A-Level Business Studies, when learning about Frederick Taylor’s ideas of motivation; stating that workers are purely motivated by money. It’s quite an archaic viewpoint.

The thing is, if the reason why we are behind is incompetence, then why are we rewarding incompetence, intead of dealing with that problem? If we are doing as expected, then why throw more money at it? I don’t really see any advantage of this idea. It seems weird to complain about something that benefits me; but I just don’t understand it.

In the early days of my Testing career, I found that when Test Managers looked at their metrics on how the project is progressing, and thought it appeared we weren’t on target; they would just throw more staff at it.

This seems a simple and obvious solution to the problem, but often they only assigned this extra person for a small amount of time, sometimes even just 1 day. So once this new person has been assigned to your project, you end up giving them a run-down of the project; explaining what the project aimed to achieve, some known bugs (so they don’t waste time investigating and relogging known issues), and then show them where the tests are to run.

So you may have taken an hour to do your introduction presentation, then they have bugged you several times throughout the day, causing time loss, and you lose your train of thought; slowing your progress down further. So at the end of the day, there may have been a net gain in productivity for the team, but it wasn’t twice the productivity that the Test Manager planned. It ended up being more like two people running at 60% effort. But then sometimes they ask you why you weren’t productive. “Erm, it was your fault for making me train someone up”.

Some days I’ve even gone through the who introduction to the project, only for the Test Manager to change their mind and take them back to their original project. All that does is increase stress for your team as you are closer to the deadline with nothing to show for your efforts that day.

Luckily, things have changed since then, and the team structure is more set in stone. Instead if the testers want help, the developers will chip in with running test cases. This means that they are already familiar with the project and functionality, and you can have them running tests as long as required.

When I worked as a Tester, we didn’t have any kind of monitoring on our efforts; we were just left to get on with things. This meant some people were productive and did their work because that’s what they were paid to do, and others abused the system and did a bare minimum. Then one day my Test Manager came up with this idea that everyone needs to run 10 Test Cases a day in order for the team to be productive.

How long is a test case though? If each one took 40 minutes to run, then 10 would keep you busy for the day. But some can take that long, and others can take only a minute – so the metric is rather meaningless. Also, some tests often found bugs because they tested brittle areas of the system, whereas others never uncovered bugs. Some tests were fine to run with a few clicks, whereas others depended on certain configurations being set up.

The most annoying configurations relied on third-parties, so sometimes you would have to log a ticket with the Environments Team, then they would contact the third party and wait for a response. This could delay you for days.

So with a demand of “you need to run 10 tests a day”, how do people respond? Well, most people then started to cherry pick the simple tests. Tests that rely on third parties got pushed to one side, as did the tests that often find bugs (logging a bug means you need to spend time investigating, filling out the bug reports, then chasing up the appropriate developer). After speeding through 10 easy tests, people just spend the entire day chilling out. Some people ran 20-30 tests, then tried to boast to the mangers saying “look how good I am”.

People like me that actually cared about getting the release out on time would pick up the important tests. I’d be logging tickets with the Environments team, ensuring we had the correct configuration. I would be running the brittle tests, logging bugs and liasing with the developers to get the fixes. I was the one getting called into meetings, asked on why I was only running 5 tests and failing to meet my targets.

The closer to the deadline you leave things, the more chance the release will have to be delayed. Plus it’s not great telling a developer there is a bug, but they only have a day to get it sorted. It’s not great telling the Environments team they have to drop what they are doing to set up a Test Environment because you left an important test to the last minute only to find the Third-Party dependency isn’t working.

It’s almost like people forget what it is like being a Tester when they progress to management level. As my Lead Developer always says: “you get what you measure”.