Colin arranged a meeting with a few Senior Developers (including me) and a Senior Tester. He stated that we are going to do a big recruitment drive, specifically targeting Graduates in India.
He said he is disappointed with the current Indian staff in general, since many seem low skilled and poor communicators, so he wants us to come up with:
- Multiple choice logic questions for a timed test
- Multiple choice programming questions for a timed test
- Verbal logic questions for an interview
- Verbal programming questions for an interview
So we are testing them on both written, and verbal comprehension as well as programming. Fine plan so far.
In the next meeting, a few of the Seniors had come up with some questions which they said they got from some websites. I had spent a few hours scouring the internet for ideas but I had found many websites with poorly worded questions, ambiguous questions, or material that I just thought was irrelevant to the job (either not C#/objected oriented/not graduate level). So I only submitted a few questions but I needed to confirm what our aim was…
I noticed many of the suggested questions were specifically programming questions on C# which is what the job is for, but do these graduates we are targeting actually know C#? I ask the question and the Indian seniors say that these days, university courses are mainly Java and Python. So I tell them the test they have come up with isn’t suitable. For multiple choice, the questions have to be just logical questions using pseudocode. For the interview, it needs to be phrased like “using a language of your choice” and we would have to allow them to use Java, Python or anything else.
I also found that they had many questions that were wrong, ambiguous, or irrelevant. I don’t think they used the same bad websites I found, but there’s some terrible questions out there for sure. I think it says something about how bad our staff are when they are meant to be Senior and they are taking questions from online and not really thinking about if they can answer them or not. We are going for Graduates as well. Really the hierarchy would go Graduate/Junior->Developer->Senior. So surely the questions have to be easy for Graduates to do.
“Why are people who don’t know what they’re doing, setting these questions?”
Colleague
“so we can recruit more people that don’t know what they are doing”
Me
Another problem was that we scraped about 50 questions of questionable quality, then Colin says: “I want 60 questions for them to do in 60 minutes…also we need a second set of 60 questions just in case someone leaks the answers to other applicants”.
Everyone seemed fine with this, but I pointed out that 60 questions were a struggle to come up with, and I didn’t think it was possible to answer them in 60 minutes – I said 45 minutes is better. 60 minutes sounds like pure tedium to me. The 3 other Seniors and Colin were all adamant that “60 questions in 60 minutes” was standard in the industry, and a couple of Seniors said they did tests like this in recent years and it was fine.
When we eventually came up with a test, Colin says
“I tried the test over the weekend, I do not think we can do the test in 60 minutes”.
Colin
Wow, I was correct, and was shot down 4 to 1 votes.
He wouldn’t specify what score he got but was insistent that he passed despite not answering all the questions.
I am never a fan of doing programming questions in an interview situation (unless it involves using a computer). Sometimes companies give you pen and paper, (or a whiteboard), and you have to write code. It’s unnatural and very awkward. Also, they come out with cliché problems like sorting algorithms, or data structures. Sometimes you can blag your way through these scenarios if you practice them beforehand. I prefer a take-home-test to build a small application in a few days. I’m conscious that this can be a lot of work especially if you are applying to other jobs at the same time. An alternative could be to show a program you have previously written and do a presentation on it to demonstrate your understanding and talk through the design decisions.
We already had a take-home-test from previous recruitment, so I suggested using that. The Indian Seniors said this would be far too hard especially if it involved calling a Web API. I said the candidates don’t even have to complete it (fully working app with many features), you just want to see how far they get and see how they structure their code and how they break-down the problem. If you just tell them to write one method like a sorting algorithm, then you don’t really get to see how they structure code because it’s minimal, isolated code.
The Indians even suggested a compromise of giving a choice of programs to write, but to give extra marks if you choose the first example. It didn’t seem that fair to me and there didn’t seem any noticeable increase in difficulty. One suggestion was even to get HR to call them and ask them on the phone to decide which test they want. How would that call even go?
HR: "Are you familiar with Web APIs?" Candidate: "no" HR: "do you want to do a banking app with a database?" Candidate: "no" HR: "you have failed to get the job"
or
HR: "Which test do you want to do? one with a Web API, or a Banking Application with a database? You get more marks for the Web API" Candidate: "erm…Web API then."
Eventually, they relented and agreed just to ask for a specific program and we would see how far they got. Next up was to decide how much time they actually got. I suggested giving them up to 10 days. One Indian Senior said 10 days was nowhere near enough time given the difficulty of the program. Another Indian Senior (who had previously stated this test was far too hard), then claimed he had a similar test that he successfully completed in 3 hours. I was like “wut“. How can he go from saying it was far too hard, to then saying it is easy and he can do it in 3 hours! He then suggested we should make them create and interact with a database which I think makes it way harder! I challenged him on this and he said he has never seen a university syllabus that has interacted with an API (which my suggested test required)… Despite him showing me a university that used ASP.net which is all about web apps and web services.
I did feel bad criticising their decisions when they had put some effort in, while I did think about the situation but didn’t have any good questions to ask. However, I didn’t think they had put much thought into the problem they were trying to solve.
Some lessons learned:
- You need to decide what level of experience you want to aim for so you can set relevant questions e.g. Graduate, Junior, Senior, Expert.
- You need to know what kind of programming languages your applicants will be using. If they are from a Java background but your job is C#, you may have to set the test in Java, but make it clear the language of the job is C#
- Make sure the questions make sense, aren’t ambiguous, and the answers are actually correct (this will be discussed further in an upcoming blog)