Dealing with Software Support

I was invited to a call with some of our users who wanted to explain some of the problems they had in a particular feature of our software. One of the problems was actually an issue with a massively-popular 3rd-party software we integrate with, so they wanted me to log a bug with the 3rd-party on their behalf.

A reasonable request I thought, but I really didn’t want to have loads of phone calls/email chains back-and-forth with support. I just wanted to log it and forget about it. Luckily, I am a developer so I have a good understanding of all the information Support/Development would want.

This company that I logged the bug report with must be one of the Top 10 software companies in the world. So you’d expect their Support to be of a high standard. Due to the fact that the bug report is still open at the time of writing, I’d better keep their identity a secret, otherwise they could easily look up my name from the bug report and my anonymity is blown.

The feature is that you click a button, then their software will open a dialog with some data pre-filled in. If their software is currently closed, it still opens the dialog (their software is just temporarily open). In this scenario, when you save/close, then their software is supposed to close back down completely. However, it was getting stuck and you had to use Task Manager to kill it. This problem can be recreated consistently every time.

So I put together a great description of the problem, the recreation steps, the version of Windows, the exact version of their software I was using. I told them our users have recently upgraded from ‘X’ edition to ‘Z’ edition; and I was using ‘Y’ edition and can now see the problem after installing ‘Z’ edition so the problem is definitely in this new edition. Since I didn’t want to provide them with our software to recreate the issue, I even found some of their software with a similar feature and recreated it on there too. But to make sure they definitely didn’t consider it to be a problem with our software, or their other software product, I found another 3rd-party, free-to-use software and recreated it there too.

So I’d say they have a 100% chance of recreating that themselves, and I had proven it was a problem with their latest iteration of software.

Within an hour, I get an email asking questions. So it seems I failed to provide them the information required:

What version of Windows are you using?
What exact version of our software are you using
When was the feature last seen working?

I had literally given him all those answers. So I reworded my original report and put them in bullet points. He then mails back saying he cannot recreate it and would like a demo. Unbelievable. I can recreate it everytime, our users can recreate it every time. It’s not a Windows issue because I was using Windows 8 and our complaining users are using Windows 10.

He specifies that the demo must be on Windows 10. Why? Is he just trying to mess me about? You can’t expect someone to have access to various computers. The version of Windows is irrelevant.

Regardless, I accept and I set up a Virtual Machine. It’s like a fresh computer install so maybe it is better than my work computer to prove this is a bug. I put their software on it, even created a brand new account. However, now my user doesn’t have the Premium licence but I accept the trial, so had 14 days. I can recreate it every time. Surely he hasn’t tried hard.

So I arrange the call, share my screen and I demo the feature to him. I ask if he did the exact same thing. He then tells me the main difference was that he was using the very latest version. Is he trying to wind me up? I told him the exact version, he asked for the version, then didn’t use it! Surely if he couldn’t recreate it on the latest version, then he could have checked on the version I was using; then rejected the bug as fixed. Instead, I waste an hour configuring the Virtual Machine and another 30 mins demoing it to him.

He was supposed to send me instructions on how to update because it wasn’t as simple as clicking a button. However, he didn’t bother and I found the instructions after some Googling. He was right that it was fixed…or kind of. I tested it several times and I saw the same issue again. So I tried some more and it worked for maybe 30 times then failed. So it’s intermittent. So I tell him it’s not perfect. He wants another demo because he can’t recreate it.

So I demo the problem and he said he cannot help because he noticed I’m “not using the Premium account” and he is in the Premium support team. I told him I do have a licence, I was just using this account to rule out if it was a problem if you had a large user profile: basically doing his investigation for him.

However, I have to deploy a new Virtual Machine since the settings don’t allow me to sign in (seems to be some restriction from our IT department). So after setting up yet another Virtual Machine and signing in with my work account (with the Premium licence), I do another demo. However, the intermittent nature means the issue didn’t happen and the meeting had already lasted 20 mins. I was pretty bored clicking a button and closing a dialog for that length of time. So I said if I had time, I’ll try and record it happening, but in the meantime, I’ll send him the recordings I had from before (with the Standard account).

After a few days, he says that he is going to log a ticket with the Microsoft Windows 8 Support Team, so I ask him why. It doesn’t make sense when the issue occurs on both Windows 8 and Windows 10. He said it was because it only happens on Windows 8 since I had demoed it working on Windows 10. I told him again that I have recreated the issue with the latest software on both Windows 8 and 10, and using 2 different user accounts. The recordings I sent him were Windows 10. It was just that the issue was intermittent and coincidentally worked during the time I was demoing it.

He apologises for the misunderstanding and would need some time with his ‘senior’ to come up with the way forward. A few days later, he says that his senior wants me to arrange another call to test out some scenarios. I told him I didn’t understand what the plan was. He apologises for the misunderstanding, but there’s some scenarios that we need to test. I told him I didn’t understand what the plan was: am I doing a demo? or am I watching them do a demo? He apologises for the misunderstanding, but he wants me to deploy another Virtual Machine and create a new user and demo this to them. Surely, I needed to know that and set up a Virtual Machine and new user account before we start the call. It would have been a bit awkward if I set up the call, he asks me to demo it on a machine I wasn’t logged into and with a user I hadn’t even created.

I told him I didn’t understand what the plan was. Why do I need yet another Virtual Machine and user account when I recreated the issue already using 2 accounts, 1 Windows 8 physical machine and 2 Windows 10 Virtual Machines? He said they had attempted to recreate the issue with multiple users and on different Virtual Machines, but since the problem seems to exist on my Standard account, then he would like me to verify it on another Standard account.

Let’s recap: when he concluded it was a Windows 8 issue, I told him it wasn’t because I’d recreated it on Windows 10. It seemed that me demoing the feature working on Windows 10 was the thing that he picked up on, and completely disregarded everything I wrote and said. I had told him on that call it was intermittent and I had seen it fail that same day on the same Virtual Machine. Now we are in a situation where I had told him I had recreated it with multiple users, but he is still thinking of that demo I did where it was working. Not only was it Windows 10, but I was using my Premium account at the time. So now he has this idea that the problem is with my Standard account.

I think I need to take him to a hypnotist to train him to forget the existence of that demo. What’s he going to claim next? That because he saw it working on a Friday, then it’s a Monday to Thursday issue?

My dream of just logging the issue and them instantly confirming the bug was completely ruined.

It’s dragged out for a month or something daft and I feel like I have invested weeks of time investigating it. This is literally his job and I am doing his work and telling him how to do his job. It’s ridiculous.

I keep saying to my colleagues that it seems that he is trained to mess people about so they drop the issue then they don’t have to fix anything. It’s a bizarre strategy, but what other logical explanation is there for his behaviour?

I like our system: Regardless of whether Support can or cannot recreate the issue, the bug gets prioritised based on what they know. If it is deemed high priority then it will get sent to Development, otherwise probably gets thrown on the backlog and we might look at it in a year… or never. But I think the key aspect is that we always believe our users, so it does get logged (“the customer is always right”). Well, I guess we get evidence like a video or screenshot, but I had already provided those – and we would never dismiss video evidence.

As a side note: I wish that low priority bugs were dealt with faster because it will discourage users from reporting them if they think we will ignore it for a year. Maybe they can be given to Junior Developers to clear down.

Major Incident With Comms

that awkward moment when there is a Major Incident, but there is also an Major Incident with the Comms system

When there is a Major Incident with our software, we send out emails to the affected customers. However, our Mailing provider, Mail Chimp had blocked our account without warning – apparently we were now violating their terms of service.

Since we couldn’t send out these emails regarding a Major Incident, we then declared a Major Incident.

Software Developer Wages

In my recent Promotion Special blog, I discussed how I was underpaid by my employer and I am going to end up with £12k extra for doing exactly the same job as before.

When I have looked at Job Adverts for developer roles of my level of experience, it seems I could possibly get an extra £5k-10k if I went elsewhere (~£50k). However most of these adverts seem to imply I would need to learn a few more programming languages and could have extra responsibilities like line management, or dealing with Senior Managers, possibly even relocate. It’s not worth it.

My first job out of uni was £18k and that was a hybrid role of Tech Support and Junior Software Engineer – I additionally spent a fortune in travel costs (I think it was like £80 per week). Last year, we hired loads of Apprentices so I thought they could be on about £18k, but when I stated that to my Apprentice, he said it was £23k. I couldn’t believe it. That was my salary when I was about 25 and I had to work hard for that. These youths have it easy these days.

The thing is, we got these Apprentices via a Bootcamp company so we pay the Bootcamp a premium too for commission. It’s bizarre, because we are well known to be cheap, so why are we paying a 3rd party, and giving a high salary to unskilled developers. Mental.

Quite often when you watch Youtube videos or read blogs, Americans often come up with salaries like $100k+. I think there was one video that said a Google salary was $230k – mental). I guess you need to take into account what a dollar means over there – ie cost of living; but these figures do sound crazy. You can look at jobs in London, England and you can easily see £60k-100k, or sometimes even more if you know an obscure language where the “supply” of developers is limited so you can demand a premium.

These wages can be misleading though. I have a friend that is a Senior Software Developer and his partner is a Lawyer. I went to stay at their house in Bermondsey, central London, and expected he was living the high-life, but he lived in a rough-looking area in an apartment block. It was cramped and looked quite dark and drab. I don’t remember the exact figure, but he said it cost something like £1600 per month in rent alone.

Recently at work, we had a casual chat and talked about wages. One Tester sounded a bit outraged at the wages we pay and she said “Testers should be paid as much as developers”. I strongly disagree with this. I think Manual Testers really could earn £30,000 max. I used to be a Tester and for the most part, I see it as low skilled. Maybe the best testers like myself (arrogant) use better analytical skills, and can read the code and understand how to possibly find gaps and therefore find the most bugs. However, most Testers I work with don’t really think and they just do what you tell them. So it mainly involves mindlessly clicking buttons in the UI.

Anyway, this outraged Tester then stated she didn’t mind disclosing her salary and she gets paid £40,500 for a Senior Tester – which is going to be higher than many developers! Senior Developer wages start at £39,000 where I work, and I was under the impression that Senior Testers were more like £32k. How she managed to wrangle that wage is anyone’s guess. Why she thinks she deserves more is even more baffling.

Crisis Meeting

Apparently my employer has recently made a big deal and sold our software to a large group of users, but the users now want to reverse their decision based on other customer’s opinions of our software. Not sure why this group of customers only found out these opinions after making the deal.

We had a meeting with the new Head Of Development to discuss what we can do about it. The overall aim is that we need to repair our reputation: fixing major bugs, improving performance, improving the user experience in general, and making it clear to our customers that we value their opinion.

He initially stated that he wants more bug fixes, especially performance improvements and delivered in smaller, but frequent releases. He states that the current process doesn’t work and we should ignore it and come up with a new one.

Sounds great in principle, and a desired philosophy of Agile Development. However, Release Management stated it wasn’t possible to update the entire customer-base so quickly. This is partially due to the way some users work, and some contractual obligations. Both of these problems pretty much come down to the idea of essentially wanting close to zero downtime, so they want to choose a specific time to accept our updates, and they like this to be around once a month.

So the discussion turned to the development side. It seemed we were in agreement that we need higher-skilled staff to target these important bugs (we love throwing Juniors at these important issues), and we would also like to work in Domain based teams. Examples of this team idea is that you’d have one team that exclusively handles the Reporting module, and another handles Registration module etc. This way you can have a fuller understanding of all the configuration, features and the codebase; so can fix/enhance the software faster. Essentially you specialise, rather than going with the jack-of-all-trades master-of-none style. We also wanted more Testers since some teams have produced a lot of fixes recently but the Testers couldn’t get through it. To be honest, I expected that some Testers were just working slow and the fix isn’t necessarily to throw more staff members at the problem (just have a stern word).

The response from the Head of Development was that “the business” has decided on the team sizes and who is in those teams – and they won’t change.

So to summarise – we are asked how we can make dramatic changes to the way we work, ignoring the current process… then get told we can’t do our ideas. We’d shot-down his suggestion, then he shoots down ours.

I think he shot down our suggestion because he is too focussed on the current process and structure, and he isn’t taking his own advice by ignoring it and coming up with a new idea. To be honest, it is easier said than done, and I think his way of thinking is – how to preserve the current in-flight projects? You can’t simply restructure the entire department without dealing with the current projects.

What you can do is just make the current project teams bigger and let them take the project and some bug fixes. With a bigger team, they can easily carry on with their project work, and with the extra people, they take on live Bugs, fixing “Technical Debt”, adding more automated tests etc.

I get confused how we get told these “Service Improvement” teams (teams that deal with bug fixes only) are the most important, yet it’s the Project teams that get the most experienced developers and the most attention. He has limited the “Service Improvement” teams to 20 developers with only 4 with Senior rank, one of which isn’t good enough to be Senior. Then most of the other developers are essentially Juniors.

How can you deal with important bugs with a lack of talent? It’s frustrating that he arranges a meeting to ask how we can solve our way of working, but then he doesn’t listen to the feedback.

If he wants a quality product and to deliver faster, then we need the staff and process to actually achieve it.

Mentoring #5

I am mentoring an Apprentice who has never done C# before, and this is his first programming job. So this is a diary of some-sort of his progression and my ability to mentor. I haven’t written about my Apprentice in detail for a while – it’s been 4 months!

The bad thing is that there’s no good news to talk about, but I can sure rant about him instead. I’ve spent a lot of time with him over the past year, giving him general advice based on my journey as a developer and telling some great stories. I’ve explained what I think makes a good developer, emphasising how it is fundamental to know how to debug effectively, and how you need to find the root cause of an issue rather than papering over the cracks.

When a batch of Apprentices joined the company, we had no plan for them, so managers passed them between teams aimlessly. When he joined my team, I promised him I would put in the effort and be the mentor he expected to have when he joined. So I can help him reach whatever potential he has, but he needs to put in the effort to get there; I can just get him there faster. I said he can ask as many dumb questions in the first year, but after that, then I will judge him.

I’ve given him exam questions, a pdf book, a list of project ideas I spent a few hours writing, a list of easy bugs to fix, and I frequently send him code samples to discuss. I’ve even made some basic programs and set the challenge to enhance them. I’ve done live coding demos, talking him through how to make a noughts-and-crosses game. I said I would code review anything he sends me.

I told him that it’s not mandatory to do everything I send him, because people learn in different ways, but he needs to put the effort in to do something. Maybe he doesn’t like reading and prefers watching videos? The key thing is just to get writing code.

However, with my Apprentice, he has been with me a year, and yet has basically nothing to show for it. He just seems to make excuses and somehow time has flown by.

I did get another Apprentice assigned to me. He did the exam questions within a couple of days. He made his own Battleships program in the following week, he completed a few easy bugs in the weeks following that. Then he moved teams, but I thought he was probably ready for proper work. He did seem far more naturally gifted, but he also had the high determination and took all my advice and instructions on board. I couldn’t have been happier with him.

A few months ago, my Apprentice also switched teams and I said to him he was nowhere near ready to be assigned proper work. The work his team had planned would feature more advanced, Senior level topics like threads, and required good knowledge of the domain.

I advised him to partake in meetings and do a bit of pair-programming so he has a feel for what is going on, but he must spend half the day practicing code, working through the assignments I had given him.

After a few weeks he calls me and says he has felt he has wasted his time and learned nothing. I asked him what code he has written and he says he has only gone to meetings and been watching a colleague work. He then asks me for advice. I remind him of my original plan and he says “oh yeah, you did say that, I should always listen to your advice because you are always right”.

A few weeks go by and I ask him why I still haven’t seen any code from him. He said he felt bad doing his own personal tasks, on work time. In his team stand-ups meetings, he felt like he needed to talk about his team’s work otherwise his colleagues would think he isn’t contributing. I ask him what he has spoken about if he hasn’t done any work for the team. He has not contributed anything. At least if he actually practiced coding over the last couple of months, he would have had the skills to try and contribute from now on.

He said I was right again, although he was still adamant he wanted to do some proper work. He promised to start his project at the weekend and continue to do it outside work. So I picked out a few bugs from the backlog that looked easy. His Product Owner and Scrum Master weren’t happy about bringing in work to their team’s backlog when it wasn’t planned. I explained that currently their work is far too hard for him and he isn’t contributing so should imagine him not in the team. So let’s give him some easy work, then he can do something. Even if the work is shelved and the Testers don’t have time to test it, at least he has learnt something and he will feel like he has done something.

So after 1 year of my mentorship, I can now judge him and he has been a complete disappointment. He has only done 2 easy bug fixes with my supervision. In some ways, I feel like a failure but I know I’m not a bad mentor. I have put so much effort in and gave him so much advice. I feel like he disregards it, then says I am right, but then doesn’t change and carries on with the wrong approach. I definitely held my side of the bargain of being the mentor that they promised him.

Has he started his project yet? No. He just comes out with excuses. He does have some good reasons, but it’s hard to justify not being able to find a few hours here and there over a year, or even just the last couple of months:

Moving into a new flat
Went to visit a friend down in London
Forgot his son was staying the weekend
His son enjoyed his stay so wanted to stay again
Decided to start boxing as a new hobby
Watching the Euro 2020 England game.

I’ve stopped asking him about it, because I feel like I am just repeating myself, and have given him so many ideas of what to code. He isn’t in my team so I feel like I’m not responsible for him anymore. I’ll still be around if he wants help or advice because I enjoy helping people, but I’m not going to be proactive anymore.

Truncate

A developer sent a review for a method that was displaying a tooltip. He was fixing a simple “null reference” exception, but the logic for displaying the tooltip seemed weird:

If the text is greater than 100 characters, we don’t show a tooltip. If it is less than 100 then we do.

So I suggest that we always show the tooltip, but if it is greater than 100 characters, then we truncate to 97 characters plus some ellipses “…” to show the user that some text is missing.

At first, the developer responds that he was already truncating it. So I explain again that it doesn’t show at all in the case the text is greater than 100. He agrees to fix it. Later, he asked me to review the code again.

Now he is showing the tooltip if it is greater than 100 characters, but he is showing 100 plus the 3 ellipses to make 103 characters in total; which is over his defined limit. It’s a bit weird but not a huge problem. Additionally, he now no longer shows a tooltip when it is less than 100 characters, so he has just reversed the problem!

So I point out both the problems and he agrees to fix them. He asks me to review the code once more. Despite fixing these issues, he has now reintroduced the original problem that he was fixing! Now it will crash again with a “null reference exception”!

So I say

“Doesn’t this reintroduce the bug you were originally fixing?”.

It’s a rhetorical question of course, I know full-well that it does; but it’s more polite than an expletive-ridden comment questioning his intelligence.

“No, it doesn’t bring back the original bug back in – I have verified that.”

I ask him if “he is sure about that?”. I don’t even need to test it, I can see that it is 100% a problem just by reading the code. I’d expect a Junior to easily spot that without testing it. Not sure how he could miss it if he had done some proper testing.

“As per my basic testing it didn’t crash but I can do some more testing and let you know.”

1 hour goes by. He then replied:

“Just now I saw the mistake from my end”

(－‸ლ) 🤦