Printing Licence Key Expiry

My employer made the news recently after the deployment team applied the wrong config to several organisations which led to a heavy spike in network traffic and caused issues nationwide.

Sadly, we made the news again.

“We couldn’t print our forms for three hours. Someone had goofed and did not keep up with their subscriptions. For a company of their size; that is embarrassing.”
Customer quote

When users reported the issue, there was a call between a few managers. I was invited to the call with zero context by someone I had never spoken to before, so I thought there was a good chance they invited the wrong person.

The call was recorded so I just listened to it when I was free. They had invited me on the suggestion that I had worked on a feature in that area many years ago, and they had no other ideas. Also on the call, they called someone else and remarked how strange it was to the recipient because they had never spoken to him before. Why didn’t they learn and send some context? He didn’t join either.

Eventually, they found someone who belongs to the team that procures the licences. He explained that they purchase licence keys for this printing software, then send them to an internal support team to update the licence keys in the database. The team receives automated emails reminding them to renew the licence keys 3 months prior to expiry, and they act on it quickly to not risk them expiring.

“It takes a while to go through the purchase process, so I usually do it early, but sometimes it can “fall through”
Procurement guy

After going through some emails, they found the key had been promptly purchased, and the licence keys were sent to support, then they were not applied.

Another guy joined the call and said a total of 43 organisations have reported errors, but that’s the only ones we know about due to direct complaints we saw via Facebook.

“And then I need to understand exactly how this is happening, ‘cause this is the second time in two weeks that a licence key was sent to support and wasn’t applied.”
Angry manager

After the issue was resolved, another manager asked for a summary of the issue. One guy remarked

“just to say, the people invited to the call (16 invited in total), are not the ones that ended up being involved or resolving.”
Manager, reflecting on what a mess the meeting was

I don’t understand how all our departments are causing chaos all at once? We don’t seem to learn our lesson either, how can we make the same mistake twice in two weeks? It’s also been a problem for years – that we struggle to invite the correct people to major incident calls so issues take much longer to resolve.

Software Support Call Centre

We used to have a very large support team in our own call centre. As a software developer, we were occasionally sent to go talk to them from time to time, and I was amazed at how busy they were. Usually, as soon as they had put the phone down, they had another user call up.

Sometimes it was that the user just didn’t know how to use the system, and other times it was to complain about a software bug or slow performance. The call centre staff were rapid at entering the information into the system, and were brilliant in asking the user the right questions to really understand their problem. They could often tell the user if the issue was logged or not, and also give them some relevant work-arounds.

After speaking to some of the staff, they explained how strict the culture was there – they were monitored on how fast they picked up the phone, how long the call was, and how many breaks they had. They said how annoying it was to be warned about being late when it was due to bad traffic.

It surprised me because it seemed a completely different culture to how the development department is run. We are flexible when we start; so you can just turn up late and no one cares. We were never challenged on how long our work took to complete. I guess if our work is poor quality, it’s the call centre staff that took the complaints!

At some point, some manager decided to use a 3rd party company for Support, and most of our Support staff either left or (presumably) got redundancy.

The amount of complaints seemed to go up on various social media platforms, and I got the impression this 3rd party company didn’t know our software so were just providing users generic statements from a script “can you try turning it off and on again?”. Maybe if they got past the initial questions, they then got put through to our smaller 2nd-line Support team.

A few years later, I think a new manager came in and decided to try to reverse the decision, but it’s going to take a bit of time to get the new staff as good as the old ones.

“As part of our drive to strengthen our customer satisfaction and experience along with simplifying our ways of working for both our customers and the service desk, the decision has been made to insource the call centre. All calls will now come directly into the service desk.

We have already run a couple of trial switch offs over the last fortnight and the initial feedback has been unanimously positive with customers preferring to be directly connected to the service desk; just in this small sample there has been an increase in both the quality of cases and first-time fixes. We will continue to invest in developing a world class service desk.”
Company Announcement

It seems obvious to me if you make the support more generic, then it’s going to decrease customer satisfaction.

The only problems I had with Support is:

A) when they would link completely different issues to the same bug report. Sometimes you see that a bug that you thought should be super rare – has had 100’s of reports from the users. Then when you look into the cases, you see that 90%+ of them are unrelated. We could have probably put some advice on how to decide if issues are the same root cause or not, in order to try to help remedy this.

B) Sometimes there’s other data entry errors that end up being misleading, like this:

Support (in the free-text description): 
"however we have been able to re-create this on the test system by"...

also support (in the mandatory fields):

"Recreated in-house: No
Reason not recreated: Unable to recreate"

Dealing With Software Support #2

I recently had to deal with another company’s software support team, and this was the second bug I had logged. The first bug didn’t go well at all. For this second issue, I had provided them detailed recreation steps, and videos of the issue occurring.

After I logged the issue, they said they would investigate and get back to me shortly. After nearly 2 weeks, I received an email requesting to arrange a phone call. I thought he was going to give me some news but he wanted me to demo it. So I asked him what that would give him over the videos. He said the videos I sent him wouldn’t play. Brilliant.

I didn’t understand why the videos won’t play. I recorded them using the Microsoft Game Bar feature and they ran fine on my machine. Instead, he wanted me to record the video with Powerpoint. This is bonkers. At least I learned that Powerpoint can record screens. It’s quite useful because you can zone out a tiny part of your window to record…but then there is no option to simply record the full screen!

The problem we had was that our software was intermittently crashing when it was interacting with theirs. However if you changed some Security settings in their software to “never warn me about suspicious activity”, then the crash didn’t occur. You would have thought it should pop up a message box rather than crash. So I asked their Support specifically if he had any idea why this would happen. If it was something I could change at our end, then maybe I could quickly resolve the issue.

“Our software may be thinking this 3rd party app is suspicious. And disabling that security setting helps!”
Support

Well, that sure was helpful. I bet he referred to Captain Obvious for that one.

Why is it intermittent? Why would it think our application is suspicious? why would it crash instead of popping up a message if it was suspicious? My line of questioning is to prompt him into getting to the bottom of the issue but it seemed he couldn’t be bothered investigating or even logging it with their Development team with this information.

I was also annoyed how he kept on chasing my responses when I’d barely had any time to respond. In his email signature, it said he was working 9-5:30 Monday to Friday, and he sometimes sent emails at 8pm on a Friday. Then I’d also get an email 9am Monday reminding me that I haven’t replied to his last email. If he doesn’t work weekends, why does he assume I do? He has literally given me 0 working hours to respond.

There were even occasions where he wouldn’t even chase me by email, but would chase me up by an actual phone call that I didn’t agree to. We had put our IT’s department’s phone number on the Support ticket. I told him many times to contact me by email, but we can arrange a Microsoft Teams call if we need to talk. He would then email saying he couldn’t get through by phone. So I would remind him

“The phone number is for our IT department. I don’t have a direct number.”

Then he would sometimes respond with something similar to:

“We tried to connect you by phone, but unfortunately unable to connect”
Support

Absolute wind-ups.

I find that they always want to arrange calls, even though they end up asking something that can have been addressed by email. They must get reviewed by how many calls they make or something. I don’t understand the advantages. Being put on the spot to give information over the phone isn’t as effective as asking in an email and waiting for the person to have time to acquire the information when they are free. But maybe that’s just my preference? Still, they should respect the customer’s preferences.

At one point, he suggested that the reason why some users didn’t encounter the issue was due to a different “Microsoft .Net Framework” version installed on their machine. I asked him the best way of finding this information out. He replies with the following:

I found the framework version in the error listed in Event Viewer for the affected machine. You can check if they are different by comparing a working machine and non-working machine.
Support

Do you see a flaw in their plan?

A working machine doesn’t log an error in the Event Viewer

Dealing with Software Support

I was invited to a call with some of our users who wanted to explain some of the problems they had in a particular feature of our software. One of the problems was actually an issue with a massively-popular 3rd-party software we integrate with, so they wanted me to log a bug with the 3rd-party on their behalf.

A reasonable request I thought, but I really didn’t want to have loads of phone calls/email chains back-and-forth with support. I just wanted to log it and forget about it. Luckily, I am a developer so I have a good understanding of all the information Support/Development would want.

This company that I logged the bug report with must be one of the Top 10 software companies in the world. So you’d expect their Support to be of a high standard. Due to the fact that the bug report is still open at the time of writing, I’d better keep their identity a secret, otherwise they could easily look up my name from the bug report and my anonymity is blown.

The feature is that you click a button, then their software will open a dialog with some data pre-filled in. If their software is currently closed, it still opens the dialog (their software is just temporarily open). In this scenario, when you save/close, then their software is supposed to close back down completely. However, it was getting stuck and you had to use Task Manager to kill it. This problem can be recreated consistently every time.

So I put together a great description of the problem, the recreation steps, the version of Windows, the exact version of their software I was using. I told them our users have recently upgraded from ‘X’ edition to ‘Z’ edition; and I was using ‘Y’ edition and can now see the problem after installing ‘Z’ edition so the problem is definitely in this new edition. Since I didn’t want to provide them with our software to recreate the issue, I even found some of their software with a similar feature and recreated it on there too. But to make sure they definitely didn’t consider it to be a problem with our software, or their other software product, I found another 3rd-party, free-to-use software and recreated it there too.

So I’d say they have a 100% chance of recreating that themselves, and I had proven it was a problem with their latest iteration of software.

Within an hour, I get an email asking questions. So it seems I failed to provide them the information required:

What version of Windows are you using?
What exact version of our software are you using
When was the feature last seen working?

I had literally given him all those answers. So I reworded my original report and put them in bullet points. He then mails back saying he cannot recreate it and would like a demo. Unbelievable. I can recreate it everytime, our users can recreate it every time. It’s not a Windows issue because I was using Windows 8 and our complaining users are using Windows 10.

He specifies that the demo must be on Windows 10. Why? Is he just trying to mess me about? You can’t expect someone to have access to various computers. The version of Windows is irrelevant.

Regardless, I accept and I set up a Virtual Machine. It’s like a fresh computer install so maybe it is better than my work computer to prove this is a bug. I put their software on it, even created a brand new account. However, now my user doesn’t have the Premium licence but I accept the trial, so had 14 days. I can recreate it every time. Surely he hasn’t tried hard.

So I arrange the call, share my screen and I demo the feature to him. I ask if he did the exact same thing. He then tells me the main difference was that he was using the very latest version. Is he trying to wind me up? I told him the exact version, he asked for the version, then didn’t use it! Surely if he couldn’t recreate it on the latest version, then he could have checked on the version I was using; then rejected the bug as fixed. Instead, I waste an hour configuring the Virtual Machine and another 30 mins demoing it to him.

He was supposed to send me instructions on how to update because it wasn’t as simple as clicking a button. However, he didn’t bother and I found the instructions after some Googling. He was right that it was fixed…or kind of. I tested it several times and I saw the same issue again. So I tried some more and it worked for maybe 30 times then failed. So it’s intermittent. So I tell him it’s not perfect. He wants another demo because he can’t recreate it.

So I demo the problem and he said he cannot help because he noticed I’m “not using the Premium account” and he is in the Premium support team. I told him I do have a licence, I was just using this account to rule out if it was a problem if you had a large user profile: basically doing his investigation for him.

However, I have to deploy a new Virtual Machine since the settings don’t allow me to sign in (seems to be some restriction from our IT department). So after setting up yet another Virtual Machine and signing in with my work account (with the Premium licence), I do another demo. However, the intermittent nature means the issue didn’t happen and the meeting had already lasted 20 mins. I was pretty bored clicking a button and closing a dialog for that length of time. So I said if I had time, I’ll try and record it happening, but in the meantime, I’ll send him the recordings I had from before (with the Standard account).

After a few days, he says that he is going to log a ticket with the Microsoft Windows 8 Support Team, so I ask him why. It doesn’t make sense when the issue occurs on both Windows 8 and Windows 10. He said it was because it only happens on Windows 8 since I had demoed it working on Windows 10. I told him again that I have recreated the issue with the latest software on both Windows 8 and 10, and using 2 different user accounts. The recordings I sent him were Windows 10. It was just that the issue was intermittent and coincidentally worked during the time I was demoing it.

He apologises for the misunderstanding and would need some time with his ‘senior’ to come up with the way forward. A few days later, he says that his senior wants me to arrange another call to test out some scenarios. I told him I didn’t understand what the plan was. He apologises for the misunderstanding, but there’s some scenarios that we need to test. I told him I didn’t understand what the plan was: am I doing a demo? or am I watching them do a demo? He apologises for the misunderstanding, but he wants me to deploy another Virtual Machine and create a new user and demo this to them. Surely, I needed to know that and set up a Virtual Machine and new user account before we start the call. It would have been a bit awkward if I set up the call, he asks me to demo it on a machine I wasn’t logged into and with a user I hadn’t even created.

I told him I didn’t understand what the plan was. Why do I need yet another Virtual Machine and user account when I recreated the issue already using 2 accounts, 1 Windows 8 physical machine and 2 Windows 10 Virtual Machines? He said they had attempted to recreate the issue with multiple users and on different Virtual Machines, but since the problem seems to exist on my Standard account, then he would like me to verify it on another Standard account.

Let’s recap: when he concluded it was a Windows 8 issue, I told him it wasn’t because I’d recreated it on Windows 10. It seemed that me demoing the feature working on Windows 10 was the thing that he picked up on, and completely disregarded everything I wrote and said. I had told him on that call it was intermittent and I had seen it fail that same day on the same Virtual Machine. Now we are in a situation where I had told him I had recreated it with multiple users, but he is still thinking of that demo I did where it was working. Not only was it Windows 10, but I was using my Premium account at the time. So now he has this idea that the problem is with my Standard account.

I think I need to take him to a hypnotist to train him to forget the existence of that demo. What’s he going to claim next? That because he saw it working on a Friday, then it’s a Monday to Thursday issue?

My dream of just logging the issue and them instantly confirming the bug was completely ruined.

It’s dragged out for a month or something daft and I feel like I have invested weeks of time investigating it. This is literally his job and I am doing his work and telling him how to do his job. It’s ridiculous.

I keep saying to my colleagues that it seems that he is trained to mess people about so they drop the issue then they don’t have to fix anything. It’s a bizarre strategy, but what other logical explanation is there for his behaviour?

I like our system: Regardless of whether Support can or cannot recreate the issue, the bug gets prioritised based on what they know. If it is deemed high priority then it will get sent to Development, otherwise probably gets thrown on the backlog and we might look at it in a year… or never. But I think the key aspect is that we always believe our users, so it does get logged (“the customer is always right”). Well, I guess we get evidence like a video or screenshot, but I had already provided those – and we would never dismiss video evidence.

As a side note: I wish that low priority bugs were dealt with faster because it will discourage users from reporting them if they think we will ignore it for a year. Maybe they can be given to Junior Developers to clear down.