Printing Licence Key Expiry

My employer made the news recently after the deployment team applied the wrong config to several organisations which led to a heavy spike in network traffic and caused issues nationwide.

Sadly, we made the news again.

“We couldn’t print our forms for three hours. Someone had goofed and did not keep up with their subscriptions. For a company of their size; that is embarrassing.”

Customer quote

When users reported the issue, there was a call between a few managers. I was invited to the call with zero context by someone I had never spoken to before, so I thought there was a good chance they invited the wrong person. 

The call was recorded so I just listened to it when I was free. They had invited me on the suggestion that I had worked on a feature in that area many years ago, and they had no other ideas. Also on the call, they called someone else and remarked how strange it was to the recipient because they had never spoken to him before. Why didn’t they learn and send some context? He didn’t join either.

Eventually, they found someone who belongs to the team that procures the licences. He explained that they purchase licence keys for this printing software, then send them to an internal support team to update the licence keys in the database. The team receives automated emails reminding them to renew the licence keys 3 months prior to expiry, and they act on it quickly to not risk them expiring.

“It takes a while to go through the purchase process, so I usually do it early, but sometimes it can “fall through”

Procurement guy

After going through some emails, they found the key had been promptly purchased, and the licence keys were sent to support, then they were not applied.

Another guy joined the call and said a total of 43 organisations have reported errors, but that’s the only ones we know about due to direct complaints we saw via Facebook.

“And then I need to understand exactly how this is happening, ‘cause this is the second time in two weeks that a licence key was sent to support and wasn’t applied.”

Angry manager

After the issue was resolved, another manager asked for a summary of the issue. One guy remarked

“just to say, the people invited to the call (16 invited in total), are not the ones that ended up being involved or resolving.”

Manager, reflecting on what a mess the meeting was

I don’t understand how all our departments are causing chaos all at once? We don’t seem to learn our lesson either, how can we make the same mistake twice in two weeks? It’s also been a problem for years – that we struggle to invite the correct people to major incident calls so issues take much longer to resolve.