The Transparent Placation of the Employee Forum

A few years ago, the directors announced they were starting a Programme to improve the company culture, and act upon feedback we had recently given them via an “Employee Forum“.

I do think due to working at home, there isn’t the same togetherness there used to be. Years ago, we seemed a close-knit bunch, but these days, I get the impression people just want to do their work and go home…although they are already home.

We were asked to give initial feedback from the scheme, and one colleague made a somewhat epically-written post, which ended up being his last.

“This “programme” is an exercise in indulging in the same old political rhetoric and transparent placation many of us have grown all too familiar with. Perhaps I have simply grown jaded and cynical, but we have corporate correspondence envisioning a future where all feel valued, and yet casual communication engenders entirely the opposite. To, on the one hand, read statements such as the above and then be told such things as “if you want better pay, get another job”, is insulting to the intelligence and dedication of employees. This, “programme”, is in fact a desperate response to the abysmal employer image the company has cultivated for years; to this day exhibiting a callous disregard for even its most dedicated employees – you reap what you sow.

In short, this isn’t a question of “focus”; rather, it’s a question of chasing each of these lofty goals with equal levels of tenacity, integrity, and humility. This “programme” is years too late and, despite a ledger demonstrating the contrary, the company has only ever proven itself to be more than “a dollar short”.

The goals outlined as part of this programme infer/suppose a complete lack of humanism in company culture and are a tacit admission of gross negligence when it comes to employees; the shoulders upon which the company stands.”

Rodger

The directors responded with predictable responses:

“this does not reflect everyone’s view of our business”

“this isn’t the right platform to raise these issues”

“we value colleagues feedback and encourage great constructive conversations”

“I can assure you that the people on the programme are thoroughly committed to making a positive difference in our dedicated areas.”

“judge at the end and not at the start”

Various Managers

I understand both sides here, they did ask for feedback and they got it. However, they don’t want something so scathing to be publicly aired. It’s a tough one though because such statements could cause people to be more confident in airing their views (which is what they claim to want), but firing him like they did will suppress people from giving true feedback.

The usual process is probably to go to your line manager, but they don’t always raise it with their line manager. Then since it has to go up a few levels of the hierarchy; the message is probably going to get lost. If you suspect that happening, do you bypass them and go higher up the chain?

It was also written a bit too poetically which meant it was hard to understand some of the points. I think there is a specific and recent issue which he was referring to, and in the following days, I did learn that an employee that had been around for 20 years or so and been forced out. I didn’t learn the reason, but that was being referenced with “exhibiting a callous disregard for even its most dedicated employees“. So I think it was coming from a place of witnessing instances of toxic behaviour and calling it out, and it wasn’t even a personal grievance; he is sticking up for fellow employees.

The obvious (or maybe I should say “predictable”) response was to sack him, which they did. But you could have actually gone the opposite way and actually let him participate in making the changes to improve the culture. I only interacted with him a few times, but he always came across as knowledgeable and dedicated.

He goes on to say others keep quiet because they need their job, although it is low paid compared to industry standards. Many roles are overworked due to restructuring and merging roles together. Managers were ignoring the problems when raised. Proposed changes in Support were predicted by many to end in failure, but managers dismissed their concerns. I have seen similar comments on Glassdoor, which is where you don’t want the feedback, since that is on a public forum, and posted after an employee has left the business.

On a quest to get some juicy content for my blog, I managed to contact him before he got deactivated:

[Yesterday 15:07] Me
have you resigned. That post is epic

[Yesterday 15:07] Rodger
No. But don't plan on sticking around long haha

[Yesterday 15:08] Me
haven't you just started in development though? or did they do the classic "we will give you half pay until you prove yourself"

[Yesterday 15:14] Rodger
Sad but true

[Yesterday 15:20] Me
They are putting my notice period up from 1 month to 3. I made the point that this is contrary to the Employee of Choice which is supposed to be improving our conditions!

[Yesterday 15:20] Rodger
Good lord, that's rough...
And yeah, certainly feels that way

[Yesterday 15:21] Me
can't remember who was in charge at the time, but he put it down from 3 to 1, stating they wanted to improve things, and now we are undoing it. Apparently Experts and Managers are on like 6 months which just seems insane.

[Yesterday 15:22] Rodger
6 months. That's ridiculous, what company would be willing to wait that long for a new employee...

[Yesterday 15:23] Me
I don't see the point of changing the notice periods. Don't people just hand over their work in 1-2 weeks, take annual leave, then mess about for the rest of it
paying an unproductive employee who doesn't want to work there is just crazy. Even if they are moving for other reasons like leaving the country, it seems like you have to then leave on bad terms. Just let them go. 1 month is enough to deal with

[Yesterday 15:25] Rodger
Indeed. Don't get the mentality at all

[Yesterday 15:26] Me
anyway, I asked Jeannette to raise it in the Employee Forum. I reckon they will just say something like "its inline with industry standards" or some corporate statement

[Yesterday 15:27] Rodger
Oh aye, wasn't expecting a response.
Just wanted the catharsis of saying it all
Thanks regardless, though

[Yesterday 15:28] Me
they will probably delete your post and not try and address it. Reminds me of the negative glassdoor reviews then Jacqui says "this doesn't reflect my experience at all"
and I think, "how do you know what those jobs are like? you are on the big bucks, probably in your own office by yourself"

<NEXT DAY>

[12:05] Me
I couldn't resist responding [my post was about lines of communication, and trying to start a discussion with others around Rodger’s points to try and diffuse the argument]

[12:05] Rodger
Thanks haha

[12:07] Me
have they actually said anything to you directly? It's good that they didn't delete your post but then I was worried they would just sack you for these outbursts. I do think you might be a bit too (passive?) aggressive, but then I reckon you do have some good underlying points that they should listen to

[12:08] Rodger
Any aggressiveness, passive or otherwise, is only because the responses to my post have demonstrated a lack of understanding and contempt.
My initial post is nowhere near anything combative...
And, no. No one has spoken to me directly. People have spoken to my Manager and others around me though...

You've raised the right questions here.
Where can one speak about these things

[12:11] Me
I think someone did say (via a question) on our Departmental Meetings that the whole "we will take it offline" response just causes these problems. If there is an official response to it, everyone doesn't hear it because it was addressed one-on-one. Then I think that's what you are also calling out, because these Senior Managers are basically implying to contact them directly, rather than post in an open forum.

[12:12] Rodger
Aye

[12:13] Me
in my standup, one guy said he had to get a dictionary out to try and understand your points. It is super poetic. It makes it funnier but I do think it probably detracts from your points

[12:15] Rodger
Fair, but it's genuinely the way I speak and I thought it best to be as erudite as possible when articulating it all; in an effort to not just be dismissed as a dribbling idiot or fool, etc.

[12:27] Me
This does sound a bit extreme to me. Then that would be perceived to be an attack on the SLT
"The goals outlined as part of this programme infer/suppose a complete lack of humanism in company culture and are a tacit admission of gross negligence when it comes to employees; the shoulders upon which the company stands."

[12:31] Rodger
It certainly is a criticism.
The company admits that its staff don't feel; rewarded, included, valued, etc. (as inferred by the programme itself).
Is that not gross negligence?
I am perhaps opining there, but it doesn't seem to me to be an unreasonable statement and I would be happy to discuss the matter if literally anyone else was...

[12:35] Me
It can be a tough one because I think most people will complain about pay regardless of how much they actually get. Then if they don't get an inflation rise, they will moan about that but they might have been overpaid. So when people do complain, it might not be a valid point. but then I have been a victim of being underpaid

I think a common problem we have is that managers see us all as interchangeable. I have only been on this project a few months but we are gonna get our 3rd PO, and we have switched architects, and there was talk about moving one of our testers to another team. How can you be productive when that happens? can't build up knowledge or team chemistry. They still see us as "resources" they can move about. Probably doesn't help if a department is called Human Resources.

[12:38] Rodger
Indeed

[12:38] Me
I find it interesting how people like me moan, but yet stay here years so we must secretly like it

[12:39] Rodger
haha

[12:40] Me
I think it is actually a low stress job, and when I've known so many people leave and then come back, it does make you think it's not always greener on the other side. I also think other jobs will give you more responsibility. So I think it is comfortable here

do you find development stressful, or were your comments about your previous role?

[12:46] Rodger
I guess I'm in something of a unique position.
My comments were about my previous role, how things are now for me (which ain't "bad", just insanely busy), and everything I've heard and seen from elsewhere in the business

Conclusion

I think the lesson here is that if you really want to make changes to culture, then you need to be open to listening to “hard truths”. So if there was a problematic manager – if an employee openly reports the problem, it shouldn’t be the case that managers stick together. The claims should be investigated and if the manager is problematic, then they should be fired and not the employee. 

I think we lost a couple of good and very dedicated employees here, and yet managers were pretending everything was perfectly fine.

Project Aurora & The Strangler pattern

Recently we have had another tech guy join the company who is reporting to the CTO. I find that people in these kind of roles want to put their stamp on things by coming up with a new idea.

He presented his idea in our monthly Tech Meeting. He wants to attempt to address our performance problems by taking traffic away from our main on-premise databases. There’s been some similar ideas recently, and although I’m not great when it comes to hardware, networks and general software/hardware architecture; I am sceptical that these ideas can work.

His idea is that we can replicate the database in the cloud (“the cloud” solves all problems you see), and then the database in the cloud can be used for Read access, whereas Write would still go to the main on-premise databases (then synced up to the cloud).

The Announcement

This programme of work is to move workload away from our primary systems to enable these systems to withstand expected load factors from upcoming initiatives as well as expected growth in usage on our APIs during Winter 2023.

The intent is to run focused cross functional teams in work-streams across the group to deliver this initiative. The approach taken here is to place multiple bets, across multiple teams. The expectation is that not all teams will deliver by September, but enough to bring in the headroom we need.

The programme is intending to free up at least 20% load across our core databases.

Upcoming aims:
• Strategic, move read-only workloads to Aurora.
• Redeploy APIs to AWS, Move to cloud technology, Containerise and Optimise Service
• Enable use of replica data when ready.
• Move Appointment Workload
• Mitigate 8am peak load.
• Use caching engine on AWS (Elasticache/Redis), mitigate 8.2% of PC DB Load 
• Reduce load on the DB during day time.
• Reduce Datacentre and DB load and improve performance
• Mitigate 6.2% of DB load by optimising how we summarise task counts
• Proof of concept is Complete, expected to cost £2m a year.

My Conversation With Architect Mark

I think the reason for the replication (as opposed to just moving it all to the Cloud) is that you can’t fully commit to ideas like this. You have to have a rollback plan. So if we find it doesn’t work, is too expensive etc., we can just return to the old way without much inconvenience. I asked one of our Software Architects what he thought of the plan because it doesn’t sound right to me:

Me
doesn't sending data out to another database just increase traffic, and they wanted to reduce it?
Mark
Yes, it will also be delayed, and often broken
Me
no pain, no gain
Mark
they're replicating data, and it's unlikely it'll be used
Me
I don't see how you migrate things. You have to keep them both running until you are confident it works, then bin off the old database. But then in reality you just end up keeping them both for longer than expected
Mark
you then also need cross-database transactions or to be very careful with queries
yeah, that's basically it. Have the same API at both ends, some sort of replicate and transform on the data to ensure it's in both. Persist to both simultaneously, then when all works, turn off the old
Me
The CTO said that “some people say there is a delay, but it is only 5 minutes”. Does that address any of your concerns at all?
Mark
no, this is only the second time I've heard about this, and the first I laughed
I agree with the principle of strangler pattern for migrating, but this isn't migrating
it's keeping multiple DBs 'in-sync'
Me
does that mean you can view an appointment book which is 5 mins out of date, and you try book an appointment, then it checks the real database and is like "no mate you cannot do that"

The conversation between architects

Mark then sent me a conversation he had with two other architects, Andrew and Jon. Mark already had concerns with the “appointment book” example.

Mark
so when this replication system goes down for a few hours, what happens then? I guess the system tries to book appointments for slots already booked, put in requests for items already issued etc.?
seems our business layer needs to be aware of how outdated the original information was, so it can compare something like a changelog number. Sounds like a big challenge to implement correctly

Andrew 11:10
Yes, any write operations will need logic to ensure that cannot happen Mark.
John and I have already called out that Appointments and Orders will have significant challenges with this replication model and have suggested that the initial focus should be on User Profiles, and any historic data, etc.

Mark 11:13
User Profiles and historic data seem just as dangerous to be honest.

Jon 11:15
The idea I suggested these is that you would check the change log on the primary system before even considering going to the replica. If the User had had a recent change (what counts as "recent" is TBC, I suggested 30 minutes) you wouldn't even consider going to the replica.

Mark 11:15
can we implement the strangler pattern properly? set up proper Appointments APIs to use in our datacentre, and AWS.
duplicate the data.
then dual file everything against the APIs? if one fails to file, the other gets rolled back.
we ensure consistency, we can transform the data, and we're using the pattern as intended
Jon, I agree your idea is the right way to do this sort of thing, but it will be adding logic and latency in a lot of places (as well as augmenting every one of our products to be aware of this), and not bringing us forward, but continuing to keep us in the primary data-store model

Jon 11:18
Honestly if the use case for customers looking at their data, then having it a touch out-of-date information isn't as critical as if our actual users sees an out of date view. As a hypothetical Customer who knows nothing about IT, if I viewed my record straight after a consultation
and it wasn't there I would just assume that there was a delay and it would appear later.
When it comes to actual Users viewing the record, it's absolutely critical that they see the up to date view. And when it comes to appointments that's also critical because appointment booking is fast moving, it'd be an awful experience for a User if every "free" slot they booked turned out to be booked minutes earlier.

Mark 11:19
depends, if you've just requested a particular item and the page doesn't update to indicate that, can you continue requesting it?

Jon 11:20
Many of our users (mine included) turned off online appointment booking entirely at the beginning of the pandemic and use a triage system now.
You wouldn’t be able to successfully request duplicate items, because the write would take place conditionally, so if it had been requested already then it'd say no (if designed even
vaguely competently).

Mark 11:22
the write wouldn't come through, but it'd be confusing for the User seeing the prescription still requestable, unless the application has its own datastore of state

Jon 11:22
Yes it would be far from ideal. But the CTO has some ideas about that (having a "recent changes" dataset in a cache that is updated live, and merged with the replica's data.
feels like there's loads of little bits of logic that need 'tacking on' to resolve potentially quite serious incidents. When the correct use of the strangler pattern gets us away from on-premise as primary DB, and moving in the direction we want to go
Yeah, this isn't easy and requires careful consideration.

Andrew 11:30
You are absolutely right Mark - there are a heck of a lot of potential gotchas and ultimately the plan has to be to use the strangler pattern, but at the moment we are looking at a rescue plan to put out some existing fires in the data centre and to handle predicted significant increase in load that will hit us in the Autumn. Everything that you have flagged is being considered.
The only fall-back plan that we currently have is to spend nearly £4m / year on additional SQL Server readable secondaries (on top of having to pay an additional 12% on our existing SQL Server licences thanks to MS hiking their prices) and nobody has the appetite for that.

Closing Thoughts

I don’t know what the Strangler Pattern is, so I’ll add that to my reading lists. However, it seems that even with my limited knowledge of architecture, our Software Architects have similar concerns as I do. There’s been plenty of ideas that the CTO (or similar level managers) have quickly backtracked on due to not consulting people who have knowledge on whether their idea is actually logically sound. I’ll keep my eye on this idea to see how it develops.