The Transparent Placation of the Employee Forum

A few years ago, the directors announced they were starting a Programme to improve the company culture, and act upon feedback we had recently given them via an “Employee Forum“.

I do think due to working at home, there isn’t the same togetherness there used to be. Years ago, we seemed a close-knit bunch, but these days, I get the impression people just want to do their work and go home…although they are already home.

We were asked to give initial feedback from the scheme, and one colleague made a somewhat epically-written post, which ended up being his last.

“This “programme” is an exercise in indulging in the same old political rhetoric and transparent placation many of us have grown all too familiar with. Perhaps I have simply grown jaded and cynical, but we have corporate correspondence envisioning a future where all feel valued, and yet casual communication engenders entirely the opposite. To, on the one hand, read statements such as the above and then be told such things as “if you want better pay, get another job”, is insulting to the intelligence and dedication of employees. This, “programme”, is in fact a desperate response to the abysmal employer image the company has cultivated for years; to this day exhibiting a callous disregard for even its most dedicated employees – you reap what you sow.

In short, this isn’t a question of “focus”; rather, it’s a question of chasing each of these lofty goals with equal levels of tenacity, integrity, and humility. This “programme” is years too late and, despite a ledger demonstrating the contrary, the company has only ever proven itself to be more than “a dollar short”.

The goals outlined as part of this programme infer/suppose a complete lack of humanism in company culture and are a tacit admission of gross negligence when it comes to employees; the shoulders upon which the company stands.”

Rodger

The directors responded with predictable responses:

“this does not reflect everyone’s view of our business”

“this isn’t the right platform to raise these issues”

“we value colleagues feedback and encourage great constructive conversations”

“I can assure you that the people on the programme are thoroughly committed to making a positive difference in our dedicated areas.”

“judge at the end and not at the start”

Various Managers

I understand both sides here, they did ask for feedback and they got it. However, they don’t want something so scathing to be publicly aired. It’s a tough one though because such statements could cause people to be more confident in airing their views (which is what they claim to want), but firing him like they did will suppress people from giving true feedback.

The usual process is probably to go to your line manager, but they don’t always raise it with their line manager. Then since it has to go up a few levels of the hierarchy; the message is probably going to get lost. If you suspect that happening, do you bypass them and go higher up the chain?

It was also written a bit too poetically which meant it was hard to understand some of the points. I think there is a specific and recent issue which he was referring to, and in the following days, I did learn that an employee that had been around for 20 years or so and been forced out. I didn’t learn the reason, but that was being referenced with “exhibiting a callous disregard for even its most dedicated employees“. So I think it was coming from a place of witnessing instances of toxic behaviour and calling it out, and it wasn’t even a personal grievance; he is sticking up for fellow employees.

The obvious (or maybe I should say “predictable”) response was to sack him, which they did. But you could have actually gone the opposite way and actually let him participate in making the changes to improve the culture. I only interacted with him a few times, but he always came across as knowledgeable and dedicated.

He goes on to say others keep quiet because they need their job, although it is low paid compared to industry standards. Many roles are overworked due to restructuring and merging roles together. Managers were ignoring the problems when raised. Proposed changes in Support were predicted by many to end in failure, but managers dismissed their concerns. I have seen similar comments on Glassdoor, which is where you don’t want the feedback, since that is on a public forum, and posted after an employee has left the business.

On a quest to get some juicy content for my blog, I managed to contact him before he got deactivated:

[Yesterday 15:07] Me
have you resigned. That post is epic
 
[Yesterday 15:07] Rodger
No. But don't plan on sticking around long haha
 
[Yesterday 15:08] Me
haven't you just started in development though? or did they do the classic "we will give you half pay until you prove yourself"
 
[Yesterday 15:14] Rodger
Sad but true
 
[Yesterday 15:20] Me
They are putting my notice period up from 1 month to 3. I made the point that this is contrary to the Employee of Choice which is supposed to be improving our conditions!
 
[Yesterday 15:20] Rodger
Good lord, that's rough...
And yeah, certainly feels that way
 
[Yesterday 15:21] Me
can't remember who was in charge at the time, but he put it down from 3 to 1, stating they wanted to improve things, and now we are undoing it. Apparently Experts and Managers are on like 6 months which just seems insane.
 
[Yesterday 15:22] Rodger
6 months. That's ridiculous, what company would be willing to wait that long for a new employee...
 
[Yesterday 15:23] Me
I don't see the point of changing the notice periods. Don't people just hand over their work in 1-2 weeks, take annual leave, then mess about for the rest of it
paying an unproductive employee who doesn't want to work there is just crazy. Even if they are moving for other reasons like leaving the country, it seems like you have to then leave on bad terms. Just let them go. 1 month is enough to deal with
 
[Yesterday 15:25] Rodger
Indeed. Don't get the mentality at all
 
[Yesterday 15:26] Me
anyway, I asked Jeannette to raise it in the Employee Forum. I reckon they will just say something like "its inline with industry standards" or some corporate statement
 
[Yesterday 15:27] Rodger
Oh aye, wasn't expecting a response.
Just wanted the catharsis of saying it all
Thanks regardless, though
 
[Yesterday 15:28] Me
they will probably delete your post and not try and address it. Reminds me of the negative glassdoor reviews then Jacqui says "this doesn't reflect my experience at all"
and I think, "how do you know what those jobs are like? you are on the big bucks, probably in your own office by yourself"
 
<NEXT DAY>
 
[12:05] Me
I couldn't resist responding [my post was about lines of communication, and trying to start a discussion with others around Rodger’s points to try and diffuse the argument] 
 
[12:05] Rodger
Thanks haha
 
[12:07] Me
have they actually said anything to you directly? It's good that they didn't delete your post but then I was worried they would just sack you for these outbursts. I do think you might be a bit too (passive?) aggressive, but then I reckon you do have some good underlying points that they should listen to
 
[12:08] Rodger
Any aggressiveness, passive or otherwise, is only because the responses to my post have demonstrated a lack of understanding and contempt.
My initial post is nowhere near anything combative...
And, no. No one has spoken to me directly. People have spoken to my Manager and others around me though...
 
You've raised the right questions here.
Where can one speak about these things
 
[12:11] Me
I think someone did say (via a question) on our Departmental Meetings that the whole "we will take it offline" response just causes these problems. If there is an official response to it, everyone doesn't hear it because it was addressed one-on-one. Then I think that's what you are also calling out, because these Senior Managers are basically implying to contact them directly, rather than post in an open forum.
 
[12:12] Rodger
Aye
 
[12:13] Me
in my standup, one guy said he had to get a dictionary out to try and understand your points. It is super poetic. It makes it funnier but I do think it probably detracts from your points
 
[12:15] Rodger
Fair, but it's genuinely the way I speak and I thought it best to be as erudite as possible when articulating it all; in an effort to not just be dismissed as a dribbling idiot or fool, etc.
 
[12:27] Me
This does sound a bit extreme to me. Then that would be perceived to be an attack on the SLT
"The goals outlined as part of this programme infer/suppose a complete lack of humanism in company culture and are a tacit admission of gross negligence when it comes to employees; the shoulders upon which the company stands."
 
[12:31] Rodger
It certainly is a criticism.
The company admits that its staff don't feel; rewarded, included, valued, etc. (as inferred by the programme itself).
Is that not gross negligence?
I am perhaps opining there, but it doesn't seem to me to be an unreasonable statement and I would be happy to discuss the matter if literally anyone else was...
 
[12:35] Me
It can be a tough one because I think most people will complain about pay regardless of how much they actually get. Then if they don't get an inflation rise, they will moan about that but they might have been overpaid. So when people do complain, it might not be a valid point. but then I have been a victim of being underpaid
 
I think a common problem we have is that managers see us all as interchangeable. I have only been on this project a few months but we are gonna get our 3rd PO, and we have switched architects, and there was talk about moving one of our testers to another team. How can you be productive when that happens? can't build up knowledge or team chemistry. They still see us as "resources" they can move about. Probably doesn't help if a department is called Human Resources. 
 
[12:38] Rodger
Indeed
 
[12:38] Me
I find it interesting how people like me moan, but yet stay here years so we must secretly like it
 
[12:39] Rodger
haha
 
[12:40] Me
I think it is actually a low stress job, and when I've known so many people leave and then come back, it does make you think it's not always greener on the other side. I also think other jobs will give you more responsibility. So I think it is comfortable here
 
do you find development stressful, or were your comments about your previous role?
 
[12:46] Rodger
I guess I'm in something of a unique position.
My comments were about my previous role, how things are now for me (which ain't "bad", just insanely busy), and everything I've heard and seen from elsewhere in the business

Conclusion

I think the lesson here is that if you really want to make changes to culture, then you need to be open to listening to “hard truths”. So if there was a problematic manager – if an employee openly reports the problem, it shouldn’t be the case that managers stick together. The claims should be investigated and if the manager is problematic, then they should be fired and not the employee. 

I think we lost a couple of good and very dedicated employees here, and yet managers were pretending everything was perfectly fine.

Project Aurora & The Strangler pattern

Recently we have had another tech guy join the company who is reporting to the CTO. I find that people in these kind of roles want to put their stamp on things by coming up with a new idea.

He presented his idea in our monthly Tech Meeting. He wants to attempt to address our performance problems by taking traffic away from our main on-premise databases. There’s been some similar ideas recently, and although I’m not great when it comes to hardware, networks and general software/hardware architecture; I am sceptical that these ideas can work.

His idea is that we can replicate the database in the cloud (“the cloud” solves all problems you see), and then the database in the cloud can be used for Read access, whereas Write would still go to the main on-premise databases (then synced up to the cloud).

The Announcement

This programme of work is to move workload away from our primary systems to enable these systems to withstand expected load factors from upcoming initiatives as well as expected growth in usage on our APIs during Winter 2023.

The intent is to run focused cross functional teams in work-streams across the group to deliver this initiative. The approach taken here is to place multiple bets, across multiple teams. The expectation is that not all teams will deliver by September, but enough to bring in the headroom we need.

The programme is intending to free up at least 20% load across our core databases. 

Upcoming aims:
    • Strategic, move read-only workloads to Aurora.
    • Redeploy APIs to AWS, Move to cloud technology, Containerise and Optimise Service
    • Enable use of replica data when ready.
    • Move Appointment Workload
    • Mitigate 8am peak load.
    • Use caching engine on AWS (Elasticache/Redis), mitigate 8.2% of PC DB Load 
    • Reduce load on the DB during day time.
    • Reduce Datacentre and DB load and improve performance
    • Mitigate 6.2% of DB load by optimising how we summarise task counts
    • Proof of concept is Complete, expected to cost £2m a year.

My Conversation With Architect Mark

I think the reason for the replication (as opposed to just moving it all to the Cloud) is that you can’t fully commit to ideas like this. You have to have a rollback plan. So if we find it doesn’t work, is too expensive etc., we can just return to the old way without much inconvenience. I asked one of our Software Architects what he thought of the plan because it doesn’t sound right to me:

Me
doesn't sending data out to another database just increase traffic, and they wanted to reduce it?
Mark
Yes, it will also be delayed, and often broken
Me
no pain, no gain
Mark
they're replicating data, and it's unlikely it'll be used
Me
I don't see how you migrate things. You have to keep them both running until you are confident it works, then bin off the old database. But then in reality you just end up keeping them both for longer than expected
Mark
you then also need cross-database transactions or to be very careful with queries
yeah, that's basically it. Have the same API at both ends, some sort of replicate and transform on the data to ensure it's in both. Persist to both simultaneously, then when all works, turn off the old
Me
The CTO said that “some people say there is a delay, but it is only 5 minutes”. Does that address any of your concerns at all?
Mark
no, this is only the second time I've heard about this, and the first I laughed
I agree with the principle of strangler pattern for migrating, but this isn't migrating
it's keeping multiple DBs 'in-sync'
Me
does that mean you can view an appointment book which is 5 mins out of date, and you try book an appointment, then it checks the real database and is like "no mate you cannot do that"

The conversation between architects

Mark then sent me a conversation he had with two other architects, Andrew and Jon. Mark already had concerns with the “appointment book” example.

Mark
so when this replication system goes down for a few hours, what happens then? I guess the system tries to book appointments for slots already booked, put in requests for items already issued etc.?
seems our business layer needs to be aware of how outdated the original information was, so it can compare something like a changelog number. Sounds like a big challenge to implement correctly 

Andrew 11:10 
Yes, any write operations will need logic to ensure that cannot happen Mark. 
John and I have already called out that Appointments and Orders will have significant challenges with this replication model and have suggested that the initial focus should be on User Profiles, and any historic data, etc. 

Mark 11:13 
User Profiles and historic data seem just as dangerous to be honest.

Jon 11:15 
The idea I suggested these is that you would check the change log on the primary system before even considering going to the replica. If the User had had a recent change (what counts as "recent" is TBC, I suggested 30 minutes) you wouldn't even consider going to the replica. 

Mark 11:15 
can we implement the strangler pattern properly? set up proper Appointments APIs to use in our datacentre, and AWS. 
duplicate the data. 
then dual file everything against the APIs? if one fails to file, the other gets rolled back. 
we ensure consistency, we can transform the data, and we're using the pattern as intended 
Jon, I agree your idea is the right way to do this sort of thing, but it will be adding logic and latency in a lot of places (as well as augmenting every one of our products to be aware of this), and not bringing us forward, but continuing to keep us in the primary data-store model 

Jon 11:18 
Honestly if the use case for customers looking at their data, then having it a touch out-of-date information isn't as critical as if our actual users sees an out of date view. As a hypothetical Customer who knows nothing about IT, if I viewed my record straight after a consultation 
and it wasn't there I would just assume that there was a delay and it would appear later. 
When it comes to actual Users viewing the record, it's absolutely critical that they see the up to date view. And when it comes to appointments that's also critical because appointment booking is fast moving, it'd be an awful experience for a User if every "free" slot they booked turned out to be booked minutes earlier. 

Mark 11:19 
depends, if you've just requested a particular item and the page doesn't update to indicate that, can you continue requesting it? 

Jon 11:20 
Many of our users (mine included) turned off online appointment booking entirely at the beginning of the pandemic and use a triage system now. 
You wouldn’t be able to successfully request duplicate items, because the write would take place conditionally, so if it had been requested already then it'd say no (if designed even 
vaguely competently). 

Mark 11:22 
the write wouldn't come through, but it'd be confusing for the User seeing the prescription still requestable, unless the application has its own datastore of state 

Jon 11:22 
Yes it would be far from ideal. But the CTO has some ideas about that (having a "recent changes" dataset in a cache that is updated live, and merged with the replica's data.
feels like there's loads of little bits of logic that need 'tacking on' to resolve potentially quite serious incidents. When the correct use of the strangler pattern gets us away from on-premise as primary DB, and moving in the direction we want to go 
Yeah, this isn't easy and requires careful consideration. 

Andrew 11:30 
You are absolutely right Mark - there are a heck of a lot of potential gotchas and ultimately the plan has to be to use the strangler pattern, but at the moment we are looking at a rescue plan to put out some existing fires in the data centre and to handle predicted significant increase in load that will hit us in the Autumn. Everything that you have flagged is being considered. 
The only fall-back plan that we currently have is to spend nearly £4m / year on additional SQL Server readable secondaries (on top of having to pay an additional 12% on our existing SQL Server licences thanks to MS hiking their prices) and nobody has the appetite for that.

Closing Thoughts

I don’t know what the Strangler Pattern is, so I’ll add that to my reading lists. However, it seems that even with my limited knowledge of architecture, our Software Architects have similar concerns as I do. There’s been plenty of ideas that the CTO (or similar level managers) have quickly backtracked on due to not consulting people who have knowledge on whether their idea is actually logically sound. I’ll keep my eye on this idea to see how it develops.

Development Environments

This blog is basically stealing “Healthy Software Developers” explanation of “Development Environments – Isolating Customers From Your Changes”

Introduction

“Development environments” allow software engineers to view a particular version of the software and control who has access to it. 

If you go back about 20 years ago, there was much less maturity with how changes were rolled out to customers, and every company had a different process. In some companies, developers would just immediately roll them out, straight into production. This might work for the most part if you have a very small and trusted team without the process or contractual requirements. These days, the standard is that most companies need a minimum of three separate environments when developing software.

Even where I work, people used to tell me it used to be much less professional, and without as much legal scrutiny – allowing them to drop DLLs onto a server to quickly fix a bug for a single customer. Now there’s all kinds of people who need to sign it off, and 99% of changes are released as an official version to multiple customers.

Development Environment

The main development environment is a place where an individual engineer can actually work on the software in isolation from any other stakeholder such as a Tester or Customer. Instead of making changes directly to a live website/application, the changes can be run on a local copy on the developer’s machine. The developer can begin looking at the impact of the changes he’s making, and will use test data. Any problem introduced only affects the single developer.

It could be as simple as a copy of a database and a version of the website. A more sophisticated program may have even more dependencies, so need multiple websites or services configured – so you could leverage modern technology like “cloud services” or “containers” to easily deploy this setup. The decision of how this looks depends on the complexity of the software architecture, team size and process.

Staging Environment

The “User acceptance” test environment, sometimes known as a “staging environment”, is another copy of the application, but developers aren’t making any changes to it, and it is also using test data. The purpose of this environment is typically to run exhaustive testing to find any issues before deploying the changes to the end users. 

When companies rely on manual testing, and don’t have modern “continuous delivery” processes, an unreleased version may sit in the staging environment for an extended period of time. In a more automated environment, automated tests can be run, and the version easily deployed to customers if passed. This could take as little as fifteen minutes to an hour depending on the tests, software complexity, and process involved.

Production environment 

The Production environment, or Live environment, is where your customers use your product; so the actual website or application. 

Demo environment

You can have other environments like a Demo environment. This is another copy, but not used by Developers or Testers. The version can be “work in progress”, or a version that is deemed to be ready; but the purpose is to show customers upcoming features and to gather feedback.

Capacity testing

You could create a special kind of test environment for capacity/load testing. This is meant to  stress test the software and see how it’s performing under heavy load. Systems can have increased traffic at specific times of the day, e.g. users all logging in at 9am to start the day, increase in website hits during lunch breaks, more shopping traffic during Christmas period etc. If users are paying to use your service and it goes down; it’s a huge problem. If you sell a product/service on your website and it goes down; then it can cause lost sales. So performing this testing can be essential.

Office Tales: Food

One thing we miss now that we work at home is the free treats people will bring in, and some interesting food thefts. Here is a collection of stories I found from old emails and chat conversations.

Suspicious Chocolates

I was once walking past the security desk, and the security guard was quizzing a guy that had turned up with a box of chocolates. When the security guard asked who the chocolates were for, he kept on saying “for the person in charge”. Yet he couldn’t specifically name someone, or explain why they would be receiving such a gift. I couldn’t guess what his aim was. Was he expecting to be let through to wander around the office on his own. I think the best he could have done is for the security guard to take the chocolates.

Office Treats

When people do bring in treats, it was always a good read when they tried to write a humorous email to declare it.

Hi all, 

There is a selection of young Tobleronette in the upstairs kitchen freshly picked on the Swiss Alps this weekend. 

It’s been a poor harvest this year due to bad weather (caused primarily by the blocking highs in the UK) so you might as well take this opportunity to savour some wholesome mountain produce. 

Who knows when you may get the chance again. 

Bon appetit 



David

Canteen

We used to have a canteen that was announced to be closing due to cost cutting. However, it ended up being very temporary.

Farewell canteen and the awesome staff that knew us all by name, how cool is that? They provided caffeine infused productivity juice that was both hot, fresh and cheaper than anywhere else. Hello vending machine, with your cold touch and bags of mini-cheddars. 2016 has really sucked.  

Sausages

This is a weird email to send, admittedly, but has someone taken my sausages out of the downstairs kitchen fridge? 

Thanks, 

Andy

This triggered some humorous replies with various sausage pictures

Stolen Milk

Me: There's a note on our kitchen door saying that "someone has been SEEN taking DI Team's milk"


Andy: hahaha

Me: not sure why they didn't reprimand the guilty culprit

it's like he was only partially seen so couldn't be identified. But all they know is that he was non-DI

Fruits

A few developers had dried up fruits on their desks, like a shrivelled orange. They claimed the office environment naturally dried the fruit rather than rotting it, then they kept it as an ornament. I wondered if they took the fruits home and still have them on their desks.

Missing Mugs

Someone has left a black flask/cup on my desk this morning which says “Looking after your world” and “Hot Stuff…” on it. 

If this is yours, please come claim it.

Morning, 

If anyone has seen my ‘Slave’s mug’ mug please can you let me know? I left it in the upstairs kitchen with my teapot on Friday afternoon and it’s this morning it was gone. 

I’ve had it 20 years and grown quite attached to it. 

Thanks,

Kettle

Hi all

If you are filling the kettle please ensure you don’t dip the bottom of it in the bowl of water (perhaps emptying the water out of the washing up bowl first would be the way forward?).  It will trip the electrics out (as has just happened) and water on the kettle base is potentially dangerous anyway.

Fí

Cafe2u 

There used to be a van that came to the office to serve fresh coffee and snacks which was called Cafe2U. Someone mailed about it and typed U2. I made a great joke about it.

From: Me

Subject: FW: CafeU2

 

Like Coffee?

Like U2?

Check out Café U2. They have parked where the streets have no name.

 

Café U2 Promoter | Bono Enterprise

 



From: Gill

Subject: Cafe2u Yeadon
 

Cafeu2 are here. 

Gill |  Testing | Development

Ant Infestation

Me: Did you hear about the ant infestation caused by Chris? anyway, Mel said that Chris left a can of Red Bull in the office for a few days, and a horde of ants were around it, so they had to call pest control.

Andy: haha i’m sure we don’t call it pest control in england. yet again an americanism 

Me: we call them Bug Bashers around here. Sam has spilled a drink behind his monitor and has left it. Mel is kicking off about it, fearing a new infestation.

Pig Balloon meeting

No idea what a pig balloon meeting is, but I love how passive aggressive this is.

I have made the executive decision, by reason of proximity to where these were left lying around unclaimed, to do something with the pop and dips left over from your pig balloon meeting.

I have put the pop (which is probably flat now) and the dips (BBQ – expire Sept 17) into the downstairs kitchen for people to take.  If they are not used up/gone by Friday, they shall be binned.  Please help yourselves.  

Dorothy

The Pregnancy

My manager received an email from HR apologising for some erroneous emails:

Email Notification Error – PLEASE IGNORE 

We are always looking to improve our HR processes and carry out regular routine work within our IT System to ensure we deliver the smoothest experience. 

It has come to our attention that a recent tweak to our maternity process has accidentally triggered an email to be sent to some employees. 

Please note, this email was sent in error. 

If you received this email, please ignore it and we apologise for any confusion caused. 

Many thanks,

Paul

The email they are referring to is a pregnancy email sent about male employees. I wonder what sort of change they made to their system. Were they modernising and allowing male employees to declare pregnancy?

Hello Andy, 

It's now been a few months since we last contacted you, advising that Steven notified us of their pregnancy. 

We'd like to remind you of a couple of things, should they be applicable:

Since completing the initial Expectant Mothers Risk Assessment, should either your or your team member feel their work circumstances have changed and would benefit from a further assessment, please complete a new assessment and upload to the "Risk Assessment screen

If the original maternity dates have changed, please remember to raise an HR ticket to ensure their payment during their maternity leave is calculated correctly. 

Should you require any further support, please raise a HR ticket.

Test Specifications

Many  years ago, when I was a Software Tester, I remember when we had to write a Test Specification based on the work that the Developers had planned. This was for both Enhancements and Bug Fixes (so new features and changes to old ones).

It would be a Word document, with the Item Number, Title, and then a description of what you would test (the description would be a bit more high level than the step-by-step description featured in an actual Test Case).

You would spend weeks writing it, then you had to get it approved by all the developers, or the Dev Lead. The developer often then told you it’s nothing like you imagined so you had to rewrite it. 

Sometimes they would demo the feature to you so you had a better idea. If they had no comments, I often felt that they didn’t read it.

When there was a new Developer who wasn’t familiar with the process, he “rejected” a Test Specification because of some grammatical issues. I think it was something to do with the wrong tense, and not using semicolons. The Tester was fuming, but he was quite a belligerent character.

I think we often worked from the Bug description, or some comments from the Developer. However, quite often, the comment section would be the Developer writing something generic like “test bug is fixed”, “check data is saved“. If it was more detailed, sometimes you would paste the developer’s notes and change a few words – and have no idea what it meant until you saw the finished thing.

The Verdict

I think both Developers and Testers saw Test Specifications as a waste of time. The Developers weren’t enthused to read it, especially when most people rewrote what the Developer provided, and that might not be the full test coverage needed. The Testers should be adding more value by using their existing knowledge of the system to come up with additional scenarios to cover “regression testing”. 

I think the only advantage is to quickly verify that the Developers and Testers were on the “same page”, but that only works if the Tester has not used the developers words and tried to illustrate that they do understand the upcoming features.

I think it eventually got binned off for what we called “Cycle Zero Testing” which was where the developer quickly demoed their changes; which I think was still hated by the Developers but was easier to understand the value, and was a more collaborative between the two roles.

The Price Of Cheese

It’s a non-programming blog again, but this story is too good to share.

One of my colleagues, Andy, bought some cheese for Christmas from a specialist, and his mate pointed out that the same cheese was way cheaper at Morrisons (a popular supermarket in northern England).

So he threatened to post a bad review, and the Cheese Care & Quality Manager at the company gave him an essay about the cheese making process and schooling him how their cheese is higher quality.

So if you are interested in the process of making Stilton, then continue reading:

The Response:

Dear Andy,

Thank you so much for emailing us and giving us a chance to respond to this for you. I can certainly see why you would be both concerned and upset to see this really big price difference.

There are two answers to this question. The first is that the cheese isn’t identical. It is certainly made by the same producer but not to the same recipe. Most of the Stilton made by Cropwell Bishop is made in the main building at their dairy. It is a good cheese made by a talented producer, but the method of production involves moving large quantities of curds from the vat to the draining table using gravity.

The cheese that we buy is their hand-ladled animal rennet Stilton. This is made in a smaller room to the side of the main production area. In this room the cheese is made in much smaller vats and is made entirely by hand. A crucial step in Stilton making is the transference of the curds from the vat to the draining table. This is done by the traditional, painstaking method of ladling scoops of curds, one by one, and layering them up gently so that the fragile curds are as undamaged as possible. This method has a big impact on both the texture and flavour of the cheese.

Here is a photo of this taking place. You can see that it is a very different process to the large movement of curds that would happen in most larger scale cheesemaking.

A tiny proportion of the overall make of the dairy is made to this method, but this is the only one that we buy. We believe that you can really taste the difference and it is very interesting to do a side by side blind tasting. I hope you would agree that the cheese is not the same if you were to do this.

The second answer is that even given this difference, I am still really surprised to see their standard Stilton being sold at this price and can’t help wondering if Morrisons had some sort of promotion going on at Christmas. The Best range Stilton from Morrisons is usually made by the Long Clawson dairy rather than Cropwell Bishop. Long Clawson Stilton is much cheaper than Cropwell Bishop and is definitely considered to be of a lower quality by most in the cheese industry. I had a look at the Morrison’s site today and the “The Best” Stilton on the site is listed as being made by Long Clawson. In fact, I have never seen standard Cropwell Bishop Stilton being sold at this price. Cropwell Bishop themselves charge £20.57/kg when buying a half cheese and £23.68/kg when buying a 454g piece. They don’t sell their hand-ladled animal rennet Stilton online so I can’t make a comparison. 

I have also looked at other online retailers to get a sense of what other people are charging. The Cropwell Bishop standard Stilton varies from around £29.75/kg to £34.95/kg. So, while it is certainly true that Morrison’s would have much bigger purchasing power than we do, I still find the price of £15/kg to be a really surprising one and wonder if perhaps there may have been some trouble with supply from Long Clawson that led them to replace “The Best” range with Cropwell Bishop but charge the same price. However, this is speculation, and I may be wrong. I shall go to my local Morrisons and take a look as soon as I can out of interest. Perhaps they really do usually charge this price, in which case, they are undercutting both the supplier themselves and any other retailer that I know. I can only say that it is a huge bargain at this price and we would not have a business at all if we tried to compete with it. Our cheese is always well looked after in our maturing rooms and is cut fresh to order. It is never pre-cut and wrapped, which has a really detrimental impact on quality and we have confidence that we are supplying the best cheese that we can, both by buying only the hand-ladled Stilton and ensuring it reaches our customers as fresh as possible.

I very much hope this goes some way to explaining the price difference. Thank you for ordering your Christmas cheese from us. I very much hope you enjoyed it.

Best wishes,

Ruth

Cheese Care & Quality Manager

Changes to the Software Delivery Process

One of the problems we have where I work – is not releasing fast enough. When you read about Software Development, you hear of these companies that can release minor updates every week. Maybe it is more of a Web Development thing rather than an Application Development one, but there are also contractual reasons why we cannot release faster.

However, over time, the release time has continuously crept up to 4-6 weeks which causes problems.

If there is more time for each release, it means that the scope of the current release often increases further. For example, if there is a fix that needs to go out within 3 weeks to meet the SLA (Service Level Agreement) and the next release will go out in 4 weeks, then you have little choice but to get it in the current release. If you are checking it in close to the deadline, then you might end up delaying the release in order to test it. The more you delay, the more chance of someone else needing to get a fix in the current release and it grows further.

If there’s many big projects targeting the same release, each team developers in their own code “branch”, then will merge into the Main branch for release. Since it’s not really feasible to all merge at the same time, you end up taking it in turns and resolving any conflicting changes. To be honest, it’s quite rare that we will change the same files for the main feature changes, but there’s certain files with a lot of churn, mainly ones containing XML. To merge in projects, it usually takes a few days, then all the bug fixes on top. The Testers can’t really begin testing until it’s all merged so it’s a lot of overhead to manage the release.

When the releases are large, the Testers insist on running more Regression Tests which increases the Testing phase and can cause further delays.

“I think we spent about 2 months on that regression. It was madness. It was a HUUUGE release”

Software Tester

So smaller releases are much more manageable, take much less time to test, incur less risk, and have lower scope for scope-creep.

Our Software Delivery team made an announcement about this (basically just saying the same things I have just discussed), and desire to plan in quarters but release in a couple of weeks.

In the past, we would scope a single release looking at the features, fixes and minor enhancements we wished to deploy. We would follow a process of merging everything into our main release branch before undertaking testing. This was a two-phased testing approach, integration/functional testing each feature and fix, and then regression testing to ensure pre-existing functions continued to work as expected. We would then spend eight to ten weeks deploying the release through Controlled Roll Out and Customer User Acceptance Testing.
 
This approach brought with it a number of challenges. Merging everything in was time consuming, issues or blockers with one feature would slow down or block other features, regression testing was a challenge, and this also put pressure on the roll out and deployment through pushing out a number of changes in one go.
 
To try and mitigate some of these challenges, we are now adopting a strategy of breaking these large releases down into smaller updates.
 
Working in quarterly cycles we will scope what we wish to deliver over a 12 week period. Each feature will be analysed and risk assessed for size and complexity by our Engineering Leads, and have a business value determined by our Product Management and Commercial Teams.
 
Using this feedback we will then determine an order in which we wish to deliver each feature. We will then merge them into a release and test them one at a time (potentially two if both are small and low risk), before signing over to Release Management to commence deployment.
 
We will then deploy the full scope over a series of smaller releases rather than in one large release.
 
The last update in the cycle will be a maintenance release to address the backlog of prioritised service work.
 
The objective behind this approach is to have our users benefit by taking elements of the release scope earlier than they would have before, whilst also simplifying the testing approach and hopefully enabling us to push code out across the estate quicker.

The Secret the Task Manager developer didn’t want you to know!

Dave Plummer, who has the Youtube channel Dave’s Garage announced on Twitter:

Big news! Someone finally noticed that if you hold down CTRL, the process list in Task Manager conveniently freezes so you can select rows without them jumping around. I did this so you could sort by CPU and other dynamic columns but then still be able to click stuff…

Dave Plummer

There’s been plenty of occasions where Task Manager rows jump around to my annoyance. Why wasn’t this a more obvious feature? Frank Krueger (who appears on Merge Conflict podcast) made the obvious point:

Don’t hide features under random key combos – undiscoverable and unmemorable UIs are user hostile. A little checkbox with the text “Pause Display” would be discoverable and you won’t have to wait 30 years for someone to find your feature.

https://x.com/praeclarum/status/1693649521375621524?s=20

Unity Runtime Fee

Intro

Unity have announced a new fee which they call the “Unity Runtime Fee” which is going to take effect in January. It  affects all Unity developers, even people that have already released their game many years ago; which has caused mass outrage among the game development community.

I think the existing model states that once you reach a threshold of revenue, you have to pay a licence fee which works out around £1500 for the year. With the new model, once you reach a similar threshold, Unity is now going to charge a fee of 20 cents every time somebody instals your game on a new device for the first time.

The threshold is $200,000, which on the face of things, 20 cents per install doesn’t sound unreasonable when they have given you a great tool to help you create your game. They need to earn money as a business and deserve some kind of cut for their service/product. According to this tweet, it looks like they are burning through money so some drastic action is probably required

https://x.com/georgebsocial/status/1702696194558816751?s=20

There’s a fair few aspects of why this is new model is complicated, but I still feel some of the anger is misplaced.

I think the whole scenario is similar to what I have written about recently where the CEO demanded we release our software weekly instead of monthly and we told him several reasons why it is technically, and legally impossible. Then later he then demands all our changes have a well-documented rollback plan, and again, we told him loads of reasons why it wasn’t possible. He still insisted and looked like a fool when it backfired and caused a few problems he thought he was solving.

The main Problem

The core problem stems from the idea that it is based on installations and not based on Unit Sales or Revenue. For comparison, the main competitor, Epic Games’ Unreal Engine charges you five percent of your total revenues after you’ve earned at least a million dollars on your game. Now, that can work out to be a lot of money and especially in the long run if your game is successful, but the difference is that they’re taking a cut of your money that you’ve already earned. When Unity charges you for an installation you’re being charged whether or not you’ve earned any money, or at different period of time to where you earned the money. That could turn into a cash flow problem.

Collage of abuse:

https://x.com/LiamSorta/status/1702325745610338646?s=20

Theoretical scenarios

Once you’re over the threshold, if somebody bought your game a long time ago and they’ve now installed your game on their brand new computer, it’s going to cost you 20 cents. I suppose if it is an old game, you probably won’t be selling $200k over the last 12 months, so it’s probably not actually going to apply.

If you decide to port your game to a new platform which is often fairly easy in the Unity engine, then all those new installations are also going to be hit with fees. I suppose if you are re-selling the game then it’s not a major problem, but sometimes developers make a free-to-play mobile version. Then you make money later with microtransactions. Often these games have 90% of players not paying a penny, but then you make your money on the 10% who often spend big. In this case, you could end up losing money on the average player of your game.

People also raised the point of bundles like the Humble Bundle where people buy a bunch of games for a small price but some of the money goes to charity. You end up selling high volumes but gain very low revenue. If you hit the threshold, and you are more likely to with a sale like this, then you could be hit with a lot of fees. I think the interesting thing with this point which people don’t seem to be mentioning; is that people often buy these games then never actually play them. So you actually have a sale, but no install, so don’t pay the fee.

Fairly similar to a bundle is a service like Xbox Game Pass, where people could play your game with an overall payment to the provider, in this case Microsoft. I think Microsoft often pays a flat fee to the publishers to gain their games but I suppose contracts can vary. But the theory is, you could get a flat fee, then either get low instals so you’ve gained, or get a surprisingly large amount of instals if it is popular and it eats into your profits.

Piracy

People who pirate games don’t pay but do install your game. This means that every time your game is pirated you’re going to be slapped with a 20 cent fee. There can be other malicious ways you could be charged, if someone abuses Virtual Machines. There’s programs that will spin up large numbers of them, so you could “Install Bomb” quickly with virtual machines, hitting the developer with a 20 cent fee. It’s like when people “Review Bomb” where you leave loads of negative reviews on a game you don’t  like in a coordinated way, but in this case you need fewer people, and they directly sap the revenues of the developer instead of just hurting their online presence.

Target Price

Unity has always positioned itself as being pro Indie. They want to help new aspiring Indies learn to program, break into the gaming market, and get their career started. New developers are also much more likely to sell their games for cheap. There’s a lot of games like this on Steam which are sold for £10 or loads for £5 or less, and that’s before you apply discounts. Steam is renowned for its high discounts in sales, and so these games are being sold for just a couple of pounds. They’re going to be disproportionately hit by having to pay Unity 20 cents every time the game is installed.

In the extreme case, imagine you’ve made a Steam game or a mobile game that sells for one dollar and then you pay a sales tax of 10-20%, then Steam takes 30%, then you know you’re left with around 50 cents. If you use a Publisher, then they will take their cut too. Then Unity takes 20 cents of it for an install and then maybe another 20 cents for another install, then you could be left with basically nothing. You could then lose money if it isn’t sold for full price.

Meanwhile, if you sell a premium game for £40+ then 20 cents is nothing. So it actually hits the indies harder. Unity have ways of getting the price per install down, but they look more aimed at larger companies who will want to pay the upfront fees to use the premium Unity features.

Patch Quest

Lychee Game Labs’ Patch Quest released on 2 March 2023 and so far has reached 182,594 total key activations on Steam (people who bought the game on Steam along with everyone who got the game elsewhere like in a bundle or a giveaway or for review purposes). So if the game keeps selling, or people install on more devices, then he will be taken over the threshold then would start being charged. He did remark that “for the sake of argument, every single person who already owns the game decided to install it on a second PC, I’d be hit with a charge of $36,400. Now it’s obviously not likely that this would happen” but it does make you think how Unity are gonna deal with these outliers.

Unity Response

Within the day of the announcement, there’s a lot of angry people, and Unity has tried to clarify the points raised. However, it’s not clear if it’s actually possible to do what they claim. They reckon they have some sophisticated fraud detection technology which can prevent the “install bombing”. Then they say that they will have a process for them to submit their concerns to our fraud compliance team. So from what I understand here it sounds like the onus will be on the developer to try and somehow keep track of how many of their instals are fraudulent and then if you have concerns, you contact the fraud compliance team, and then they will hopefully give you your money back. I think the majority of people don’t have a lot of faith in such a system when Unity have to put in some work to decide if they want less money from you.

https://x.com/thomasbrushdev/status/1702797688838775134?s=20

Unity have clarified that if you’re part of a bundle like Xbox game pass or you’re in a charity bundle then you’re not going to be charged for the install, although it’s not exactly clear how they’re going to know which instals come from charity bundles or game passes. They seemed to imply that for Game Pass, they would send the bill to Microsoft but I can’t imagine Microsoft will be too happy to have sprung upon them. It would probably have to be negotiated in future Game Pass deals and it might just be the case that Microsoft just doesn’t add any Unity-based games to their service.

Unity tried to justify this whole new fee structure by pointing to the thresholds and saying “if you don’t already earn loads of money on your game then you’re not gonna pay extra”. This is where I think a lot of developers are wrongfully attacking Unity, when they would never pay them anyway. I suppose in the Patch Quest example, I’m not aware of it being a major hit, and he has pretty much reached the payment threshold. But given that it’s been many months after release, you would imagine sales will now be low and he will only be liable for minor fees which he should be happy to pay.

Conclusion

There probably is a clause somewhere deep in Unity’s terms and conditions that says something like “we retain the right to change our terms and conditions”. Companies love to have that kind of future-proofing in their legal small print, but how many actually go through with major changes? It can be logistically difficult to implement drastic changes, and evidently a PR nightmare. However, despite that, many companies are against Unity for switching the Terms and Conditions with only a few months notice. When games can take years to make, you need that predictability to adequately budget, and if Unity can charge you more on a whim, then it’s unpredictable. People also wonder if they really can change the terms built on an older Unity software version as you essentially have an agreement at the time of release; but that needs to be left to the lawyers.

Switch Engines?

I think a key statement that many are using to justify their decision to abandon Unity at this time is “Is this the last time they’re gonna change their terms? 

Jumping ship to another one might be possible when you’re just starting up on a new project but the deeper into development you get, the harder this becomes. Your game gradually ends up dependent on the engine it’s built in. Switching to Unreal Engine will require programming in C++ plus instead of C# which is a massive learning curve. Godot seems to be gaining popularity but people seem to say it specialises in 2D games at the moment. I think C# doesn’t have full support so their own GDScript is more popular.   

https://x.com/TruantPixel/status/1702132911976194091

https://x.com/DarkestDungeon/status/1702378602895941837

References:

Unity Pricing Thoughts...
 
For context, we are a small studio (7 people) with a Steam game with 3M~ players.
 
I'm seeing many non-developers tell developers that this pricing change is not a big deal, here is why the entire community is lighting a fire:
• Massively disproportionately punishes indies
• Only three months notice
• Double dipping (Licence fee/ads cut)
• Dangerous precedence for charging "runtime"; you no longer fully own that exported build. If Unity continues to struggle, pricing could become more aggressive
Here are a few examples:
• Unity's own example on their site has a hypothetical scenario:
-- $2M USD Gross in 12 Months
-- 300k Users/month (200k Standard/100k "Emerging Market"), $23.5K USD/month
-- This means $282K/Year in fees, 14% of gross revenue, 3x Epic's 5%.
• F2P Games that are NOT excessively monetised are penalised:
10M Players:$1M USD
1M Players:$10M USD
The first case, with a vastly less predatory set of MTX is now punished significantly worse than one purposefully building money-extraction machines.
Our team has been hard at work for 2 years on a massive update to our game, with a F2P mobile ver coming next year. We built this from the ground up to be ethically monetised/for whaling to be impossible, so we are particularly unhappy with the news.
This affects developers everywhere, of all sizes. I am grossly disappointed by any industry figures brushing this off as "developers complaining." that do not understand the severe damage this can cause smaller studios.
Unity's trust within the games industry has been steadily eroding for years now, this latest change is a testimony to how horrendously mismanaged the board is. Personally dumped all of my Unity stock after this announcement was made.
I'd bet heavily on the people making these decisions have never even opened the editor, let alone released a game.
 
From <https://threadreaderapp.com/thread/1702189840383832408.html>