Changes to the Software Delivery Process

One of the problems we have where I work – is not releasing fast enough. When you read about Software Development, you hear of these companies that can release minor updates every week. Maybe it is more of a Web Development thing rather than an Application Development one, but there are also contractual reasons why we cannot release faster.

However, over time, the release time has continuously crept up to 4-6 weeks which causes problems.

If there is more time for each release, it means that the scope of the current release often increases further. For example, if there is a fix that needs to go out within 3 weeks to meet the SLA (Service Level Agreement) and the next release will go out in 4 weeks, then you have little choice but to get it in the current release. If you are checking it in close to the deadline, then you might end up delaying the release in order to test it. The more you delay, the more chance of someone else needing to get a fix in the current release and it grows further.

If there’s many big projects targeting the same release, each team developers in their own code “branch”, then will merge into the Main branch for release. Since it’s not really feasible to all merge at the same time, you end up taking it in turns and resolving any conflicting changes. To be honest, it’s quite rare that we will change the same files for the main feature changes, but there’s certain files with a lot of churn, mainly ones containing XML. To merge in projects, it usually takes a few days, then all the bug fixes on top. The Testers can’t really begin testing until it’s all merged so it’s a lot of overhead to manage the release.

When the releases are large, the Testers insist on running more Regression Tests which increases the Testing phase and can cause further delays.

“I think we spent about 2 months on that regression. It was madness. It was a HUUUGE release”

Software Tester

So smaller releases are much more manageable, take much less time to test, incur less risk, and have lower scope for scope-creep.

Our Software Delivery team made an announcement about this (basically just saying the same things I have just discussed), and desire to plan in quarters but release in a couple of weeks.

In the past, we would scope a single release looking at the features, fixes and minor enhancements we wished to deploy. We would follow a process of merging everything into our main release branch before undertaking testing. This was a two-phased testing approach, integration/functional testing each feature and fix, and then regression testing to ensure pre-existing functions continued to work as expected. We would then spend eight to ten weeks deploying the release through Controlled Roll Out and Customer User Acceptance Testing.
 
This approach brought with it a number of challenges. Merging everything in was time consuming, issues or blockers with one feature would slow down or block other features, regression testing was a challenge, and this also put pressure on the roll out and deployment through pushing out a number of changes in one go.
 
To try and mitigate some of these challenges, we are now adopting a strategy of breaking these large releases down into smaller updates.
 
Working in quarterly cycles we will scope what we wish to deliver over a 12 week period. Each feature will be analysed and risk assessed for size and complexity by our Engineering Leads, and have a business value determined by our Product Management and Commercial Teams.
 
Using this feedback we will then determine an order in which we wish to deliver each feature. We will then merge them into a release and test them one at a time (potentially two if both are small and low risk), before signing over to Release Management to commence deployment.
 
We will then deploy the full scope over a series of smaller releases rather than in one large release.
 
The last update in the cycle will be a maintenance release to address the backlog of prioritised service work.
 
The objective behind this approach is to have our users benefit by taking elements of the release scope earlier than they would have before, whilst also simplifying the testing approach and hopefully enabling us to push code out across the estate quicker.

The Secret the Task Manager developer didn’t want you to know!

Dave Plummer, who has the Youtube channel Dave’s Garage announced on Twitter:

Big news! Someone finally noticed that if you hold down CTRL, the process list in Task Manager conveniently freezes so you can select rows without them jumping around. I did this so you could sort by CPU and other dynamic columns but then still be able to click stuff…

Dave Plummer

There’s been plenty of occasions where Task Manager rows jump around to my annoyance. Why wasn’t this a more obvious feature? Frank Krueger (who appears on Merge Conflict podcast) made the obvious point:

Don’t hide features under random key combos – undiscoverable and unmemorable UIs are user hostile. A little checkbox with the text “Pause Display” would be discoverable and you won’t have to wait 30 years for someone to find your feature.

https://x.com/praeclarum/status/1693649521375621524?s=20

Unity Runtime Fee

Intro

Unity have announced a new fee which they call the “Unity Runtime Fee” which is going to take effect in January. It  affects all Unity developers, even people that have already released their game many years ago; which has caused mass outrage among the game development community.

I think the existing model states that once you reach a threshold of revenue, you have to pay a licence fee which works out around £1500 for the year. With the new model, once you reach a similar threshold, Unity is now going to charge a fee of 20 cents every time somebody instals your game on a new device for the first time.

The threshold is $200,000, which on the face of things, 20 cents per install doesn’t sound unreasonable when they have given you a great tool to help you create your game. They need to earn money as a business and deserve some kind of cut for their service/product. According to this tweet, it looks like they are burning through money so some drastic action is probably required

https://x.com/georgebsocial/status/1702696194558816751?s=20

There’s a fair few aspects of why this is new model is complicated, but I still feel some of the anger is misplaced.

I think the whole scenario is similar to what I have written about recently where the CEO demanded we release our software weekly instead of monthly and we told him several reasons why it is technically, and legally impossible. Then later he then demands all our changes have a well-documented rollback plan, and again, we told him loads of reasons why it wasn’t possible. He still insisted and looked like a fool when it backfired and caused a few problems he thought he was solving.

The main Problem

The core problem stems from the idea that it is based on installations and not based on Unit Sales or Revenue. For comparison, the main competitor, Epic Games’ Unreal Engine charges you five percent of your total revenues after you’ve earned at least a million dollars on your game. Now, that can work out to be a lot of money and especially in the long run if your game is successful, but the difference is that they’re taking a cut of your money that you’ve already earned. When Unity charges you for an installation you’re being charged whether or not you’ve earned any money, or at different period of time to where you earned the money. That could turn into a cash flow problem.

Collage of abuse:

https://x.com/LiamSorta/status/1702325745610338646?s=20

Theoretical scenarios

Once you’re over the threshold, if somebody bought your game a long time ago and they’ve now installed your game on their brand new computer, it’s going to cost you 20 cents. I suppose if it is an old game, you probably won’t be selling $200k over the last 12 months, so it’s probably not actually going to apply.

If you decide to port your game to a new platform which is often fairly easy in the Unity engine, then all those new installations are also going to be hit with fees. I suppose if you are re-selling the game then it’s not a major problem, but sometimes developers make a free-to-play mobile version. Then you make money later with microtransactions. Often these games have 90% of players not paying a penny, but then you make your money on the 10% who often spend big. In this case, you could end up losing money on the average player of your game.

People also raised the point of bundles like the Humble Bundle where people buy a bunch of games for a small price but some of the money goes to charity. You end up selling high volumes but gain very low revenue. If you hit the threshold, and you are more likely to with a sale like this, then you could be hit with a lot of fees. I think the interesting thing with this point which people don’t seem to be mentioning; is that people often buy these games then never actually play them. So you actually have a sale, but no install, so don’t pay the fee.

Fairly similar to a bundle is a service like Xbox Game Pass, where people could play your game with an overall payment to the provider, in this case Microsoft. I think Microsoft often pays a flat fee to the publishers to gain their games but I suppose contracts can vary. But the theory is, you could get a flat fee, then either get low instals so you’ve gained, or get a surprisingly large amount of instals if it is popular and it eats into your profits.

Piracy

People who pirate games don’t pay but do install your game. This means that every time your game is pirated you’re going to be slapped with a 20 cent fee. There can be other malicious ways you could be charged, if someone abuses Virtual Machines. There’s programs that will spin up large numbers of them, so you could “Install Bomb” quickly with virtual machines, hitting the developer with a 20 cent fee. It’s like when people “Review Bomb” where you leave loads of negative reviews on a game you don’t  like in a coordinated way, but in this case you need fewer people, and they directly sap the revenues of the developer instead of just hurting their online presence.

Target Price

Unity has always positioned itself as being pro Indie. They want to help new aspiring Indies learn to program, break into the gaming market, and get their career started. New developers are also much more likely to sell their games for cheap. There’s a lot of games like this on Steam which are sold for £10 or loads for £5 or less, and that’s before you apply discounts. Steam is renowned for its high discounts in sales, and so these games are being sold for just a couple of pounds. They’re going to be disproportionately hit by having to pay Unity 20 cents every time the game is installed.

In the extreme case, imagine you’ve made a Steam game or a mobile game that sells for one dollar and then you pay a sales tax of 10-20%, then Steam takes 30%, then you know you’re left with around 50 cents. If you use a Publisher, then they will take their cut too. Then Unity takes 20 cents of it for an install and then maybe another 20 cents for another install, then you could be left with basically nothing. You could then lose money if it isn’t sold for full price.

Meanwhile, if you sell a premium game for £40+ then 20 cents is nothing. So it actually hits the indies harder. Unity have ways of getting the price per install down, but they look more aimed at larger companies who will want to pay the upfront fees to use the premium Unity features.

Patch Quest

Lychee Game Labs’ Patch Quest released on 2 March 2023 and so far has reached 182,594 total key activations on Steam (people who bought the game on Steam along with everyone who got the game elsewhere like in a bundle or a giveaway or for review purposes). So if the game keeps selling, or people install on more devices, then he will be taken over the threshold then would start being charged. He did remark that “for the sake of argument, every single person who already owns the game decided to install it on a second PC, I’d be hit with a charge of $36,400. Now it’s obviously not likely that this would happen” but it does make you think how Unity are gonna deal with these outliers.

Unity Response

Within the day of the announcement, there’s a lot of angry people, and Unity has tried to clarify the points raised. However, it’s not clear if it’s actually possible to do what they claim. They reckon they have some sophisticated fraud detection technology which can prevent the “install bombing”. Then they say that they will have a process for them to submit their concerns to our fraud compliance team. So from what I understand here it sounds like the onus will be on the developer to try and somehow keep track of how many of their instals are fraudulent and then if you have concerns, you contact the fraud compliance team, and then they will hopefully give you your money back. I think the majority of people don’t have a lot of faith in such a system when Unity have to put in some work to decide if they want less money from you.

https://x.com/thomasbrushdev/status/1702797688838775134?s=20

Unity have clarified that if you’re part of a bundle like Xbox game pass or you’re in a charity bundle then you’re not going to be charged for the install, although it’s not exactly clear how they’re going to know which instals come from charity bundles or game passes. They seemed to imply that for Game Pass, they would send the bill to Microsoft but I can’t imagine Microsoft will be too happy to have sprung upon them. It would probably have to be negotiated in future Game Pass deals and it might just be the case that Microsoft just doesn’t add any Unity-based games to their service.

Unity tried to justify this whole new fee structure by pointing to the thresholds and saying “if you don’t already earn loads of money on your game then you’re not gonna pay extra”. This is where I think a lot of developers are wrongfully attacking Unity, when they would never pay them anyway. I suppose in the Patch Quest example, I’m not aware of it being a major hit, and he has pretty much reached the payment threshold. But given that it’s been many months after release, you would imagine sales will now be low and he will only be liable for minor fees which he should be happy to pay.

Conclusion

There probably is a clause somewhere deep in Unity’s terms and conditions that says something like “we retain the right to change our terms and conditions”. Companies love to have that kind of future-proofing in their legal small print, but how many actually go through with major changes? It can be logistically difficult to implement drastic changes, and evidently a PR nightmare. However, despite that, many companies are against Unity for switching the Terms and Conditions with only a few months notice. When games can take years to make, you need that predictability to adequately budget, and if Unity can charge you more on a whim, then it’s unpredictable. People also wonder if they really can change the terms built on an older Unity software version as you essentially have an agreement at the time of release; but that needs to be left to the lawyers.

Switch Engines?

I think a key statement that many are using to justify their decision to abandon Unity at this time is “Is this the last time they’re gonna change their terms? 

Jumping ship to another one might be possible when you’re just starting up on a new project but the deeper into development you get, the harder this becomes. Your game gradually ends up dependent on the engine it’s built in. Switching to Unreal Engine will require programming in C++ plus instead of C# which is a massive learning curve. Godot seems to be gaining popularity but people seem to say it specialises in 2D games at the moment. I think C# doesn’t have full support so their own GDScript is more popular.   

https://x.com/TruantPixel/status/1702132911976194091

https://x.com/DarkestDungeon/status/1702378602895941837

References:

Unity Pricing Thoughts...
 
For context, we are a small studio (7 people) with a Steam game with 3M~ players.
 
I'm seeing many non-developers tell developers that this pricing change is not a big deal, here is why the entire community is lighting a fire:
• Massively disproportionately punishes indies
• Only three months notice
• Double dipping (Licence fee/ads cut)
• Dangerous precedence for charging "runtime"; you no longer fully own that exported build. If Unity continues to struggle, pricing could become more aggressive
Here are a few examples:
• Unity's own example on their site has a hypothetical scenario:
-- $2M USD Gross in 12 Months
-- 300k Users/month (200k Standard/100k "Emerging Market"), $23.5K USD/month
-- This means $282K/Year in fees, 14% of gross revenue, 3x Epic's 5%.
• F2P Games that are NOT excessively monetised are penalised:
10M Players:$1M USD
1M Players:$10M USD
The first case, with a vastly less predatory set of MTX is now punished significantly worse than one purposefully building money-extraction machines.
Our team has been hard at work for 2 years on a massive update to our game, with a F2P mobile ver coming next year. We built this from the ground up to be ethically monetised/for whaling to be impossible, so we are particularly unhappy with the news.
This affects developers everywhere, of all sizes. I am grossly disappointed by any industry figures brushing this off as "developers complaining." that do not understand the severe damage this can cause smaller studios.
Unity's trust within the games industry has been steadily eroding for years now, this latest change is a testimony to how horrendously mismanaged the board is. Personally dumped all of my Unity stock after this announcement was made.
I'd bet heavily on the people making these decisions have never even opened the editor, let alone released a game.
 
From <https://threadreaderapp.com/thread/1702189840383832408.html>

Rockstar Games cracked sales

Blog Based on Modern Vintage Gamer’s video:

Rockstar Games created big franchises like Grand Theft Auto, Red Dead Redemption, Max Payne, and other games like Bully and Manhunt.

Rockstar are quite protective of their Intellectual Property, and will take game modders to court, but were recently caught selling their own games with a cracked executable created by software cracking groups “Razer 1911” and “Myth”. The cracked executable files remove anti-piracy DRM (Digital Rights Management) checks from the game. On analysis of the executable file, you can see the group’s logo in there.

Rockstar had apparently placed the games on storefronts like Steam with this cracked file for their games Manhunt, Max Payne 2, and Midnight Club 2. Ubisoft has also been caught using cracked executables as well for Rainbow Six Vegas 2.

These software cracking groups would work on “No CD cracks” for many games, which would remove the requirement of having the CD-ROM in the disc tray. It would normally be a requirement to check if you have a valid copy of the game, otherwise you could just fully install the game and return it/sell it/let your friend borrow it.

These groups could also bundle the cracked file along with the full game to be illegally downloaded and shared, which meant piracy was rife in the PC market. They obviously didn’t have the aim for software preservation, but it seems they have provided that role in the current day.

Not only do many modern PC’s not have disc trays and are digital only, but some of the other DRM methods are obsolete. This was also a problem years ago when Microsoft removed the “Games For Windows” DRM, so you had to try workarounds to bypass the checks. People that legitimately own the games can struggle to play them on modern hardware.

The alternative for game publishers is to find the old source code, rebuild without the DRM checks, then publish this on a digital store. It’s not as simple as that though, because many game’s source code goes missing over time and as game companies go bust, or be taken over. To even build the game in old tools could be problematic if you need to install and run other software which itself could be hard to locate or get running. If successful, they would need to dedicate a small team to develop and test the product. It’s probably just easier to “illegally” download their own game along with the crack file, then sell it.

It raises an interesting dilemma. Since the cracking groups have performed some work, should they get credit and maybe even receive payment? Even if the likes of Rockstar take criticism for their actions, they can probably simply ignore the criticism and keep on doing it for any old game they want to resell.