Changes to the Software Delivery Process

One of the problems we have where I work – is not releasing fast enough. When you read about Software Development, you hear of these companies that can release minor updates every week. Maybe it is more of a Web Development thing rather than an Application Development one, but there are also contractual reasons why we cannot release faster.

However, over time, the release time has continuously crept up to 4-6 weeks which causes problems.

If there is more time for each release, it means that the scope of the current release often increases further. For example, if there is a fix that needs to go out within 3 weeks to meet the SLA (Service Level Agreement) and the next release will go out in 4 weeks, then you have little choice but to get it in the current release. If you are checking it in close to the deadline, then you might end up delaying the release in order to test it. The more you delay, the more chance of someone else needing to get a fix in the current release and it grows further.

If there’s many big projects targeting the same release, each team developers in their own code “branch”, then will merge into the Main branch for release. Since it’s not really feasible to all merge at the same time, you end up taking it in turns and resolving any conflicting changes. To be honest, it’s quite rare that we will change the same files for the main feature changes, but there’s certain files with a lot of churn, mainly ones containing XML. To merge in projects, it usually takes a few days, then all the bug fixes on top. The Testers can’t really begin testing until it’s all merged so it’s a lot of overhead to manage the release.

When the releases are large, the Testers insist on running more Regression Tests which increases the Testing phase and can cause further delays.

“I think we spent about 2 months on that regression. It was madness. It was a HUUUGE release”

Software Tester

So smaller releases are much more manageable, take much less time to test, incur less risk, and have lower scope for scope-creep.

Our Software Delivery team made an announcement about this (basically just saying the same things I have just discussed), and desire to plan in quarters but release in a couple of weeks.

In the past, we would scope a single release looking at the features, fixes and minor enhancements we wished to deploy. We would follow a process of merging everything into our main release branch before undertaking testing. This was a two-phased testing approach, integration/functional testing each feature and fix, and then regression testing to ensure pre-existing functions continued to work as expected. We would then spend eight to ten weeks deploying the release through Controlled Roll Out and Customer User Acceptance Testing.
 
This approach brought with it a number of challenges. Merging everything in was time consuming, issues or blockers with one feature would slow down or block other features, regression testing was a challenge, and this also put pressure on the roll out and deployment through pushing out a number of changes in one go.
 
To try and mitigate some of these challenges, we are now adopting a strategy of breaking these large releases down into smaller updates.
 
Working in quarterly cycles we will scope what we wish to deliver over a 12 week period. Each feature will be analysed and risk assessed for size and complexity by our Engineering Leads, and have a business value determined by our Product Management and Commercial Teams.
 
Using this feedback we will then determine an order in which we wish to deliver each feature. We will then merge them into a release and test them one at a time (potentially two if both are small and low risk), before signing over to Release Management to commence deployment.
 
We will then deploy the full scope over a series of smaller releases rather than in one large release.
 
The last update in the cycle will be a maintenance release to address the backlog of prioritised service work.
 
The objective behind this approach is to have our users benefit by taking elements of the release scope earlier than they would have before, whilst also simplifying the testing approach and hopefully enabling us to push code out across the estate quicker.