I work at a software company whose flagship product (which I’ll refer to as SystemNow) has been around for maybe 20 years now. However, even way back in 2016, the software architects began a project to look into what the next software would look like (ProjectFutures), and development really started ramping up around late 2018/early 2019.
I reckon it took around 5 years of development for SystemNow to be initially released (then continuously enhanced and fixed), so maybe you would think 2023 could have been a good release date for ProjectFutures – its core features at least.
However, in 2025, as I began drafting out this blog. Despite the claim from managers that ProjectFutures has been “released” and is a “success”; in my opinion, it’s like 10% of the scope, and barely been spoken about. 2025-26 has seen little progress.
I think it’s been an absolute disaster and I do wonder where it stands among other high profile software disasters. So I thought I’d reminisce and go back through the many poor decisions and failures along the way. You could say there’s all kinds of failures, and you probably can’t point your finger at any one aspect/decision, but many intersect.
Why Rewrite It?
When we have a very successful software product, then why rewrite it? I had a quote from Cory House who explains some of the reasons for the initial drive:
“Each year, I work with a few teams that are basically doing the same thing: Rewriting old web apps to use React. The old apps use ASP NET Web Forms, JSP, jQuery, Cold Fusion, Perl, etc. Often, the requirements are simple: “Make it work like the old one”.
This might seem silly. If it ain’t broke, don’t fix it, right? But there are good reasons:
They’re having a hard time hiring and keeping good help.
It’s harder for them to add new modern features compared to modern technology.
The old tech isn’t actively supported.”
Each year, I work with a few teams that are basically doing the same thing: Rewriting old web apps to use React.
The old apps use ASP NET Web Forms, JSP, jQuery, Cold Fusion, Perl, etc.
Often, the requirements are simple: “Make it work like the old one.”
Internally, I’ve heard various of our Software Developers complain about “Technical Debt” (or the perceived amount of it); which was the excuse for slowly getting features out, or an excuse for introducing more bugs as we fixed other bugs. Common excuses like “this code is messy and hard to read or change”, “there’s a lack of tests so if I change it, some requirement may break which I didn’t know about”. The “old tech” was often criticised for contributing to a slow deployment process. We did lose a few developers who wanted to go work on modern web technology using Cloud and other buzzwords.
So for a company with previous experience making a market-leading software product, and with a large team of passionate developers who think they know what great code looks like; it should be pretty easy to deliver, right?
Failure Reasons
Reason 1: Skills in the wrong areas
SystemNow was C# front end and SQL Server backend. The vision for ProjectFutures was that it would be React (Javascript) frontend, then using various Cloud aspects (Lamdas, API, DynamoDB). Some developers have the capacity to learn quickly and could instantly claim they are a Senior in the new language. However, most people basically go from a Senior in C# to Junior in React. If most people are Junior, then you end up writing code that is full of technical debt and so you cause the exact same problem you criticised the old SystemNow code for. (I think this was proven by the restarting or cancelling projects as explained later).
As a similar point, we also hired people too quickly. We have severely reduced hiring people in the UK and gone for hiring in India. Aside from some being poor communicators (and then having Indian English grammar and phrases which people have to get used to; (ie “do the needful”, “preponing meetings”); being Indian isn’t inherently bad. The problem is that as a business we have wanted to hire large amounts at once; which then leads to an “accept all” type of hiring. In addition to wanting to pay low wages as well; you end up attracting people that can’t get a job elsewhere. So you end up getting loads of Junior developers all starting at once, some with no experience of making software at all, and some only having basic understanding in a programming language we aren’t even using e.g. Python. There were some good hires but I think it would be better just to go with far fewer staff but higher quality engineers that can really drive the decision making.
Reason 2: The Cloud
The current way of working is that the developers created software, then gave the build to the Deployment team to release. The clear separation means you can specialise your knowledge.
This mindset to move to “the cloud” means that now the Developers have to have more understanding how their code is actually deployed.
Historically, over time, specialist “Frontend” developers and specialist “Backend” developers were replaced with more generalist “Full Stack” Developers. Now you are throwing servers and deployment processes into the mix (DevOps), so now it’s like “Full Stack plus DevOps”.
Due to this, you spend more time learning, more time investigating, and giving worse output because you don’t specialise in it.
The world of cloud computing is full of buzzwords, especially if you use multiple providers (Azure vs AWS naming is completely different). This is extremely daunting and gives you a massive list of things to learn before you can even begin to communicate with people.
No idea where I stole this from, but I think it sums up the attitude that managers had:
The company’s technical leadership thrives on keeping tabs on the rapidly growing technology industry and new innovations built on top of fully-managed services provided by cloud providers. Like many other technical leaders, they are keen on learning new buzzwords and leaving it up to their developers to do something useful with them. The hottest trend these days seems to be ‘serverless’. The promise of consumption-based pricing, where you only pay for what you use and nothing more, is enticing. Furthermore, they have heard many good things about how serverless platforms help you prototype and develop faster by reducing the amount of code and configuration required by more traditional web platforms. The cherry on top is the fact that serverless platforms also tend to manage scaling resources to meet demand automatically, giving them dreams of releasing a wildly popular new ride share app and enjoying near-instantaneous customer growth.
We were a massive company and had our own Data Centres. I think Cloud is vastly beneficial to small companies who cannot invest in the up-front cost of their own server room, but also can easily scale up if demand drastically increases. We already had the resources for large demand but then moved to the Cloud just for the sake. I think managers were so obsessed with this “serverless” buzzword that they demanded it was the focus, rather than designing a solution with an open-mind, and using what was necessary to achieve the objective.
Reason 3: Fail Fast, the problematic developer, and working on tools/process
When we begun development on ProjectFutures, managers kept on telling us that a “culture of innovation” means “there is no such thing as failure”: You can try new things and as long as you feel like you have learnt, then it’s not a bad thing because failure then leads to you doing the correct thing.
This flexibility of being able to choose our tools/approach was taken a bit too literally, or extremely. So you had multiple teams investigating the same ideas. You also had teams successfully implementing something, then rewriting it using something else just to compare.
There was one problematic Principal Developer that they hired called Liam who seemed to have massive influence. Liam always persuaded people to investigate a different tool to what we were currently using. Then once people had made progress, he would then recommend something else. He wasted so much time. He’d end up arguing his case using some strange analogies too.
They are the same as chocolate bar wrappers. You need it for the delivery of chocolate, but once used, you get rid, otherwise it’s just messy 😉
Also I see a load of chocolate bar wrappers on a desk and I instantly check them all for chocolate.. Friends don’t let friends do this.
He’d often cause conflict, switch teams, then repeat his behaviour of changing all the tools and processes, then moving on once more. Instead of just cracking on and actually writing the software, we wasted months, even years switching between different tools which then got abandoned.
“these guys makes me wonder if I can even call myself a developer if I just don’t care about processes and tooling. To me right, I consider us to be like master carpenters building beautiful cabinets. And these guys are having a massive debate about the lockup out the back where we keep the wood. Another problem is people just getting excited over every different bit of tech
grunt
WTF IS GRUNT?????????
gulp too
I’m sure some of these are being made up just to piss me off. I bet a lot of people can’t even remember how to code any more. All they can do now is manage Nuget packages”
Dean (Senior Developer)
I’m sure most developers wouldn’t actually care how their code is deployed. Just that it is deployed and is easy to sort when things go awry. Most teams were using GitHub Actions, but Liam demanded teams he worked with switch to Jenkins, and even convinced many managers that this should eventually be pushed out to all teams. However, as teams began to switch over, managers realised that Jenkin’s self-hosted nature was costly in resources, and comes with even more overhead of actually maintaining its use. So then they had a change of plan.
“The current Jenkins solution is estimated to cost $2,126.83 a month just in AWS resources.”
“(By not using Jenkins) we can become a leaner outfit that can deliver value to the business quicker by focusing on our own strengths which is delivering software, not CI/CD software”
After exploring different options, they realised that GitHub Actions made the most sense to use after all because our code is in GitHub so there’s no cross-account security issues to worry about, and we already had some free-build minutes in our subscription.
“So far we’ve been told to do ADO -> AWS CodePipeline -> Jenkins -> ADO. So we’re excited to try Github Actions next!” – Lead Developer
A month later, the manager who announced we should be using GitHub Actions then said his team had been exploring using Netlify for his next project. Facepalm.
When the problematic Developer Liam eventually did quit the business (or we think forced out), one manager seemed hell-bent on undoing everything he implemented. Another idea that Liam had was a Tech Radar. It was basically a decision log of sorts, listing all the software/tools we have tried, the pros and cons and why they were chosen/declined. If other teams looked at it and maintained it, then it should mean they don’t waste time exploring using tools that had already had a decision made.This was also binned off due to:
“The Tech Radar required a significant amount more administration and the rules around reviewing were not sustainable, we were not able to commit to the reviews when it was in full flow.”
Reason 4: Using other tools are expensive
If you asked me to come up with a plan to remake SystemNow, or similar existing large-scale complex software, I would try to limit the initial scope by reusing as much of it as possible. So you could just create a new user interface, but when it comes to saving the data, you could still call your current/old servers. So we could have replaced the UI to be a website using React, but still use our existing C# server code and still keep the SQL Server database.
I’d also look to basically outsource part of it too. If there is a company that specialises in something, it makes sense to pay them to do it, rather than making an inferior version yourself, and having the overhead of maintaining it.
You can always decide to implement these yourself later if you have no other projects to do.
If you try to replace the entire system yourself, it will extend the timescales further and you run the risk of failing to deliver at all.
To me, authentication is an ideal candidate for this. Originally we used Okta, but then a few months later, a Software Architect decided to create a project to make our own authentication. So that’s several developers spending months of their time making their own security solution, and none of them are security experts. What can go wrong?
The reason I heard for this quick switch is that Okta was far too expensive. It was projected to cost around £15k per month to use. This sounds mad. However, when I mentioned this to one of our Architects, he said “that equates to around 1p per user per month”. I suppose when you look at it like that, then it does sound cheap.
I don’t know how much we have truly saved when we have to pay several developers for the length of time it took to build the solution. It’s not that simple though because you have to then hope that it works, and hope we never get fined for any security issue. Also there is the “opportunity cost” that they could have been working on some actual functionality and not just allowing the user to log-in. Surely it wasn’t a priority at that moment because we had Okta working and only Developers were using it to log in as we developed.
UI Suite
Another thing we made in-house was the “UI Suite”. So this refers to the basic components like text boxes, combo boxes, buttons, icons, and the styling. Now, what is the cost of using a third-party library? Well most of them are free, and it took YEARS for us to create something decent. Every time I heard about a basic bug we introduced; I was just facepalming since we wouldn’t have had any problems if we just used MaterialUI.
As the UI Suite was being built, teams were reluctant to use it because it barely had any components. It really needed to be feature-complete by the time any teams actually started their projects. When components did exist, the bugs slowed down all development teams which caused some teams to use something else as a stop-gap. But without many teams using it, then that stifles adoption by other teams. Whose idea was the “UI Suite” anyway? Yeah it was Liam’s again!
Within a few days of me using it, I reported that the combo box did not allow you to select an item, and the text box didn’t allow you to change text programmatically. Another team reported that the Table component didn’t display unless you manually called the Resize method as a workaround. The Table also was a fixed size due to a hardcoded value. Then, once that was fixed, the column headers were not resizing correctly. Another team reported that the List component’s itemSelected property didn’t work when set programmatically.
To me, it’s inexcusable that a combo box won’t allow you to select an item. With severe bugs like that, it just slows down your development. Then it’s the opportunity cost that the team could be assigned to a meaningful project, and we could have been using a popular UX library like Material UI that we know works.
On a Development Department Meeting, one angry developer asked:
“Is having a UI Suite team a pragmatic allocation of an entire team’s knowledge and resources?”
This question then triggered a 30 min meeting the following week to try to justify why the UI Suite was important. It was full of blagging from the manager because no one really could think it was a good idea. In my opinion; if you have to create a presentation after 3 years into a project to convince people it is a good idea – it probably isn’t a good idea. He acknowledged there was “growing concern” across the department that people “didn’t think it was a productive use of our developers and there are already ready-made solutions available”.
I’m not even sure the team that was making it believed they could actually complete it. 2 members had left which meant they were down to: an experienced Developer, another Developer who had spent most of his career in UX design (drawing not coding), and a Junior Developer who had just began her year off for maternity leave. Then they had a Product Owner, Technical Manager, then a Manager overseeing it which is inflating the costs and bureaucracy.
His main slides stated the following:
Why UI Suite? Why not? We're not all the same. • There are several design systems out there already such as Bootstrap and Material - They all have their pros and cons • Someone else's idea of a design system may not always work for us; having our system gives us the flexibility to define our own style • Creative freedom to ensure our products stand-out from the rest, and are easy to use • Large organisations such as the NHS, Government, Atlassian and Uber have their own design systems because it is right for them.
“is that right for us? eeeeerm…………………..”
manager struggling to blag
My View: I think he is blagging justifications. MaterialUI has been around for years and you can apply your own theme so I don’t understand this “design system” nonsense. You could also make tweaks to it because it is open source so could deviate if there’s some things you don’t like.
He then implied they were having a reboot of sorts and changing direction, but I’m not sure what that entailed. But if the UI kit isn’t ready, then how can people develop their apps effectively?
As more teams used it, the backlog of requests grew, and other teams were asked to fix bugs themselves. The claim was that the project was always intended to be “open inner source” and all teams needed to contribute. In September 2024, the UI Suite team disbanded and now every single new feature or bug fix needs to be implemented by teams that desire it (or you could just abandon it and use Material UI). The messaging from the managers was that it was a ”lack of shared ownership” why UI Suite failed in its current form.
“This change is made with the best intentions… It’s also just the start of a journey. As we learn and identify challenges, we will adapt and improve the approach.”
More recently, I did notice that some of the components were now simple tweaks of another 3rd party UI library so I think they had binned off their original implementations in their last reboot. Now they are on a second reboot. “ It’s also just the start of a journey” What a waste of 5-6 years.
Reason 5: Deprecating unreleased functionality
Due to the idea of “microservices”, people went wild creating repositories for everything. It does make sense to have general shared functionality like “logging” for error logging which everyone can use. Some of the projects ended up consisting of multiple repositories because they took the idea too extreme. With the sheer volume of repositories under our organisation, it became difficult to see what was actually useful. Just like the UI Suite, we had rebooted projects multiple times so then there were a few repositories for “logging”. You’d see similar names like “logging-sdk”, “logging-sdk-typescript”, “logger”, “monitoring”. Did one replace the other? Are they for different scenarios? Are they poorly named?
Occasionally, someone would announce something like the following and you do wonder who was using it and why no one had deleted it earlier:
“Hello, the client-logging package exists within the logging-sdk-typescript. It hasn’t been updated in almost two years, and more importantly it offers no functionality to store logs so it’s not fit for purpose. This client-logging package was meant as a proof of concept. We have no more need of it and all it offers is a false sense of security, so we will be deleting this package at the end of the month”
Intermission: Q4 2021 – Q2 2022 – Hyping Up Basic Features: Powered By ProjectFutures (and the framework reboot)
“This week we activated <new feature> which is a new web based application, powered by ProjectFutures cloud technology, and integrated into SystemNow.”
“Powered by ProjectFutures” sounds like some jargon to sell to customers, but was repeated by the managers internally. It was like we were gaslit by what our current progress was. There was nothing noteworthy at this point. I got sick of hearing it and we all knew it didn’t mean anything. Even when they didn’t use that phrase, it was just the usual corporate buzzwords and pretentious tech jargon:
“The teams working in this Application Composition Platform space are doing some amazing, innovative work focused on future technology that will allow us to respond rapidly to a changing market. Keep coming back to see how this develops and please ask any questions you might have.”
Check out this guff:
“I’ve worked here for nearly 20 years and have played a part in, or witnessed, a number of momentous achievements during that time. I’ve seen us adapt to a constantly changing market, grow as a business and continuously offer new and exciting challenges for our people. However, we have often struggled to move at the pace our users demand and this becomes more of a competitive challenge as new players move into the market. We have to continue evolving and never allow ourselves to become complacent about our position in the market.
Technology is a major enabler in allowing us to achieve business agility, giving us the ability to rapidly respond to market changes and user demand. We have recently taken an important step forward in using modern tech to our advantage with the first release of the Application Composition Platform (ACP), powered by ProjectFutures.
The ACP is a suite of technology enablers that allow us to rapidly build high quality cloud hosted web applications in a consistent design system. It is essentially a lego kit that gives Product, UX and Development the tools to quickly go from idea to design to implementation and then getting it into our users’ hands. It allows us to provide apps that are standalone or integrated/embedded within our existing product suite
Our first release provides a simple view of the SystemNow news feed in a standalone web app. This has been released to two sites so far and the rollout will continue once we've had a bit of feedback. Although this feature isn’t ground breaking, it allows us to prove the technology and confidently move forward with rapid app development. There are a number of dependencies to overcome, so that apps created in the ACP can easily access data through consistent APIs and this work is going on alongside development of the ACP.
This is the beginning of a journey to modernise our technology stack and take us to the next level of business agility. I will be providing further details this month on the next plans for the ACP during Q4 and into 2022. I have an incredibly talented team working in the app development space and I’m excited about what we can achieve.
So what does all this jargon mean?
When I worked on ProjectFutures during 2019, I was working on what we called the Application Shell. It seemed that this was rebooted and renamed the Application Composition Platform (ACP), (powered by ProjectFutures, of course). All it was back then was basically a Home Page with some “Context” so it could access the User Token to see what modules the user was allowed to access, and would populate the menu with the apps they could use. So a simple explanation was: it was just the basic framework to embed other applications in.
The original idea was ProjectFutures was a separate application that only really would be used once all the major modules were available (what we were calling “the Big Bang”; one day the user would just completely switch over and stop using SystemNow). Now the plan is that ProjectFutures is a “Companion” app, so will run alongside SystemNow. We also have the option of embedding it inside SystemNow. Therefore if we actually make an amazing module, we can replace SystemNow’s version. So now it’s a gradual phasing out.
Note: In the intro, I said the core modules should be out by 2023 . At this point they have quickly pushed out an RSS news feed by January 2022.
Dean Did they scrap your framework and do it again?
Me: I think they scrapped everything we did. We called it Application Shell
Dean: If you put it to your ear you can hear the distant sounds of developers crying.
In Summer 2022, our CEO was hyping up ProjectFutures to our customers. She was really fixated on the concepts of Cloud, and that it’s not a “big bang” approach. So she announced there is no longer a brand new product, we will now just migrate the existing product “module by module”. The phrase “Powered By ProjectFutures” was going strong. It’s still just a meaningless buzzword because the “technology” is just AWS. We haven’t got any innovative proprietary technology at all, but we just have to pretend. We do have the RSS feed though.
“We’ve got an interesting and enjoyable journey with ProjectFutures which has evolved over the last 2-3 years. The Key to ProjectFutures strategy is moving systems into the cloud but we will avoid disruption by doing things in a gradual way.”
Our CEO made an announcement which was available to view on YouTube. The gist was that COVID had created an extra demand on certain services which required companies to increase innovation in technology and processes. This change had to happen rapidly. This is evidence that software changes in today’s world need to happen faster. So actual development time needs to come down, but also delivery time to get the change to the end user.
“Companies had seen time scales that would have previously been thought impossible”
Most of it was nonsense, like she claimed we had evolved our existing software to use ProjectFutures’s “Cloud Infrastructure”.
“So now we are “using the latest, trusted secure technology… we are building new modular components powered by ProjectFutures to enhance existing applications in the cloud that can support improved performance and enhancements….our development to release cycle is reduced so we can bring innovation to our customers faster. We no longer need to have a big bang approach. Instead we can build cloud modules that will interface with our existing systems and also with other supplier systems using industry standards retaining the familiar look and feel and the workflows for our users but at the same time being able to deliver additional functionality without impacting performance. We can reduce the impact of change on our customers releasing enhancements in an incremental way rather than building the entire new systems”
After that, customers will surely expect the Companion (ACP) to be populated with features pretty consistently, right? I mean, we are bringing innovation to our customers faster, and delivering in an incremental way.
Q4 – 2022
The Internationalisation project for SystemNow was a massive one, lasting a couple of years. They had already assigned more people to it, putting on hold other projects for it. Then around October, they had temporarily assigned developers to do code reviews and help out on various tasks to get it complete. However, a few months later, it was announced to be scrapped.
Instead ProjectFutures was the focus, and would initially target England only. Then the focus will be then on Internationalisation once more, but they would get ProjectFutures instead.
Since SystemNow was the system bringing in money for the company, I imagine cancelling the contract was a massive loss. Surely there are financial penalties for the cancellation, then the loss of any future sales in other regions, then it was costly in terms of the number of developers working across all those years. They have really put faith in ProjectFutures bringing in the money years down the line.
Reason 6: The Big Restart: X2
In addition to the cancellation of the Internationalisation Project, there was some shuffling around with managers and teams which was announced in a Development Department meeting. You’d think managers would be sacked rather than moved around because surely only having a News Feed after 4 years is a shambles.
I do wonder if managers just blag each other. On the Department meeting, the newly promoted ProjectFutures director said “subsequent difficult market challenges that have led to attrition globally, this has led to delays in delivering the solution”. What is he on about? ProjectFutures hasn’t been impacted by any outside influences at all.
He then explained a new plan. It sounded like all work so far would be rebooted under a new internal moniker “ProjectFutures2”.
On the roadmap, launching straight away is a ProjectFutures product which is actually from a company we bought out last year, but we are slapping our branding on it so we can say it is Powered by ProjectFutures. Then there will be 3 small features embedded into SystemNow which are also Powered by ProjectFutures.
In 2023, we will have 4 more inside SystemNow, as well as one that will actually run in the ProjectFutures framework alongside the News Feed; Instant Messaging.
In 2024, 3 massive core features will be delivered as well as some secondary modules.
By the end of 2025 we will have the remaining modules completed. So the start of 2026 will have the full system launch for new customers.
Why will we succeed this time? - Commercial driver to get it done - Resourcing appropriately - Focus on successful onboarding - Well defined roadmap and features - User research and design - Mature technology choices - Strong architecture input - Proven approach with ACP and Companion
He said that with the extra Developers from the Internationalisation project plus a few new hires, there will be 134 people assigned to ProjectFutures.
“We also continued to work on our core UI capability which gives teams a consistent, modern suite of components built in React. These form the lego blocks of every app we’ve developed, and the integration between our UX designers and the team who looks after UI has been phenomenal” (as previously described, this “phenomenal” team disbanded in September 2024).
Reason 7: LATENCY IS NOT A PROBLEM Q3 2023
The original plan was that ProjectFutures would have its own data storage, but the obvious problem is “how do we get users to switch over to the new system?” There will be a period of time when they have to run them side-by-side in a training period and data needs to be available in both.
Since we then had the philosophy of being able to migrate modules independently, we really needed ProjectFutures to write into the SystemNow database.
I don’t understand the complexities involved, because using the same database sounds like a simple solution. However, the Database Experts then came up with the idea that ProjectFutures will write to the Primary database, but then that data is then copied to a read-only duplicate called a “Secondary”.
When teams were developing and testing on internal development environments; it all seemed to work fine. They would update data and see it reflected straight away in the User Interface. However, when they tried it on a demo site, they found that the data didn’t update until you manually refreshed it. What was happening is that after the update (which goes into the Primary database), ProjectFutures was then requesting the new data from the readable Secondary database, but it hadn’t synced with the Primary database yet; so then retrieved the old data.
So the solution was to make sure that every page had a refresh button and the user had to click refresh after they saved any data. It should be available a second later, so there shouldn’t be the need to keep refreshing the same page.
What makes this situation funny is that ProjectFutures is supposed to be an improvement in every way, and not having the current data when the user has just changed it locally is a massive step backwards. Another reason it’s funny is that I was asking a Software Architect about this “readable Secondary” idea when they announced it because it sounded problematic to me. He said he was arguing with the other Architects about it because the latency would be an issue, and he was repeatedly told that the delay is only a few milliseconds and so wouldn’t be a problem.
Missing features and various quirks just keep adding up to make the new modules inferior to the current solutions.
Reason 8: Not prioritising the right projects
Due to the restarts, it means the basic prerequisites aren’t there ahead of time, so you end up needing teams to do work almost in parallel.
One significant module was released by August 2023. In the internal announcement, the Development Director said that 11 teams were involved, alongside auxiliary staff in UX, Product and Architecture. Changes were needed in the Users API, SystemNow, Tracking API, Person API, Product API, Documents API, ACP, and UI Suite.
The way I see it, the ACP changes should already be complete, and be linked with a working Users and Person API. The UI Suite controls should be available and working. The Tracking (monitoring and auditing) is another fundamental module that should already be available.
Due to delays, restarts, and prioritising modules that aren’t important, you then get chaotic development to deliver one module. Hopefully now that the work is done, then other modules should be able to be delivered without as many changes to other aspects.
We’d barely got a few modules out before managers were then talking about planning brand new functionality. A couple of teams had been assigned to research how AI could be used in our products. Speech to text is cool, right? How about predicting what tasks you need to complete next and drafting them out?
Why would we be working on brand new features rather than converting all the existing modules over? Because “AI” is the cool buzzword.
Isn’t it just “shiny object syndrome” again? “oooh Jenkins let’s use that”
Reason 9: The Load Q1 2024
One reason why SystemNow is criticised is the poor performance seen by many customers. There can be various reasons for this. Sometimes it is caused by any change that causes more server calls, and it’s quite noticeable when it happens.
Rather than architecting the Project Futures “Companion” to share data with SystemNow, Companion performs its own server calls. So if you log into SystemNow and select to view a User’s record, it makes a server call to get the data. The Companion then makes its own call to the server to get the same data so now the apps are in sync. If we enable the Companion to all users in England, then this is essentially going to essentially double the server load in most cases.
In 2022, the Companion was released with an RSS feed, but this was just to a few pilot users. Then Instant Messaging followed much later. At the start of 2024, the decision was made to release it nationally to all customers in England at once. There were a few other modules available, albeit in Read-Only mode.
“ProjectFutures is behaving extremely well, however, there is an incident currently opened against Identity Service in SystemNow as it isn’t handling the load from all the sites.”
Who would have predicted that?
It wasn’t the first issue like that.
As an aside, someone shared the AWS bill. It was $20k a day! Imagine the cost when everything goes live. There are modules that are finished but not enabled. Or are enabled but in Read-Only mode. When there’s more requests and more users, then that price is gonna jump significantly.
We hyped up the functionality to say that if you have two monitors, you can have SystemNow on one screen and the Companion on another screen to view information as you are working in SystemNow. What was even more amazing is that you can copy and paste information across! Wow!
“This is a hugely valuable milestone for the business – our customers love this new tech, this is what will allow us to retain our existing customers and go out to win new business across the UK. The Companion App is the proving ground for ProjectFutures2”
Reason 10: Cumbersome development set up
“for anyone wondering why ProjectFutures itself seems slow, this is our integration environment. It’s lightning fast in Production. Just thought I’d add that – can’t help being protective over it.” – Development Manager
Why don’t we get a good test system? Isn’t it costly to slow our developers and testers down?
I get frustrated that we have authentication enabled on our environments too. When I’m developing, I don’t want to constantly log in. I don’t want to get logged out if I leave it for a few minutes either.
Also, instead of having a set-up where we can run changes in isolation, we end up running most of our changes on a shared environment. Some weeks it has been a daily occurrence that the environment is broken in some way. You can’t log in. You can log in, but cannot select a customer. You can select a customer but the module won’t load.
We are constantly hindering ourselves.
Conclusion
In “The Big Restart” the Development Manager said: “the start of 2026 will have the full system launch for new customers”.
It’s now Q2 2026, and there hasn’t been much progress. There’s still a few modules Read-Only, and some that I heard are finished but were never released. Maybe they will be released. Maybe they will be scrapped. Time will tell.
There are many reasons that are related and tie into each other. I think they can be summarised by the following:
Skills in the wrong areas: too many juniors, not enough experts to drive decision making.
Wanting to use certain technology ie Cloud, AI, rather than choosing technology based on the requirements.
Developers taking the idea of “fail fast” too extreme. Spending time on tooling rather than making features.
Reinventing the wheel: not using out of the box features like Authentication, UI controls.
Restarting projects.
No manager accountability, so they can make mistakes and carry on making them.
Changing fundamental ideas of architecture: big bang vs incremental. Performance issues: latency, plus high server load.
Not prioritising the right projects, prerequisites blocking other projects, core modules not delivered to customers.
Cumbersome development set up so development of any feature is slower than it should be.
I’ve written blogs about how our CTO tried to change our release process and announced it on a “Town Hall” call with the entire department; then loads of teams told him it couldn’t be done, so he had to back down.
Then later, on another Town Hall, he tried to change the Change Control process, but wouldn’t back down when we told him it wouldn’t work. He made the claim of it being a sackable offence if we didn’t follow it. Then a month later, he found out someone couldn’t turn a server on because of the Change Control process. He said it was “malicious compliance” and that will be a sackable offence in future. Within a few months, nearly the entire process had been reverted.
Last week, he was talking about how we needed to move a Data Centre to a new location. He said he preferred to move to the Cloud since it is “inline with our strategic targets”. However, after having several meetings with the experts involved in the Data Centre, they decided the best solution would be to move to another data centre. However, the CTO didn’t like this because it wasn’t inline with their strategy and he thought the move would be too slow.
Therefore, he took the executive decision to overrule them and demand they move to the cloud.
“Do I know we can move all servers to the cloud in time? No. But I was prepared to take the risk. I would rather make decisions and be wrong, than sit by and not make any”
CTO
It seemed strange to me to claim that moving to a physical data centre would be slow, but then moving to the Cloud probably couldn’t be done in time.
He then claimed that
“we have wasted enough time deciding what the plan should be; to move to the cloud or to move to a physical data centre”.
CTO
Isn’t this the worst scenario though? He could have made the decision before any meeting was arranged. But it sounds like they had debated the decision, came to a conclusion, then he told them he didn’t like their conclusion. Then he moaned that they wasted time debating.
So they have had meetings with the experts, and conclude the data centre is the best decision, but since the CTO loves the cloud, he has overruled them. So what was the value of the meeting? And will the staff be motivated to do something they don’t believe in?
“I think I’m compassionate with my team. It’s what binds us together as a team. Otherwise we are a bunch of individuals.”
CTO
I don’t get how he can make these statements and not realise the hypocrisy. How can you be compassionate if you have shown no confidence in their opinions and decision making?
When migrating from on-prem servers to the cloud, the Deployment team decided to change the way servers were allocated, presumably to minimise the cost. They:
“designed separate pools for the public side and private side so that the computer and memory could be dedicated to (and protected from) other types of traffic. Due to this split we reduce the ration of CPU cores to sites from 0.42 per site to 0.28 per site (as the cores were now dedicated to public, private all separately)“.
Deployment expert
Initially, this new way worked fine, but then during a particular busy week, they saw slower response times. It actually led to a discovery of a problem we must have had for a while, that SQL connections weren’t being properly disposed of, which created a bottleneck of the remaining possible connections.
They added a temporary fix which was something to do with “Shared app pools“, rather than autoscaling the application servers which would cost money. But this is a major advantage of the cloud – that you can scale on demand.
So to no one’s surprise, when another increase in load happened, performance issues happened once again.
So now the fix should be autoscaling right? No, they are still reluctant to do so. Instead, they added a fixed number of application servers. Surely that costs money, and increases our costs at quieter periods. I suppose I don’t know all the details but it seems risky to choose a set number and hope that the load never exceeds that.
On Viva Engage, a manager posted a positive message stating that the AWS migration was a big success:
“I am thrilled to announce that we have completed the migration to AWS!
This is a major milestone for our cloud migration programme and has involved many team members across multiple disciplines working together.
We have migrated a whopping 505 TB of data across 4178 databases and have stood up over 1,080 application servers. There has been meticulous planning (over 130 steps in each migration), preparation and countless hours spent migrating our systems, including overnight and weekend working.
The impact of this collective effort extends beyond numbers and statistics. We have successfully improved stability and performance for our end users. The migration has enabled us to navigate the increased load challenges.”
Manager
Yet, someone shared this angry message from a customer. I’m not sure if the first sentence is sarcastic, or if they thought we had been somewhat supportive:
“Thank you for your support in what seems to be a “run” of problems for the business. After our awful experience in November when your software literally tipped over leaving us without a system, I did request that both the ombudsman and your company treated this as a significant event, looked into what went wrong and responded to me with an answer. To date I have not received any such feedback from either party.”
Sarcastic customer
I asked a Software Architect what he thought, since he is usually close to the gossip or involved directly.
The Chief of Smoke and Mirrors will have some explaining to do. performance improved quite a bit as a result of the 64-bit work done behind the scenes (not to the client) but now users do things faster with longer sessions and they have plenty of infrastructure issues around the AWS changes that caused a lot of customers problems as always, one group of people fix certain things, while one group breaks lots of things at the same time
Architect
So it sounds like there’s been some good work done, but also some mistakes made. Then internally, we are announcing it as a great success.
Someone also showed me this complaint where someone had visited a customer and reported back what they had witnessed:
“We visited a site yesterday displaying nearly all of the problems we have discussed to date – still having to reboot the software 10 to 15 times per day! System slowness (witnessed), documents not opening, closing when going into the orders module, first record load slow, changing an order – system crashes.”
Another reason for performance issues was due to missing config after the migration:
“some of our app servers are downloading/installing Windows Updates in core hours, which is causing poor performance for users.”
A simple workaround that sometimes happens is a “cache reset”. That sounds like it’s a similar mindset to the “turn it off and on again” which does magically fix some problems. However, due to the migration, Support had got a bit confused how to remote onto the new servers:
“cache resets were done on the wrong servers. ” –
Manager explaining why performance issues lingered for longer than expected.
Even after further tweaks to the cloud migration, updating the client to 64 bit, fixing SQL connections, and some other miscellaneous changes, the Support team were saying some sites were still having problems:
Can I confirm that things should be improving for all sites following all the brilliant work done? The customer is experiencing the below and I am visiting them tomorrow;
Customer issues
loading can take several minutes
Slowness and crashing every day, at least 9 or 10 times a day
No discernible pattern or time of day for slowness or crashing, and no particular machine is noticeably better or worse
Been experiencing performance issues for 2 years, but have gotten much worse recently (last 6 months)
experiencing significant delays when uploading records
Can take up to 1 hour to approve a small amount of external requests which can involve multiple restarts
Switching between records can lead to delays and ‘greyed out screen’ (not responding)
Constant and randomly crashes and needs restarting – staff having to partition out tasks such as viewing documents and approving tasks
Closing statement
It does seem like our performance issues are a bit of a mystery. I think we have run out of things to blame. Customer internet, SQL connections, 32 bit client, on-prem servers, caching bug. Hopefully one day we will have a fast system.
Over the last few years, my employer has gone Cloud crazy. We are a large company so we have our own data centres. These are costly to run when you need physical space, staff, electricity, software licensing, and a plan of action when things go wrong.
I wonder if it is better to have your own servers when you are a big company. I always think Cloud is best for smaller companies that don’t have the resources to host it themselves.
“Our reasons for using the cloud are the same as others using the cloud.”
Our CTO
Not really true though is it? From what I saw quoted for the virtual machines for our test systems, I think Cloud is more expensive over time. On-prem has a massive up-front cost which is what they don’t like, but we have the capital to do it, unlike small companies that the Cloud is perfect for.
The recent drive to move away from our data centres is that we needed to replace some old hardware, and perform SQL server upgrades.
I could imagine us moving to the cloud, managers then panicking when they see the monthly costs, then demanding we go back.
One aspect of an SQL Server upgrade sounded like they needed to migrate the data to a new physical server. One of the tables they were concerned about was Audit, which adds a new row every time the user edits a record, which they stated was around 9 Billion records. A copy of the changed data is then saved as XML, so then you can do a before/after comparison. So that particular column is a problem.
So for the data that would still remain in our data centres and moved to a new server with a modern SQL Server version, the plan was to migrate the table without the XML column in it. Instead a new boolean (true/false) column was added to state if there should be data there, and instead, the data is moved to the cloud.
So now we are paying to host the database on our own data centre, but then have certain data in AWS which sounds like it should be more expensive. The justification is that we didn’t need to buy as much hard disk storage which they reckoned could have cost a massive £500k! Then it would mean the migration to the new server in the data centre was faster.
Still, we needed to transfer the data to the AWS Cloud storage. I think the idea was that Audit data isn’t accessed much, so it’s better to move it to a cheaper but slower storage method, then request it on demand. So in our software, instead of displaying the data instantly when you view that record, there would be a “view more detail” button, and only then do we request it and show it.
I think the mindset is just to focus on the cost figures that are apparent. Seeing a figure like £500k sounds like a crazy figure, but if we look at the cost to store it over a few years, does storing it in our own servers outweigh the cost of paying Amazon to store it?
A new corporate buzzword that gets thrown around in this subject is FinOps, as in Financial Operations.
One of the challenges we have when we start to build a new service is around estimating the potential cost of that new service in AWS. This ultimately goes towards setting the budget expectation for that service and therefore how we monitor it from a FinOps perspective. Do we have any experience within the department or anything we can leverage to help us get better at understanding the potential budget expectations for a new service we’re building?
Concerned staff member
In one of the recent “Town hall” meetings, the CEO was ranting about how high our cloud costs were. He said we currently had £250k in AWS servers that are switched off (not sure if that was a yearly figure, or even more unbelievable; monthly). These were servers just for development/testing. If our testing teams are spending £250k on servers we aren’t really using, how much are we spending on ones we are actively using? Then how much does our live system cost?
Now when you see those figures, that £500k hard disk storage doesn’t sound too bad.
“FYI – Stopped instances don’t incur charges, but Elastic IP addresses or EBS volumes attached to those instances do.”
When people start talking about the cloud, they quickly start dropping in jargon terms. Sometimes they use multiple terms in the same sentence and it quickly becomes hard to understand when you aren’t familiar with the cloud providers. Even if you are familiar with one particular provider, other providers use different terms for their equivalent service. I think AWS is particularly bad for their naming which often aren’t intuitive. So when people start talking about Elastic Beanstalk, Route 53 and Redshift; it’s hard to grasp what the hell they are talking about.
Here’s an example of equivalent services by four different cloud providers.
Recently we have had another tech guy join the company who is reporting to the CTO. I find that people in these kind of roles want to put their stamp on things by coming up with a new idea.
He presented his idea in our monthly Tech Meeting. He wants to attempt to address our performance problems by taking traffic away from our main on-premise databases. There’s been some similar ideas recently, and although I’m not great when it comes to hardware, networks and general software/hardware architecture; I am sceptical that these ideas can work.
His idea is that we can replicate the database in the cloud (“the cloud” solves all problems you see), and then the database in the cloud can be used for Read access, whereas Write would still go to the main on-premise databases (then synced up to the cloud).
The Announcement
This programme of work is to move workload away from our primary systems to enable these systems to withstand expected load factors from upcoming initiatives as well as expected growth in usage on our APIs during Winter 2023.
The intent is to run focused cross functional teams in work-streams across the group to deliver this initiative. The approach taken here is to place multiple bets, across multiple teams. The expectation is that not all teams will deliver by September, but enough to bring in the headroom we need.
The programme is intending to free up at least 20% load across our core databases.
Upcoming aims: • Strategic, move read-only workloads to Aurora. • Redeploy APIs to AWS, Move to cloud technology, Containerise and Optimise Service • Enable use of replica data when ready. • Move Appointment Workload • Mitigate 8am peak load. • Use caching engine on AWS (Elasticache/Redis), mitigate 8.2% of PC DB Load • Reduce load on the DB during day time. • Reduce Datacentre and DB load and improve performance • Mitigate 6.2% of DB load by optimising how we summarise task counts • Proof of concept is Complete, expected to cost £2m a year.
My Conversation With Architect Mark
I think the reason for the replication (as opposed to just moving it all to the Cloud) is that you can’t fully commit to ideas like this. You have to have a rollback plan. So if we find it doesn’t work, is too expensive etc., we can just return to the old way without much inconvenience. I asked one of our Software Architects what he thought of the plan because it doesn’t sound right to me:
Me doesn't sending data out to another database just increase traffic, and they wanted to reduce it? Mark Yes, it will also be delayed, and often broken Me no pain, no gain Mark they're replicating data, and it's unlikely it'll be used Me I don't see how you migrate things. You have to keep them both running until you are confident it works, then bin off the old database. But then in reality you just end up keeping them both for longer than expected Mark you then also need cross-database transactions or to be very careful with queries yeah, that's basically it. Have the same API at both ends, some sort of replicate and transform on the data to ensure it's in both. Persist to both simultaneously, then when all works, turn off the old Me The CTO said that “some people say there is a delay, but it is only 5 minutes”. Does that address any of your concerns at all? Mark no, this is only the second time I've heard about this, and the first I laughed I agree with the principle of strangler pattern for migrating, but this isn't migrating it's keeping multiple DBs 'in-sync' Me does that mean you can view an appointment book which is 5 mins out of date, and you try book an appointment, then it checks the real database and is like "no mate you cannot do that"
The conversation between architects
Mark then sent me a conversation he had with two other architects, Andrew and Jon. Mark already had concerns with the “appointment book” example.
Mark so when this replication system goes down for a few hours, what happens then? I guess the system tries to book appointments for slots already booked, put in requests for items already issued etc.? seems our business layer needs to be aware of how outdated the original information was, so it can compare something like a changelog number. Sounds like a big challenge to implement correctly
Andrew 11:10 Yes, any write operations will need logic to ensure that cannot happen Mark. John and I have already called out that Appointments and Orders will have significant challenges with this replication model and have suggested that the initial focus should be on User Profiles, and any historic data, etc.
Mark 11:13 User Profiles and historic data seem just as dangerous to be honest.
Jon 11:15 The idea I suggested these is that you would check the change log on the primary system before even considering going to the replica. If the User had had a recent change (what counts as "recent" is TBC, I suggested 30 minutes) you wouldn't even consider going to the replica.
Mark 11:15 can we implement the strangler pattern properly? set up proper Appointments APIs to use in our datacentre, and AWS. duplicate the data. then dual file everything against the APIs? if one fails to file, the other gets rolled back. we ensure consistency, we can transform the data, and we're using the pattern as intended Jon, I agree your idea is the right way to do this sort of thing, but it will be adding logic and latency in a lot of places (as well as augmenting every one of our products to be aware of this), and not bringing us forward, but continuing to keep us in the primary data-store model
Jon 11:18 Honestly if the use case for customers looking at their data, then having it a touch out-of-date information isn't as critical as if our actual users sees an out of date view. As a hypothetical Customer who knows nothing about IT, if I viewed my record straight after a consultation and it wasn't there I would just assume that there was a delay and it would appear later. When it comes to actual Users viewing the record, it's absolutely critical that they see the up to date view. And when it comes to appointments that's also critical because appointment booking is fast moving, it'd be an awful experience for a User if every "free" slot they booked turned out to be booked minutes earlier.
Mark 11:19 depends, if you've just requested a particular item and the page doesn't update to indicate that, can you continue requesting it?
Jon 11:20 Many of our users (mine included) turned off online appointment booking entirely at the beginning of the pandemic and use a triage system now. You wouldn’t be able to successfully request duplicate items, because the write would take place conditionally, so if it had been requested already then it'd say no (if designed even vaguely competently).
Mark 11:22 the write wouldn't come through, but it'd be confusing for the User seeing the prescription still requestable, unless the application has its own datastore of state
Jon 11:22 Yes it would be far from ideal. But the CTO has some ideas about that (having a "recent changes" dataset in a cache that is updated live, and merged with the replica's data. feels like there's loads of little bits of logic that need 'tacking on' to resolve potentially quite serious incidents. When the correct use of the strangler pattern gets us away from on-premise as primary DB, and moving in the direction we want to go Yeah, this isn't easy and requires careful consideration.
Andrew 11:30 You are absolutely right Mark - there are a heck of a lot of potential gotchas and ultimately the plan has to be to use the strangler pattern, but at the moment we are looking at a rescue plan to put out some existing fires in the data centre and to handle predicted significant increase in load that will hit us in the Autumn. Everything that you have flagged is being considered. The only fall-back plan that we currently have is to spend nearly £4m / year on additional SQL Server readable secondaries (on top of having to pay an additional 12% on our existing SQL Server licences thanks to MS hiking their prices) and nobody has the appetite for that.
Closing Thoughts
I don’t know what the Strangler Pattern is, so I’ll add that to my reading lists. However, it seems that even with my limited knowledge of architecture, our Software Architects have similar concerns as I do. There’s been plenty of ideas that the CTO (or similar level managers) have quickly backtracked on due to not consulting people who have knowledge on whether their idea is actually logically sound. I’ll keep my eye on this idea to see how it develops.
Recently we have had several major incidents due to: software bugs, incorrect configuration being applied, not renewing licence keys, and migrating servers to the cloud and failing to check all services were correctly configured and running.
Our Hosted Services team gave a presentation of work in their department, and gave more insight to even more failings that have happened recently. As far as I am aware, Hosted deal with servers, data centres and networks.
Hosted explained that due to the decision to move all servers to the cloud, when their usual time came to replace old servers – they didn’t bother. But the migration has been a slow process and delayed which meant our software was running on inferior hardware for longer than anticipated.
“We don’t need to invest in the in the architecture that we’ve got, which was not the right decision in hindsight
We had a team of people who, in some cases, were the wrong people. They didn’t have the appetite to go and actively drive out issues and reduce the points of failure in our networks.”
Hosted Manager
He then goes on to say the change in strategy caused many of their long-term staff to leave. These people that really knew how the business worked.
“So we lost around about 90% of the team over a relatively short space of time and that put us into quite a challenging position to say the least. And needless to say, we were probably on the back foot in the first quarter of this year with having to recruit pretty much an entire new team.”
Hosted Manager
Then, because they were short staffed, their backlog of work was increasing, putting more stress on the people that remained:
“We had to stop doing some tasks, and some of our incident queues and ticketing queues were going north in terms of volumes, which was really not a good place to be.”
Hosted Manager
I’ve written about this situation in the past. It has happened in the Development department when a new CTO comes in, and says that manual software testing is archaic; so people have to learn automation or lose their jobs. Then a few months later, they realise their plan isn’t so feasible, yet have lost some good software testers to other companies, or allowed others to switch roles and aren’t interested in going back. Then the releases slow down because we can’t get the fixes tested fast enough due to the last of software testers.
They go on to say the Firewalls suffered 50 major incidents in Quarter 2, and now they have “procured new firewalls” to solve it. They have reduced bandwidth into the main data centre by routing certain traffic through an alternate link. The “core switches” at our offices and data centres are “End of Life” and will be upgraded to modern hardware (Cisco Nexus 9K).
So it sounds like they have a plan, or at least are doing the best with what they have. It sounds like all departments are currently shooting themselves in the foot at the moment.
Recently, we have a new CTO and a new Head Of Development. Like all new managers with power, they want to make their own changes.
In The Changes #1, I explained changes to job titles of Testers. In Changes #2 I explain that Developers are also affected.
Our previous management decided our upcoming software would specifically use the cloud service AWS, whereas the new managers want a “cloud-agnostic” approach (work with all the clouds!). So they want us to have the possibility of using one, or a combination of: Azure, Google or AWS – although the emphasis seems to be moving to Microsoft’s Azure.
Currently, I don’t think there is much rationale for this. Everyone specifically learned how to use AWS, so will need some time to adapt to the differences. When people started learning about Cloud Technologies, the problem of “Vendor Tie-in” was raised and instantly dismissed by the previous management. It makes sense to ensure you have that flexibility, because if your provider increases their costs, then you can easily migrate if you had the architecture for it.
Another change is that they want every product to have their source code in GitHub. Maybe it does make sense to have all our products in the same place. The reason we have them scattered is mainly because our products were acquired by a series of acquisitions, so the original companies had chosen their own languages and technologies. However, our core, flagship product is already in Azure DevOps, and given that the main cloud provider is going to be Azure, surely it makes sense to keep it all in one place?
These changes to jobs, processes and technologies seem to have solely been decided by the CTO and Head Of Development. I feel like they haven’t discussed these ideas with the Development staff at all. I’m intrigued by what other arbitrary changes will be forced on us. With any change, I think you always have to ask yourself “what is the problem we are trying to solve?”. If you can’t explain the problem, then it shows you are making changes just for the sake of it.
Recently, I have been reading Sir Alex Ferguson’s “Leading” book and came across a great quote:
“There is no point suddenly changing routines that players are comfortable with. It is counterproductive, saps morale and immediately provokes players to question the new man’s motives. A leader who arrives in a new setting, or inherits a big role, needs to curb the impulse to display his manhood.”
Sir Alex Ferguson (ex Manchester United football manager)
I thought it was a very relevant and appropriate quote for this situation.
Stranger Coding Tings 