When we add new, optional features, we often put in a flag to enable or disable the feature for certain users. This allows us to slowly roll-out the feature, or only enable it for customers that pay the premium. If there’s problems, you can also disable the feature quickly without pushing out a new version of our software.
One team had decided to rename their module, and therefore were updating the configuration flag’s name.
A lead developer, who reviewed the change, questioned if they could do that without running into incompatibility issues. The project team’s lead stated:
“No, we have the feature validation at source and target separately before we do anything. So, there should not be any compatibility issues.”
Project Lead
However, I was convinced the lead developer was correct. We have multiple versions of our software deployed, but we only have one version of the Configuration Manager tool.
So let’s say in Version1, the new module is called “User Manager“, but in Version2 they want the module to now be called “Staff Management” – and so they update the main software and the Configuration Manager tool to use this new name.
When we use the Configuration Management tool for new users that are using Version1, we update their config to use the new name “Staff Management“, however Version1‘s software will be looking for “User Manager” and will not find it, so will think the module is disabled.
Existing users on Version1 with the old flag in their configuration will work as normal, but it won’t work for new users. For Version2 users, the Configuration will have to be redeployed since their config will have the old name, but Version2 will be looking for the new name.
If the Configuration Management tool used ID’s rather than matching on text; it wouldn’t be a problem, so we have screwed ourselves over there. Matching on text is rarely a good idea due to possible spelling mistakes, case sensitivity (is “User Manager” the same as “user manager“?), and usually less efficient matching on something else like a number ID.
I spent a while trying to think of ways around this issue. Ideas that I thought of involved writing complex database scripts, running scripts outside the release process and getting other people involved. But then I think all my ideas still wouldn’t solve the incompatibility issues and it seemed way too much work for something trivial.
The team were adamant they wanted to rename it though, but it didn’t really matter too much. Only our staff see the Configuration Management tool, and we can update the main software so the users see the new name. It just adds confusion if someone tells you to enable “Staff Management” but you can’t see the option, so they have to correct themselves and ask for “User Manager” instead.
I would have thought the project team would have ran through different scenarios to test if their idea was feasible for new and existing users. But even after questioning it was feasible, they were adamant there wouldn’t be any compatibility issues so I had to explain the scenarios to them.
I am mentoring an Apprentice who has never done C# before, and this is his first programming job. So this is a diary of some-sort of his progression and my ability to mentor.
Managers love rearranging teams and sometimes I facepalm at how ridiculous it gets. When our teams were first formed, I was in Team A and my Apprentice was in Team B. I flagged this to my manager as a strange choice because it would make more sense if I was working alongside him and knew the work that he was assigned to do.
After several months, I got reassigned to Team B. A few months later, he has been reassigned to Team A but will work as a Software Tester. So now we are in opposite teams once more.
All this switching is just stupid, but I don’t object to him switching roles. Recently, he did say he understands C# a lot more now and can read more code that he could before, but he struggles to understand what our software currently does.
The annoying thing is, I did suggest to him early on that; temporarily, (or intermittently) switching role to a Software Tester may be beneficial to him. Testing the software will mean he learns the functionality of what it is supposed to do, he sees more of our process, and also understands how the Test Environments are deployed/configured etc. Our software is really complex so I felt his insistence to get stuck into the code was actually hindering him.
He actually sounded quite excited to try it now, and I felt that if he does enjoy it, it might suit his abilities more. If he doesn’t do well, then we may end up letting him go.
This is the thing, he has been here a few years and hasn’t really learnt anything, and when he does ask basic questions, I find myself telling him I had already gone through all this stuff before. I don’t expect him to retain everything I tell him, but I did emphasise there was a lot to learn, and this team is incredibly difficult so needs to put the effort in early on – which he didn’t do. Now Colin (who I have written about many times on the blog) has moved roles from Developer to Manager, Colin told me he is aware that there are several underperforming people, and that he is going to be looking to move them on; my apprentice is one of them.
Colin seemed to suggest that if he can’t complete work himself (either from a Development or Testing point of view), then he could be let go. A week later, my apprentice was asking questions from a Developer perspective and I asked what happened to the Testing work. He said something about “not wanting to be a Tester” and “didn’t want to get stuck doing that work”.
Not a good idea.
Showing he can do Testing would be much easier than trying to fix some difficult Bugs. Going against a manager’s idea when you are close to being let go is also playing with fire. Also, what happened to the enthusiasm he showed a week prior?
I’d also previously explained to him how there’s always times where Developers have to chip in and help Testers. Those that kick off a fuss or don’t contribute enough are always looked down upon since they are not “team players” and don’t show a “quality” mindset which everyone should show.
We will have to see what happens. I’ve only ever known people to be sacked for having major arguments with managers, or doing something really offensive. It’s a rare occurrence so I’d imagine we would just keep him around, even if he doesn’t contribute much.
Looking through my draft blogs, I have quite a few to do with performance, so I have grouped them together to make a complete blog entry.
“the whirling circle thing is just whirling”
User annoyed at the slowness of our system
Do we care about performance?
I’ve listened to some Tech podcasts that have discussed Big O notation which is a way of classifying the performance of an algorithm such as O(n), O(n^2), but outside of studying Computer Science at university, I personally have never heard anyone else reference this. I think with some programming jobs, the efficiency of algorithms is crucial. For the most part, I don’t think people care unless you are doing something that turns out incredibly slow. In the age of Cloud Computing where you can be charged by the millisecond, then it will become increasingly important for server-side algorithms.
Where I work, we are very reactionary rather than proactive. Recently, we have had loads of complaints about our performance, so then we have identified the slowest areas and addressed them.
Thinking about performance upfront would mean the user gets the best experience and doesn’t complain. However, there’s plenty of examples of code which runs fine on your machine but runs poorly in live. This can be because you have a faster computer, the network speeds are faster, lower latency between the servers (when developing, the client and server is on the same computer so there is no latency), or you don’t have a database which represents live – you have 100s of rows rather than 100,000s. I think this last reason is often a cause for us, in addition to concurrency (thousands of live users simultaneously using the system, compared to a few testers using a test server).
An example of how we don’t often consider performance is as follows:
Example 1
I showed some initiative recently and essentially rewrote a project that was inefficient.
Some server calls had a check to see if a feature was enabled before attempting to retrieve data from the database, but some didn’t. Therefore these code paths ran queries against the database before returning no data.
When the feature was enabled, there were other parts of the code that were making the same server call twice in the same sequence of actions,
and other places that made a server call even though the client already had the data from a previous call.
It was a mess basically.
The functionality behaves correctly and the performance didn’t seem out of the ordinary, so the Software Testers never flagged it. We were probably talking an extra 50ms to 200ms in total when testing internally, although this would be worse when deployed live. It should have been obvious to the Development team that there were inefficiencies if they were thinking about possible improvements to their codebase, or if they were paying attention to the server calls and noticed their new server calls when the feature was switched off.
Volume and Performance Testing
We have loads of test environments – virtual machines with databases and servers where the Software Testers can test our code changes. I would have thought these would be fine to do performance testing, and spam these test servers with calls, but a Test Environment Engineer once sent an angry email:
Our standard test environments are Absolutely not to be used for Volume and Performance testing. This can impact the whole devtest infrastructure, as well as some live services that share the same infrastructure. Doing this could cause a Major Incident.
Test Environment Engineer
This seems bizarre to me. Why would you share resources with the live production environment? Surely the Test Environment should be in its own isolated sandbox, where you can create, update, and delete with the safety and freedom.
Example 2
We had another performance issue where a large amount of data was attempted to be retrieved. When the developer analysed it, we weren’t actually trying to retrieve a large amount of data afterall.
“There were 142,858 Id’s passed into the comma separated string, I found that there were only 64 distinct ID’s once I removed the duplicates. After the duplicates were removed it went from 76minutes duration to under 1 second”.
Software Developer
What a performance improvement!
I wonder if passing the data in a form that isn’t a comma-separated string would also help. If you are passing in “12, 156, 201, 202, 284”, then, to get the ID’s, you need to remove the commas and place the numbers in a table. If you just pass it in as a table, then you are cutting out a step on the database layer.
Example 3: SQLite
One part of our system used a local file cache (loads of files were downloaded to the users machine, then our software would just load up these xml files rather than going to the server every time). It has worked fairly well, but overtime, the number of files has grown.
I think the performance issue comes from the fact that all the files are then loaded and are kept in memory, so it is a memory hog, and sometimes leading to “Out Of Memory” errors. Maybe the solution is to work out if we can dynamically load these files in full only when they are needed, while only having the “metadata” in memory at all times.
Colin came up with an idea to move the files into a local database using SQLite. This innovative idea impressed the managers and were eager to do it. I was sceptical because I think we would still end up loading all the files, just the place where they are stored has changed.
Day 1 of release, the following Major Incident was raised:
“SQLite corruption. Out of memory”
After that was fixed, I believe the changes caused 3 additional Major Incidents. Colin still swears by it. He thinks it was an amazing idea. I ain’t convinced.
User Feedback
It’s rare that I come across comments from users about our software, but they do often make me laugh. Part of the problem is that they use software, but aren’t really technical enough to use the correct terms, so when complaining about a performance issue, one user remarked:
“the whirling circle thing is just whirling”
User annoyed at the slowness of our system
Another thing about user complaints is that I tend to only hear about the real angry complaints and not just generic comments. I think Support had told a user there was a large amount of data on that particular record, and the more data on the record – the longer it is going to take to load.
“Regarding larger records, this is a predictable problem. What mitigations are being put in place to offset that? I still have 20 years of working life yet, don’t tell me that by 2035 I will have to wait 20 minutes to load a record???!”
Angry User
It’s a valid complaint. We need to come up with a different way of loading the record so they aren’t sitting around waiting for the full record to load.
I think I have discovered a new excuse to use in order to buy time; “Performing Analysis”.
Example 1
There was a Major Incident on Friday which I debugged and suggested a simple fix. However, instead of me going ahead with this fix, the managers wanted the team responsible for introducing it; to take accountability for it.
After the manager sent the email and Cc’d me in, the team responded with “The Team is analysing this issue; will update you shortly”. Then, after the entire day has passed, they send a follow-up email “this will be ready by close of play Tuesday.”
It was currently Friday, and we wanted it released for Monday (since it was a Major Incident) – not released on Wednesday. So I had to sort it out.
Example 2
Dave commented on a Code Review that he thought the Developer had changed too many files; and there was a simpler way of introducing their fix. They responded: “we need to perform some analysis” and took a day or so before coming up with new changes. It would have just taken 10 minutes to rewrite it; I didn’t see any scope for this so-called “analysis”.
A couple of years ago, I was working on some new software but the management of it was an absolute disaster. Even though it was an absolutely massive project, I was often sat there not knowing what to do. Additionally, there were teams that were doing exactly the same work as each other. It was a mess, so eventually I moved back to work on our existing software.
This week, there was a project update demo by a team still working on this new software. I thought all the speakers were working on this feature, but it seems Jennifer might have been on annual leave or something because she didn’t understand what work was done.
After a few slides of their presentations, Jennifer interjects.
Jennifer : “How is this different from what Igor is doing?”
(Igor is part of another team)
Luke: “I’ve had the conversation and I still don’t understand why this was brought in twice.”
Jennifer: “I don’t know what the use case is for the two versions.”
Mark: “We never got to the bottom of it, so we just cracked on and built what we got asked to do.”
Brilliant. So there’s multiple members of the team that were aware that another team was also making this functionality. Despite having conversations with managers, they still were none-the-wiser, so then they just made it anyway, despite knowing that one team’s work is going to be binned off.
I can’t believe a couple of years have gone by and they are still doing stuff like this.
Recently, a software developer refactored some method calls in a strange way, so I questioned him on it. He stated “you cannot have more than 6 parameters”.
This is nonsense, but he means there is a Code Analysis rule that flags this up. The theory is that: if there’s many parameters, some of them are probably related, and so they can be grouped into a common object.
public void AddUser(
string forename,
string surname,
Date dateOfBirth,
string addressLine1,
string addressLine2,
string town,
string county)
In the example above, you can group the last 4 fields into an Address object.
When the Code Analysis rule flags your code, I think it’s more of a question of “can you group these parameters together?” and not a “you must never have more than 6 parameters”.
If you think the code is fine, you can simply suppress the rule. If you think it is correct, then you can create a new class to group these variables together (or use an existing class if there is something relevant).
I’ve written quite a few blogs on Developers writing code in strange ways because Code Analysis prompted them to. I think Code Analysis aims to prompt you to write code in a standard way, and avoid common errors/inefficiencies. It seems to have the opposite effect because many developers don’t think for themselves.
Out of interest, I asked a nerdy developer if there really was a limit to how many parameters you can have. So like a total nerd, he tested it out. Apparently, in C#, the max you can have is 65536 parameters in a constructor. Then he gave me additional nerd stats. His file was 4MB of source code which compiled to 1.9 MB exe file.
No idea why 65536 is the limit, and I didn’t spend any time investigating it, or quizzing him further. I just accepted it’s such a large number, you would never reach it. It’s more than 100, so you may as well say there isn’t a limit.
Recently, we had a bug with the Microsoft Word integration, specifically for MS Word 2010.
There was a meeting about it, and someone stated that we removed it from the list of versions we supported; therefore we don’t have to fix it. We can just tell users to upgrade to a version of Word that we do support.
However, after other people looked into confirming this claim, it turns out we never sent out any communications to notify users of this change. Since we didn’t give them adequate notice, we have to fix the bug and keep supporting this version for at least another 6 months.
Additionally, we don’t have Office 2019 listed on software we support, yet we support o365. Or we say we do, but no one in Development had a licence for it; not even Testers. So we just hoped everything worked fine up to this point.
There was a bug in our software where a crash occurred when an address with over 50 characters was specified. For the fix, the developer changed the code to truncate the text down to 49 characters.
“why are you using 49 and not 50?”
Me
He thought the “length” property was actually 1 less than there are; so 50 characters would report the Length property as 49. He was thinking of arrays where they are zero-indexed, so the first position in an array is Index 0, second position is 1 etc. Obviously he didn’t test it, and it was the wrong assumption. Length means length.
Also, there were 5 other address fields with no validation (Line 2, Town, County, Postcode, Notes). Why didn’t he add validation for them too? Just because the bug report was for the first line of the address, why wouldn’t you consider the rest of it?
After I prompted him, he added validation for all fields, truncating them to 50 except one where he truncated to 51. Why 51?
“I manually checked in the registration screen. That field holds 51 chars.”
Weird, because I manually checked it and looked at the code – it is 50. Let’s check the database column too… That’s right. It’s 50.
“Thanks, I’ve changed it to 50. Sorry, some kind of counting mistake.”
We seem to have so many developers that don’t put much thought into their work. This is one of the easiest bugs you can fix, and yet I’m basically hand-holding them through it. When there’s more complicated bugs for me to review, I end up scrutinising the code in great detail because I end up assuming they put zero thought into the solution, and didn’t test it.
I am mentoring an Apprentice who has never done C# before and this is his first programming job. So this is a diary of some-sort of his progression and my ability to mentor.
The months have flown by, and my last Mentoring blog was back in August!
Back then, we were in different teams and I stated that I wasn’t going to be proactive in helping him since I had constantly given him advice, offered to spend time with him, but I didn’t see much effort on his part. Since he was in a different team – it wasn’t really my responsibility to help him progress with his team’s work.
Things have changed slightly since I got reassigned to his team (managers sure do love reassigning people!). I gave him a list of bugs to look at, which were (what looked like) easy ones. It is proper work to do (rather than working through tutorials), although hasn’t gone through the normal prioritisation process, so even when he fixed them (well with a lot of supervision from me), they are just now lingering in a code branch and won’t actually get tested and released.
I spoke to him at the start of December and said there’s going to be downtime as most of the team are off, and we end up having a “change freeze” where we don’t release anything to our users anyway. December is the perfect time to do his personal project and write as much code as he can without any expectation/pressure from any management or team members.
A few weeks later, I asked him what he had done and he said he was just “finishing off some online courses”. Which to me, should have taken a week at most and it mainly involves just watching videos. He said he would move onto his personal project next. A week later, I asked him if he had any code to show me or discuss and he said
“I’ve been chilling out… it’s Christmas”.
Apprentice
I really wanted him to get involved in the projects next year because I think January/February is going to be really busy. We’ve been putting off starting projects (although a few of them I actually had a look at in the December downtime, and prototyped a solution), but they will no doubt want to rush them out as they suddenly become “urgent”.
However, since he has barely put the effort in again, he still is way behind the skills level he should be performing at – which means it’s going to be difficult to assign him parts of a project he can actually do.
Stranger Coding Tings 
