Tag: Software Development

Development Environments

This blog is basically stealing “Healthy Software Developers” explanation of “Development Environments – Isolating Customers From Your Changes”

Introduction

“Development environments” allow software engineers to view a particular version of the software and control who has access to it. 

If you go back about 20 years ago, there was much less maturity with how changes were rolled out to customers, and every company had a different process. In some companies, developers would just immediately roll them out, straight into production. This might work for the most part if you have a very small and trusted team without the process or contractual requirements. These days, the standard is that most companies need a minimum of three separate environments when developing software.

Even where I work, people used to tell me it used to be much less professional, and without as much legal scrutiny – allowing them to drop DLLs onto a server to quickly fix a bug for a single customer. Now there’s all kinds of people who need to sign it off, and 99% of changes are released as an official version to multiple customers.

Development Environment

The main development environment is a place where an individual engineer can actually work on the software in isolation from any other stakeholder such as a Tester or Customer. Instead of making changes directly to a live website/application, the changes can be run on a local copy on the developer’s machine. The developer can begin looking at the impact of the changes he’s making, and will use test data. Any problem introduced only affects the single developer.

It could be as simple as a copy of a database and a version of the website. A more sophisticated program may have even more dependencies, so need multiple websites or services configured – so you could leverage modern technology like “cloud services” or “containers” to easily deploy this setup. The decision of how this looks depends on the complexity of the software architecture, team size and process.

Staging Environment

The “User acceptance” test environment, sometimes known as a “staging environment”, is another copy of the application, but developers aren’t making any changes to it, and it is also using test data. The purpose of this environment is typically to run exhaustive testing to find any issues before deploying the changes to the end users. 

When companies rely on manual testing, and don’t have modern “continuous delivery” processes, an unreleased version may sit in the staging environment for an extended period of time. In a more automated environment, automated tests can be run, and the version easily deployed to customers if passed. This could take as little as fifteen minutes to an hour depending on the tests, software complexity, and process involved.

Production environment 

The Production environment, or Live environment, is where your customers use your product; so the actual website or application. 

Demo environment

You can have other environments like a Demo environment. This is another copy, but not used by Developers or Testers. The version can be “work in progress”, or a version that is deemed to be ready; but the purpose is to show customers upcoming features and to gather feedback.

Capacity testing

You could create a special kind of test environment for capacity/load testing. This is meant to  stress test the software and see how it’s performing under heavy load. Systems can have increased traffic at specific times of the day, e.g. users all logging in at 9am to start the day, increase in website hits during lunch breaks, more shopping traffic during Christmas period etc. If users are paying to use your service and it goes down; it’s a huge problem. If you sell a product/service on your website and it goes down; then it can cause lost sales. So performing this testing can be essential.

Changes to the Software Delivery Process

One of the problems we have where I work – is not releasing fast enough. When you read about Software Development, you hear of these companies that can release minor updates every week. Maybe it is more of a Web Development thing rather than an Application Development one, but there are also contractual reasons why we cannot release faster.

However, over time, the release time has continuously crept up to 4-6 weeks which causes problems.

If there is more time for each release, it means that the scope of the current release often increases further. For example, if there is a fix that needs to go out within 3 weeks to meet the SLA (Service Level Agreement) and the next release will go out in 4 weeks, then you have little choice but to get it in the current release. If you are checking it in close to the deadline, then you might end up delaying the release in order to test it. The more you delay, the more chance of someone else needing to get a fix in the current release and it grows further.

If there’s many big projects targeting the same release, each team developers in their own code “branch”, then will merge into the Main branch for release. Since it’s not really feasible to all merge at the same time, you end up taking it in turns and resolving any conflicting changes. To be honest, it’s quite rare that we will change the same files for the main feature changes, but there’s certain files with a lot of churn, mainly ones containing XML. To merge in projects, it usually takes a few days, then all the bug fixes on top. The Testers can’t really begin testing until it’s all merged so it’s a lot of overhead to manage the release.

When the releases are large, the Testers insist on running more Regression Tests which increases the Testing phase and can cause further delays.

“I think we spent about 2 months on that regression. It was madness. It was a HUUUGE release”

Software Tester

So smaller releases are much more manageable, take much less time to test, incur less risk, and have lower scope for scope-creep.

Our Software Delivery team made an announcement about this (basically just saying the same things I have just discussed), and desire to plan in quarters but release in a couple of weeks.

In the past, we would scope a single release looking at the features, fixes and minor enhancements we wished to deploy. We would follow a process of merging everything into our main release branch before undertaking testing. This was a two-phased testing approach, integration/functional testing each feature and fix, and then regression testing to ensure pre-existing functions continued to work as expected. We would then spend eight to ten weeks deploying the release through Controlled Roll Out and Customer User Acceptance Testing.
 
This approach brought with it a number of challenges. Merging everything in was time consuming, issues or blockers with one feature would slow down or block other features, regression testing was a challenge, and this also put pressure on the roll out and deployment through pushing out a number of changes in one go.
 
To try and mitigate some of these challenges, we are now adopting a strategy of breaking these large releases down into smaller updates.
 
Working in quarterly cycles we will scope what we wish to deliver over a 12 week period. Each feature will be analysed and risk assessed for size and complexity by our Engineering Leads, and have a business value determined by our Product Management and Commercial Teams.
 
Using this feedback we will then determine an order in which we wish to deliver each feature. We will then merge them into a release and test them one at a time (potentially two if both are small and low risk), before signing over to Release Management to commence deployment.
 
We will then deploy the full scope over a series of smaller releases rather than in one large release.
 
The last update in the cycle will be a maintenance release to address the backlog of prioritised service work.
 
The objective behind this approach is to have our users benefit by taking elements of the release scope earlier than they would have before, whilst also simplifying the testing approach and hopefully enabling us to push code out across the estate quicker.

Security audit/Penetration Test

Last year, my employer paid for a “security audit” for our software, and any “issue” on their report was then a high priority to fix.

I think the majority of the issues they stated were incredibly unlikely to happen, or there would be an easier means of acquiring such information. One of them was that UDP messages, which are only sent on the user’s local network – had an unencrypted username. But if the attacker was already inside the user’s building, it would probably be much easier just to look at the user’s screen rather than plugging into their network and running a packet-sniffer.

Problem 1:

Shortly after we released some “security fixes”, we had a few bugs reported, one of which was:

“Users unable to reset their own passwords using the self-service password reset process”

So the security improvement created a Security major incident. Brilliant.

Problem 2:

A colleague was explaining another problem which I only had a vague understanding of. But it was to do with the version of a protocol being sent in the header of a message. It is classed as a security flaw because an attacker can look up known security flaws for that version and try to exploit the system that way. I suppose they can still guess what the version is, and try different “attack vectors”; but I suppose the less information you give them, then the more hassle it is. As my colleague explained it, a change had been made to remove the version, and tested on a specific version of SQL server, but we have multiple versions on our live infrastructure. When it came to demoing it with a client, they discovered a particular feature was broken on a different SQL Server version. A little awkward, but at least it didn’t go live.

Potential Problem 3:

When it comes to authentication failures, if you tell the user that the login attempt has failed due to the username being wrong, or the password being wrong – you are making it easier for malicious users to attempt to gain access to someone’s account. So if an attacker is guessing usernames and normally sees “Invalid username”, and eventually gets an “invalid password” message, then the attacker knows the account exists and now just needs to get the password.

There was one API call that returned an error code along with the message “Invalid username”. So as advised by the security audit, the developer changed the message to “Invalid username or password”.

On the code review, I pointed out that the developer also needs to change the Error Code too, and they replied saying the security audit stated that the message was the problem. They definitely hadn’t thought about the actual problem. If Error Code 40 is “Invalid username” and Error Code 41 is “Invalid password”, and you change both texts to say “Invalid username or password”, then it’s not really any more secure is it? since we are still returning 40 when the username is wrong, and 41 when the password is wrong. What we need to do is make 40 and 41 obsolete, and make a new Error Code for “Invalid username or password”. However, can we actually do that? When you have third parties using your API, if they have written code which relies on certain return values – then you break their software when you change yours. So we would need to inform everyone that the change is coming, then release this change.

UDP and ChatGPT

A senior developer questioned why we were encrypting elements of the UDP message. The developer struggled to explain and then incorrectly used ChatGPT. It was obvious by the increase in technicality and vocabulary in his response.

George
Out of interest, why are you encrypting this?

Karthick
According to the system audit query, MAC address and Sending user display name and the location name should be encrypted.

George
But Mac Address is sent in plaintext as part of UDP - I can see it in your screenshot above

Karthick
It is encrypted at Controller before sending.

George
Yeah I know you encrypt it in the payload, that was my question, why are you encrypting something that is in plaintext elsewhere on the network traffic?

Karthick
In most cases, the visibility of the sender's IP address is not a concern. IP addresses are public and essential for routing data across networks. While it's possible to use techniques like VPNs or proxies to obscure your real IP address from the recipient of the data, these techniques are not directly related to the encryption of the UDP packet payload.

George
Did you ask ChatGPT for that?
Also, I'm talking about MAC Addresses, not IP addresses.
I don't really care if you encrypt it or not, I was curious what decision led to you encrypting a MAC address.

Karthick
ChatGPT gave a good answer 😃
I cannot see MAC address of Wireshark UDP packet trace done in my laptop.

Govind
From the Pen test, it is reported that the
"Disclosing MAC Addresses and display names of users provides a wealth of information to a potential attacker"
via panic alerts.
That's the reason we are encrypting the Mac address.
And I believe the Mac address showing on the Wireshark, is the system Mac address(the interceptor machine) on which the Wireshark tool is running.

George
All you had to say was "Pen testers" and I would have been happy 😄

How To Confuse Women

How to confuse women

  • Step 1 centre the text in a cell
  • Step 2 watch the confusion

Alright, it’s a somewhat clickbaity title, but I did cause a lot of confusion to one person with one simple feature, and she still didn’t understand when I felt I had perfectly explained it.

I had made a grid which shows a person’s name, alongside a list of items they have ordered, in alphabetical order; as per the requirements.

Later on, we would add alternating row colours to make different people’s orders more distinctive. In the example I came up with, there were only 2 orders on a page, and I had left the row selection highlight on, so it actually looked like we did have alternating colours.

The examples from the UX team only had a few items per person, so it wasn’t clear how they had aligned the text. I left it centred which looked a bit weird for large orders, which is the example I came up with.

PersonItems
Lisaphone
 CD
 hat
 lamp
Jamesphone
 shorts
 t-shirt
 towel
I don’t see how I can accurately illustrate this with WordPress. Imagine Lisa’s row is fully highlighted

 Just to show my progress on this new project, I posted a screenshot to generate a bit of hype 😉

Olivia was confused about why it wasn’t in alphabetical order – but it is in alphabetical order!

Olivia: why is “phone” at the top?
 
Me: Ulrika's designs or my screenshot? Mine is alphabetical. Phone is for a different Person. We still need to add the alternating colours, and maybe don't centre the name to make it clearer
 
Olivia: this one <sends screenshot of top row>
 
Me: Different Person
  
Olivia: I'm confused. Lisa's name is the same row as Phone
 
Me: There's 2 People there. Just that the second person has loads of items, and because we have centred the Person name, it's halfway down the row. We can use alternate row colours as Ulrika’s design, and we can stop centering the name to make it clear.

I message Tim:

Me: I can't believe the confusion that has arisen even though I have explained what it is.
 
Tim: I got your back

<Back in the main chat>

Tim: Phone is for Lisa there
 
Tim: Everything else is for James
 
Tim: Which is when the alphabetical order kicks in.
 
Olivia: Thank you Tim, That was seriously messing with my mind. The name needs to be at the start of the list of items.

<Back to private message with Tim>

Me: I cannot understand why my explanation wasn't good enough. Is it because I am a developer nerd and cannot communicate with people?
 
Tim <jestingly>: the trick is to keep your answers on one line. Colours and shapes help too.

I’m so baffled. Why couldn’t they understand what I wrote? I wrote a perfect explanation then Tim just put his explanation in short messages, and he was thanked for his explanation. It’s like I don’t exist!

16 weeks work

Some teams post fortnightly updates on what they have done on our internal social media platform Microsoft’s Yammer (Now named Viva Engage). No matter what they post, managers seem to respond with positive comments.

So here is an example. On team stated they had completed:

3 user stories. 1 dev investigation. Improved code coverage.

Then attached was there code analysis report which stated:

  • 0 Bugs
  • 0 Code smells
  • 100% coverage on 1 new lines to cover
  • 0% duplications on 8 new lines

I never know how to make sense of these stats. The duplications mention 8 new lines, but yet the code coverage mentions 1 new line. It can’t mean it is still to cover when it states they have 100% coverage.

Their video demo stated it was “Sprint 8” which means it’s after 16 weeks of work. They logged into their application and there are 2 buttons (not functional).

My Work

I’ve been working on a game in Unity and have also reached the 16 week mark. I have only been working on it sporadically. Early on, I might have worked a few hours and many days per week, sometimes working several hours at a weekend. Other weeks, I have just worked on it for 3 hours or so in one afternoon only.

I have written loads of code for the AI state machine. Your units can automatically move in a formation, break away in certain conditions. Then the defending group of units have their own behaviours, and work together in their own way. Some behaviour is configurable so I have the logic and UI for the player to change it. There’s controls for the cameras, game speed, pausing, save and reloading. A few options for volume and a few graphical effects. There’s some miscellaneous aesthetics such as animations, particle effects.

I am a single developer working fewer hours, and I have to play-test it myself. Compared to a team with a few developers, a few testers and different managers assigned – all working a 9-5 day, 5 days a week; and all they have is a menu and a title banner.

Hackathon Progress

Another comparison is to compare the progress of developers during a “hackathon”, or “game jam”.  Developers can often put together a prototype, or a fairly fun game within a few days…

Developers during hackathon: We built an entire application in just 3 days. Developers after hackathon: Adding that icon is going to take 3 weeks.

Mark Dalgleish

Conclusion

Recently, I’ve been thinking about this a lot. I think if you start with no code, then any code you write has a much bigger impact. You also don’t have to work out how it works alongside other code.

So the more code there is already, then the longer it takes to write. So if the Hackathons are productive, then you can write something fast. Same situation with my Unity game, and since I am working on it myself, then I have no one slowing me down.

The other reason why working on my own is great is that I can set the process. Due to the 90/10 rule, I think it is best just to get it 90% done, then move on. If you change your mind of how something works, it isn’t as big of a time waste. If it turns out to be good, then you improve it later, alongside other features when you are changing code in that area.

So going back to this original team who did 16 weeks work and got nowhere – even though it’s a new software product, and should be able to code fast – I think they are slowed down by their insistence that everything has to be perfect and has all the extras like Unit Tests. So they could get a feature 90% complete quickly, but then they are told to spend several days perfecting it for that last 10% of quality. Then they are messing around with non-functional stuff like the Deployment Pipelines and miscellaneous processes. When you are months away from releasing, I don’t see the point in dedicating so much time to how you are going to deploy it, so I have been 100% focussed on features and gameplay.

I think this could illustrate how process can severely slow you down, and striving for perfection can also lead to inefficiencies. I think my idea of “90% quality is good enough” could be a winning approach, then you can always perfect the features if you are ahead of schedule. If I keep working on my game, I will worry about how to release it when it is close to a releasable state (ie how to get your game listed on Steam).

Failing in different ways

I occasionally meet up with my old university mates. One friend works for a contracting company. He is a really good developer on a juicy £100k wage which is crazy, and it shows how much money companies waste on software on short contracts. Often they may only have small, permanent development teams, then hire some temporary staff for extra capacity, or they might just fully outsource their software requests entirely.

Since he works on short contracts, often 3-6 months, (sometimes up to a year), he has experience seeing how many companies operate.

I love having discussions with him because he is incredibly knowledgeable and always keeps up with the latest jargon. He knows all the Cloud and Web-development jargon, popular software tools, and Agile process.

He came out with this statement:

“It’s reassuring that you get to work with different companies doing different things, but they are all terrible. You are often asked to help – not because it’s going well; but because it’s going wrong”

What he was saying is that companies have different philosophies when it comes to software. It could be Cloud only, on-prem only, strict Agile development, different levels of automation etc, but whatever they do, it doesn’t quite work, and they will make baffling decisions.

So when I write about the baffling decisions where I work, it’s just that we are failing in a different way to other companies.

Angry Unicorn

I was getting a 503 error when trying to use Git. After searching for internet for solutions, I came across this “unicorns have taken over Error – 503“.  I wondered what that was all about? Was it one of those stupid error pages some companies do as a joke?

After a bit more searching, I came across another one, Title: “Intermittent angry unicorn”, and it’s clearer what they are on about 

“I’ve been subject to the angry unicorn for over a week now.”

That angry unicorn sure does look fierce.

Commit Messages

As a developer, you need to “commit” your work into source control. When you do this, you need to provide a message that describes the change. A bug fix or enhancement can consist of a few commits. When you use a source control like Git, you can even “rewrite” the “history” by using squash/rebase commands.

When I am writing code for my job, I often put a bit of thought into the wording to make it clear and professional. If it is for a personal side-project, I sometimes write meaningful messages, but sometimes I may group together multiple fixes, then come up with something generic like “Bug fixes”, or might commit more experimental work-in-progress under a name like “attempt” or “x feature part 1”.

It’s quite frustrating though to see colleagues write generic messages like “bug fix” which doesn’t describe what it is fixing, or how it is fixing it. Seeing messages littered with spelling mistakes is also annoying and unprofessional.

Examples include:

  • “EventNotificationBugFix After Resolving James’ Comment”
  • “bug resolved”
  • “Dev test changes from tester” (literally what does that mean?)
  • Updated the findCorrectCodeSp to findCurrectCode.
  • Taken off completly the fix Amanda done for the fix for 25477 and fixed. Included the fix for 8017 as well
  • Fix for SQLCrop issues (should be SQL Cop, our Code Analysis)
  • Fioxed further typos (ironically creating more typos when fixing typos)
  • fixed the failed unit testes (testes rather than tests. Brilliant)
  • “Post Christ’ comments re coding standards” (it’s the second coming of Christ to teach us coding standards! They meant to write Chris.)

There was a guy who worked in our short-lived Scotland office who sounded like an absolute nutcase and I have never seen someone not take their job seriously like this:

  • instructions unclear, got dick stuck in tfs 
  • what a waste of eyes
  • but fuck wasps
  • be nice to bees
  • what if there were landmines in dinosaur times

A colleague recently showed me this website https://whatthecommit.com/. I don’t know if they are based on real messages, but it shows you a new message every time you refresh. Most are pretty basic along the lines of “does this work”, but there’s some more outlandish ones if you persevere and refresh many times.

One of my team members recently submitted a change that was labelled “Crash when cancelling out of the dialog”. That describes the bug that he fixed, rather than what he fixed. Another team member provided the following good advice:

The change looks good, but your commit message isn’t describing the change. Could you just reword it so that it completes the sentence “When applied, this commit will…” please. Another way of looking at it is that every commit should read like an instruction in a to-do list. I’d use something like “Return empty collection if user cancels out of the dialog”.

Senior Developer

A stricter idea of commit messages is described here: https://www.conventionalcommits.org/en/v1.0.0/

One of our Principal Developers loves linking people to this guide https://cbea.ms/git-commit/. One interesting idea is to “Use the imperative mood in the subject line”, whereas most people would write in the past tense.

Having a clear history of file changes helps you when it comes to finding which change caused an issue, and also gives you a better understanding why it changed.

Errors logging errors

Today, I saw some code that looked like the classic “I can’t recreate this bug, so I’ll write some code to log extra information, and I’ll hope this somehow helps me diagnose the issue”.

They had a list of data, and the ID in this list should be unique, but it was in there twice due to a bug. So it would look like something like this (ID =1 is in there twice):

private List<Code> _codes = new List<Code>()
			{
				new Code
				{
					IsSelectable = true,
					ID = 1
				},
				new Code
				{
					IsSelectable = true,
					ID = 1
				},
				new Code
				{
					IsSelectable = false,
					ID = 2
				},
				new Code
				{
					IsSelectable = true,
					ID = 3
				}
			};

Then their method contained this code to validate the data:

		private void EnsureValid(int idToFind)
		{
			try
			{
				_codes.Single(s => s.ID == idToFind);
			}
			catch (InvalidOperationException)
			{
				Dictionary<string, object> customData = new Dictionary<string, object>
					{
						{ "ID", _codes.ToDictionary(x=>x.ID) }
					};

                LogToErrorDB(customData);
			}
		}

The Single will throw an exception due to there being more than one matching element (Single will succeed when there is a Single match). In the catch block, they then convert the code list to a dictionary and will log this in the error database. However, a dictionary requires the keys in the dictionary to be unique. Since there’s duplicate IDs, this will throw an exception again, this time with:

“System.ArgumentException: ‘An item with the same key has already been added.'”

Error thrown in the catch block

So you get an error whilst trying to log your error. The extra information they intended to log is never logged.

I’ve seen this mistake happen a few times before, and usually you can test it by artificially creating the data condition. Then they would realise their error logging doesn’t work.

Computers cannot handle floating point maths

I’ve been writing a game using Unity, and I’ve just realised how rare it is to use floating point (decimal) numbers in my day job as a software developer. I remember my Computing teacher back in College stating how computers are rubbish at calculating decimal numbers, but I never really understood. Calculators work just fine. They may only have several decimal places but that is usually more decimal places than you need.

In game development, decimal numbers are frequently used for sizes and positions in a 3D space and therefore calculations with these objects involve lots of decimal maths.

After writing some logic to position my characters on the map, I tried writing a unit test. Unit testing this method could be beneficial when I add extra features later.

I wanted to compare a Vector which is composed of 3 floating point numbers. Since I already had written the logic and was happy with the character’s positioning, I just grabbed the output of the method and specified it as the expected output, but my test failed!

Expected: (-34.28, 0.00, 20.63)
But was:  (-34.28, 0.00, 20.63)

so then I inspected the Vectors more closely

"(-34.28, 0.00, 20.63)"
	magnitude: 40.00208
	normalized: "(-0.86, 0.00, 0.52)"
	sqrMagnitude: 1600.166
	x: -34.275
	y: 0
	z: 20.625
 
new Vector3(-34.28f, 0.00f, 20.63f)
"(-34.28, 0.00, 20.63)"
	magnitude: 40.00894
	normalized: "(-0.86, 0.00, 0.52)"
	sqrMagnitude: 1600.715
	x: -34.28
	y: 0
	z: 20.63

I think the numbers it shows you is a text representation which involves rounding the numbers to 2 decimal places. But if you look at the actual X, one is -34.275 and the other is -34.28. So there’s a 0.005 difference. A similar problem with Z.

What also threw me, is when I looked at the documentation, https://docs.unity3d.com/ScriptReference/Vector3.Equals.html  I saw this:

“Due to floating point inaccuracies, this might return false for vectors which are essentially (but not exactly) equal. Use the == operator to test two vectors for approximate equality.”

I was originally using Assert.Equals(expected, actual) so I switched to Assert.True(expected == actual) but it still wasn’t working.

I looked at the Unity source code and they are providing their own == operator for vectors that does some “square magnitude” calculation then compares if it is less than a certain number – a margin of error. 

this is Unity’s version

[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static bool operator ==(Vector3 lhs, Vector3 rhs)
{
   float num = lhs.x - rhs.x;
   float num2 = lhs.y - rhs.y;
   float num3 = lhs.z - rhs.z;
   float num4 = num * num + num2 * num2 + num3 * num3;
   return num4 < 9.99999944E-11f;
}

So I did something similar and increased the number. I came up with this, using Unity’s provided square magnitude function, but wasn’t sure if it made mathematical sense: It worked for my scenarios though

private static void AssertAreApproximatelyEqualDueToFloatingPointImprecision(Vector3 expected, Vector3 actual)
{
   var difference = Vector3.SqrMagnitude(actual - expected);
   Assert.IsTrue(difference <= 5.0E-04);

/*
eg
0.0004996415
<=
0.00050
*/
}

Looking back at this, I probably over complicated it. I suppose I should just change the expected values to have more decimal places because it’s the rounding to 2 decimal places that will be the actual problem in this case. In other cases with more decimal points, you will encounter issues due to the way computers do handle large decimal numbers which the Unity docs are alluding to.

In other situations, you also need some leeway in your calculations. If you want to move a character to a certain position, you may find that they could be 0.1 away from their destination, then on the next frame, they move their usual distance based on their movement speed – and then overshoot it. So you end up needing to write code like “if distanceRemaining < 0.2f then stop”

So floating point numbers, especially when it comes to game development, really does add extra complications to think about.