Self-inflicted problems

Instead of reinventing the wheel, Software Developers like reusing code. When using Javascript, a common place to use such code is NPM; Node Package Manager.

The recent NPM incident caused malware to be shared via many NPM packages. As far as I understood, I believe Github security tokens were stolen which allowed impersonation to check in malware and propagate to more packages.

As a precaution, we were told to cease development so you had hundreds of developers sitting idle.

We knew our teams didn’t use these packages directly but I suppose investigation was needed for transitive dependencies, and there was always the risk of packages we did use suddenly being infected, but we could just stop upgrading to new versions.

We thought it would only take a day or two, so I took the day off and would come back over the weekend to normality.

However more investigation was needed, then we had to switch over to a different internal NPM server with only approved packages being available.

I had no idea what the criteria is for approved packages. How do you know if something is secure other than the currently known, published security issues? Typically, bugs and fixes are added all the time. 

I think there was some kind lead time before new versions were accepted. I suppose it prevents this quick publication of malware if you just have a lead time of only taking packages more than a month old.

Since we had a new NPM server, we needed to publish packages to the new one, but we weren’t allowed to use the old ones. So new versions had to be published which then means you need to update all your software, and it’s a massive chain of dependencies involving all teams in the department.

I wasn’t personally involved but it sounded horrendous to coordinate and we were delayed by around 3.5 weeks.

We weren’t even affected by malware. We have loads of sub-companies in the overall group and no real incidents between them. Hundreds of developers were idle, and it was self-inflicted.

Leave a comment