Hi, I'm Ciara Carey and welcome to Cloudsmith's monthly webinar on all things supply chain security and package management. Cloudsmith is the, your only cloud native universal artifact management platform. We support all your packages and we have lots of features around supply chain. So today we're going to be talking about a real world, zero day supply chain attack and the response to the attack.

So we're really lucky to be joined by David Gonzalez, who will walk us through the supply chain attack and a really popular node package called event stream. So. David is a principal engineer at Cloudsmith. He's the author of many books, including developing microservices with Node. js. He's a Google Developer Expert, a part time lecturer, and he's also a member of the Node.

js Security Working Group. And this is where he's experienced a lot of vulnerabilities and how to get them converted to a CV and get the wider community updating. So let's bring them on. Hi. Hello. Hey, we're so delighted to have you. And we're, David is a new hire for CloudSwiss. So we're really happy to have him on the

team.

Yeah, me too. I'm very Yeah,

so so all we're going to be talking about something they experienced on your Node. js security working group, maybe first we could talk about what you do on that group. What's the aim of the

group? Yeah, spoiler alert, I haven't been very active in the last couple of years because, you know, like, kids, COVID and life happens, but usually it's like the process hasn't changed.

Like, basically what happens is somebody reports a vulnerability in HackerRank. HackerRank is a third party website. Where they just ban security programs for companies, uh, HackerRank, HackerOne actually, not HackerRank. Oh yeah. Yeah. And that landed into our inbox, the team on the security working group in Node.

js, and then we will triage the vulnerabilities. So that means verify the vulnerability. I can assign, assign a CVSS score, like how serious this vulnerability is and things like that, and then contact the package maintainer. From the report up to the point that the package maintainer either fixes or don't want to fix the vulnerability, that information is private, so you can't or you shouldn't even talk about it.

That report because it's a security issue. Somebody was using the package and then usually the normal route for that is the maintainer gets contacted They acknowledge the vulnerability and publish a patch for it. And then after that you just request a cd. So create a like a CVE for that package for the given versions, create a report and then publish it after it's fixed.

So the recommendation is great to the new version or versions, depending on the package and you are done. So we go from somebody noticing something into requesting the CVE and publishing the report, how to reproduce it and what's the effect of that vulnerability. Cool,

and we're talking today about a zero day vulnerability.

Are all vulnerabilities zero day until they get a CVE score? No,

no. The CTO day term to me has always been confusing. CTO day to me seems like, you know, something which is like almost a nuclear explosion in the security chain. But like, you know, in reality, most of the CTO days. This might not be even impactful in many systems because like the network topology or the way they are using the library.

It's something that has the potential to cause a lot of damage, a lot of harm in a, in a system. Like in particular, you know, like I handled a couple of those and one of them was you could get a shell straight away to execute commands via the HTTP interface. Yeah. So that's, that's bad. That's pretty bad.

But if you are in an enclosed ecosystem, like you are working in a private net that doesn't have access to the internet, it's not that bad. I mean, it's terrible, but it's not going to destroy your company.

Yeah. And so let's focus in on this particular supply chain attack. It's event stream and can you walk us through how you found out about it?

And yeah,

yeah. Yeah. So what happened is somebody raised a ticket or an issue in GitHub saying I fold something in Well, it wasn't really in the event stream. I think it was in flat map stream, which is a dependency of event stream. That's why. And then everybody started commenting that thing that they found these malicious code that was trying to steal crypto wallets from the user's machine.

And, you know, like if the wallet was unlocked and you execute. That package, or functions of this package, it will go there and read your wallet, the private keys, and things like that, and send them. If I recalculate it to a server... Or something like that. So we saw the GitHub incident, like the GitHub issue.

And we started looking into it because that's a fairly serious situation. MPM.

I was checking like straight away, this guy was like getting money basically straight from this dependency.

Yeah. And not only getting money, like we didn't know the extent of that code, that malicious code, would that be just getting money out of crypto wallets?

Could that be stealing passwords, stealing, you know, like files from. Like encrypting computers and then demanding a ransomware, like a, like a payment to encrypt the computers. So we, we started looking into it and then we, we saw that what happened pretty much was the FlatMapStream package was handed over to somebody else because the maintainer won't have time to maintain it anymore.

And that new person actually published a new version, which apparently was harmless. Fixing a couple of books, but it contained that code, like it was cheeky. So if you look at GitHub, like the link you had in GitHub in NPM, like the code wasn't matching the NPM package. And then basically like you will click on NPM going to GitHub, see the code, the code will be like completely different.

And then the package you download will be. Would be the malicious codes. So

even

if you are like, you were really vigilant and you were like, oh, there's a new version I'm gonna check, do do a, that's check's. You would be, you wouldn't see

this, this. That's how they get you. Yes. so cheeky. Yeah. Yeah. And that raises an interesting question is like the work I did, and probably we'll get back to in the security working group, it's not something.

I'm getting paid for it. It's not something I'm getting anything other than professional realization and, you know, enjoyment out of it. And it's hard, like, I mean, it's, it's really hard. So like you can build a tower, but pretty much that's always. resting in a very, very narrow surface, which is people willing to do work for free and without any, you know, like kind of like a reward other than professional recognition and things like that.

And it's dangerous. I believe the third party ecosystem now is managed by Snyk. But, like, it's not the only leg of the security ecosystem that, that is standing around. So, yeah, it's an interesting one, because what happened then is, like, Hacker 1 report was raised, and then we immediately managed that. We gave the maximum CVSS score, because it was like, doesn't matter what you do, you will always be affected.

And then we issue the CVE straight away. Now, the time from where that is discovered to the time where the CVE is issued, it's a very dangerous time because your security scanner will come back to you telling you, Ah, you're fine, you have no issues. Yeah, because it's

not a known vulnerability, so... Exactly, yes.

You might also have, you know, a lot of the time you're told to update as soon as possible and that is, you know, and so if you will bring in that update and actually bring in that zero day vulnerability, if

you ask me I always keep the same advice is your version should be locked to the patch version.

So you have. x. y. z, you will log to x. y, but then leave the point set in the version to just upgrade as the new versions are released. So that means anybody who will have followed my advice will be affected by that because it was a minor, like it was a patch version. And I still think it's the right thing to do.

Yes. I've been

looking a lot about this, like. Are you meant to like pin, is it more important to pin or to update to latest? And I think it's, yeah, yeah. At minimum. Latest seems to be what they're, they're saying.

Yeah. Updating to the latest version is always recommended, but sometimes you just can't because you know, you are stretched or you just can't make a big change to the application right now.

So that's why I suggest being to the minor version and then leave the patch version. So it's, yeah, major, minor, and then patch. So pay to the minor version and then leave the patch version to evolve with the library because that's going to fix bugs and security incidents, security vulnerabilities.

They should all be fixed in patch versions. You should never, honestly, it's a good reason to fix a vulnerability in a minor or major version. You should always have patch versions for all your... Versions that are affected by, by this. So let's imagine that you have library XYZ version 4. 0, 5. 0. And a vulnerability is discovered.

You will need to create a version 4. 1 and 5. 0, sorry, 4. 0. 1, 5. 0. 1 to fix the problem in the two versions. So it's, it's tricky because if you follow the best practices, you will be affected by this one. And by the next one's coming, which is not going to be. The last one. So,

yeah, but so can you tell us what, what were what were the initial, I suppose the initial signs were that you got, you got a some actually found it.

Yeah. So, and then afterwards it's detected. You decided. What was the score to, to, to give this vulnerability? What score did you give it? I think it's a big one. It's

a 10 out of 10. So like that, there's a calculator called the CBSS score calculator somewhere in a, in a URL. Like if you Google for CBSS V2 calculator, it will be straight away there.

And you can specify all the vectors of the system and it will give you back a score. So this one in particular, it's like, you know, you don't need to do anything. Just install the package. And as you run your system, you are immediately compromised. So, yeah, it's it's an interesting one because there's very little you can do other than using a different version.

Now, there's a good point here. Like, NPM took the package down very quickly. But how many enterprises who cache dependencies, still nowadays, three years, four years later, have a copy of that

version? It's like these vulnerabilities have really long tail. I saw something about like Heartbleed, which is like from 10 years ago or something.

DLS, yes. Yeah, it's some bug in OpenSSL. And it was, there was a report from like, say, six years later, six years after it was patched now. And people, it, it was still an attack vector for attackers to get into your system. So, you know, it's, it's not just about discovering the vulnerability. It's about making sure that you're, that you're patching and you have a good process for, for finding these vulnerabilities.

Yeah,

exactly. I can give you another one. Log4j in Java. There was a serious vulnerability in Log4j like two or three years ago. And I can tell you, I was talking last week to a friend who works on embed. And he was telling me, it's like the only possible way I have to fix that vulnerability in my system is to replace all the hardware because the library comes in a, in a, in a memory, which is read only, so I can't possibly upgrade the system without updating everything.

Oh God, yeah. That's a bit, apparently it's on Mars as well. The Mars rover lander thing has it, so it's like it's an intergalactic problem.

So with the, so how do you, so you've discovered the vulnerability, you've verified it, you've created a CVE for it, so now if you're People's vulnerability databases should be updated and what's, how do people, how do the wider community know that they need to patch it? Is that they have to be monitoring the database or was there some sort of communication outside of that?

It's just how you manage your software supply chain. My advice is always scan for your dependencies. Because like, vulnerabilities in your dependencies. The last thing you want is not knowing what's going on, because you can't really make any decision. You might decide that this vulnerability does not affect me, so I'm going to release into production anyway.

Okay. But you make an an informed decision on, this is a risk, I know about it, I have thread model my system, and this risk is actually very low on my context. But if you don't know what's going on in your dependencies, you are actually taking a big risk. So my advice is always scan for vulnerabilities, all your dependencies and transitive dependencies, and then after that make a decision.

I heard through the years, things like you should always fix, you know, medium plus vulnerabilities, and you shouldn't do any release with any medium plus CPSS scores vulnerabilities. I don't necessarily agree with that. I think yes, medium plus should be fixed if possible. But sometimes you are okay to leave with critical vulnerabilities that don't affect you.

So I can give you one example, one package that has two or three features. There's one critical vulnerability in one of the features and you are not using it. It doesn't affect you.

Yeah, actually, this is there's a new for, I know, have you heard about software bill of materials? It's like an ingredient list.

So it might say, oh, you're vulnerable, but there's another format called VEX, which is meant to to like tell people, yes, I have this package, but I'm not vulnerable. And you should give a reason for why you're not vulnerable. Vulnerable. They're kind of working on workflows for these to, to work together, but it's hard.

Yeah. It's almost like the best way a phone to model that it's on a low list and a block list. So you allow certain vulnerabilities to go and flow through the system. Whereas some others is like, Nope, this is an immediate. Nope. So. Yeah, that's the best way I have to, and to do that you need a software build of materials applications.

I am following a project for a few years called Grafeas, which was started by Google, which exactly does that. You can add materials to your software and you have a kind of a metadata collection of your software. But I still haven't seen it working 100%.

So, yeah, I want to see a lot more work workflows around that, like kind of with your CICD, like automation, like I don't want to be doing, I suppose you, you do have to, if you're, in fact, you still have to like.

I, I can't see the automation where it says like, I'm not vulnerable. That's the, wouldn't that be great. We'd be all sucking diesel. If if that.

It also is interesting because you scan today, the scan comes back saying you are not vulnerable. And then tomorrow there's a new vulnerability, one of your packages.

How did you know about it? The only possible ways to re scan all your repositories. So

yeah, yeah, you have to like continuously scan, even if you're featured complete and all, all done on your product, you probably still need to be scanning that if people are using it out in the wild. Yep, correct. And do you think that NPM ecosystem, because it seems like some ecosystems have more of these type of vulnerabilities, like Like package related ones like, like maintain or hijack, typosquatting, dependency confusion.

It seems like NPM, maybe Ruby, maybe PyPy, Python are more susceptible to that. Do you think it's, it's just because there's so ma The community is more used to consuming them from public repositories more used to small small packages being, being brought into software, or is there something fundamentally about these systems that are that.

I can, I can give you one reason for that, which is, it's just my opinion. So, you know, like anybody disregard that, but I come from a Java background. When I was in Java, I would install a spring, I would install Ivernite and that's pretty much everything I needed. And then those libraries will come from a reputed, like somebody who.

Yeah, like

a kind of someone that's, that's like Red Hat or, you know, where there's a community making a curated list of, of libraries.

And it's, it has a lot of exposure, like all the dependencies are concentrated there. Like then, you know, Waba from Google started showing up and that's fine. You know, you started using Waba.

But the reality is you had a list of dependencies, but they all came from the Apache Foundation from, you know, Spring from others. And yes, there were vulnerabilities, but. There was pretty much no events, like somebody abandoning a package. Like, I mean, about two, three years ago I was working in a project and I just looked into the dependency of an OJS application and there was one critical vulnerability of one of the packages.

And then I, I just went to GI Hop and that guy, the last time he dated the library was like six years ago. And somebody in an initial was saying. There are no new versions coming and the guy came back to us saying, sorry, I just stopped programming. I am doing something else, so I'm not going to do that.

And I don't want to give access to anybody to maintain the library. So it's the full system which companies spend millions on securing, depending on a small library which somebody is maintaining or not. That's, that's very common in Node. js. I don't see that happening in, in Java. I think the community in Node.

js is amazing. But I also think that exposure you get by... Installing dependencies on npm. It's almost like well, it's almost no it's exactly taking someone's code which hasn't been vetted Nobody has enforced any quality on it and pasting it in your system. I A few years back. I read a comment, which is something like only 17 of the code That your room in your production servers is written by you I would be willing to say that anybody using fastify express or similar If you go above 5%, you will be doing well.

So,

there's a risk there. I think I don't see that changing. I don't think, I know the supply chain attacks are coming to light and people are kind of getting a grip to them. But I don't see us saying, oh no, I'm going to rewrite Kubernetes because... Yeah, it's, it's too hard and you gain so much from using open source.

It's, it's quite, it's really positive, but there needs to be probably more checks on and on how to maintain, how to reduce your vulnerabilities and know what you're using as well. That a big thing is like, I don't even know what I'm using. It's the

80 20 percent rule. You heard about that, right? So 80 percent of the features will take you 20 percent of the time.

The other 20 percent of the features will take 80 percent of your time. It's the same in security. So you can do only so much to protect yourself from vulnerabilities. And that's going to take you 20%. The other 80 percent will be going the last 200 meters, which actually are going to be pretty much impossible to breach the gap for.

So my advice is always, yes, you have to protect against vulnerabilities, but you need to be able to react. When something happens, and that's the other 80 percent that can be done very quickly. So, one, one angle of this is, yes, scan for dependencies, yes, make sure you have the software build of materials or at least know what you are running in production, but if tomorrow there is a CEO day, you need to be prepared to even shut down the system if necessary.

I mean, I work in systems where they decided to shoot it, shoot down the application when something like that happened, because the damage that could have been done to the app or to the company was way bigger than shooting it down for three, four hours, fix it, and then bring it back again. It's an option.

So, yeah, and I'm actually I'm looking into this framework called S2C2F. It's like a framework for consuming open source securely. And a lot of that is like, you know no warrior. Open source coming from keep an inventory like an S bomb, you know, use an artifact repository and also incident response, which I think you're alluding to there were like what you do when something does happen.

And how do you make sure that you quickly update your vulnerabilities. So I think like it's all the stuff that you're saying there. Yeah,

I mean, I would like to do an exercise at some point, which is. You know, any of the Node. js applications that I, I work with and the open source projects I worked on, see how many contributors are in my dependencies.

I can tell you that at least 40 to 50 on average, at the minimum, like it's, it's just crazy. Like, I mean, if you go to any chief security officer in a company and tell, can I take calls from this random guy on the street and put it in my system? The answer would be no. But if you do npm install, it's fine.

And I know they're actually working on this providence. Have you, have you heard anything about that? No. Like how you can inject information about how this package was built and who built it. And I think that would improve some things. Like maybe that would help against this particular type of attack and event stream.

Well,

yeah, one thing I suggested a few years ago is. It's having a flag in Node. js to sign packages, like only accept signed packages, and if there is any signature change, you just raise an alarm saying, hey, I'm not going to run this, because this package was maintained by Mateo Colina, and now it's maintained by David González.

And I don't trust this guy, or I might trust him, but I don't know if I trust him or not. So just bail and then see what's going on. But that, that's not obvious to implement and requires people to collaborate. And 90 percent of people on NPM will be like, you know, I'm just some student in college. Well, not 90, but a good number.

Like I'm just a student in college, which I thought this was going to be useful for somebody else. I just publish it. I'm not going to look at it ever again. Yeah. I created a library a few years back called 11StateMap. The only thing that does is you can create a hash map, and then you can get keys by an approximate string.

So if I misspell my name, I will still get my key. And like about six months ago, I got an email from somebody saying, can you add this functionality? And it was like, somebody using this. So

that's, yeah, I suppose it's abandoned software is a big, I think it's a huge amount of, of packages have one or no maintainers.

I think most packages have no maintainers really, where like there's been no activity for over a year and nobody's updating those vulnerabilities.

Yeah, and there's one even worse, like packages which look maintained, seem maintained, and they have maintainers and you approach them with a vulnerability and they tell you this is not a vulnerability and I'm going to fix it.

And then you are like, well, I'm going to issue. What do you do? I issue a CVE at my discretion, and then tell them, here's a CVE, it's up to you not to fix it, but this is flagged as a vulnerability. So,

yeah, yeah,

it's, it's got, it rarely happens, but I've seen it happening with very popular libraries in a

beer.

So, and I suppose, you know, maintainers are people and this is not their day job. So there's lots of reasons why they don't want to update it. They might personally think that it wasn't a vulnerability because they don't think it should be used that way or but yeah. You know, I, I, as well, this framework that I'm looking at, it's like at the highest level, level four of S2C2F, it's it actually recommends if you see a vulnerability to have the ability to like fork the code and then fix it with the, with the hope of it being merged.

At a later stage. So it's like a temporary fix, the highest level at the moment. I've

done that a couple of times. Never did subquit.

No, no, it's hard maintaining two different things. It means that you're on different paths and it's not great. So, so is there any way to find this type of attack? The maintainer is a legitimate maintainer, according to us.

But I suppose what you were saying is you could have a score attached to this new maintainer to say they're not as trustworthy as it was before. And you have to make a decision on that. I suppose that would help.

Not at the moment, to be honest. I mean, there's not any mechanism to enforce that at the moment.

There's not any mechanism that is going to give you a number of. Like, humans are unidimensional, like, you know, people don't want to see complicated matrices, they want to see a number, like, this person is a 10 out of 10, I trust that person. But even then, like, I mean, simple things like, somebody who is very trustable, and then that person adds somebody else as a maintainer, because...

That person is collaborating to the library and, you know, like, they just add it as a maintainer to give a hand. And then you gain the trust of the original maintainer, and then you can publish whatever you want. So, you know, even if the package is in my name and you trust me, I might have given access to somebody else to publish new versions without me.

Looking into it. So

even like maintainers that were, you know, good maintainers for a very long time can make decisions that would affect people using it. Like the colors and fakers that package where I think they our left part is when they removed it, right? Yeah. So that's an interesting one.

So that's, That's not a vulnerability as such, but somebody removed, I think it was left out, I think you're right. And that caused about 95 percent of all the applications in Node. js in the world to fail. Because in any way or another, everybody's using LeftPath. So is that not worse than having a vulnerability?

Because if you can't build your application, you're actually screwed. Because you basically need to be able to release at any time. What happens if suddenly there is a new vulnerability? Or it's a coordinated thing. Like I remove LeftPath and then somebody else discloses a vulnerability on another package.

Which depends on LeftPath. It's a disaster. Like it's an absolute disaster.

So all these things are where a maintainer, they legitimately did something now, you know, obviously with event stream or with flop pack, it was, it was malware, but with these other packages, they were allowed to do it and being able to discover those things.

I suppose the best you can do the most is to update as soon as you get that vulnerability in.

Yep. And you know, like you will. So you could get hit by things like flat map stream if you date and keep up to date, but the alternative is not doing it and you get hit by everything else. So you just choose which one's

worse.

And have an incident response plan so you know that this could happen and we know how to fix it. Well,

my initial reaction to it is make sure that can be fixed. Like if it cannot be fixed, consider what's the worst thing that could happen to my system. Somebody uses that. Like in this particular case, I will even fork the code and fix it.

It was easy. Just remove a couple of lines and that's it. But in other cases, it's just, you know, assess what's the potential plus radius and what would happen. And after that, it's like, you know, it's just a matter of. managing the information in your system. Security is always a problem of people accessing things that shouldn't be accessing.

That's it. Like, I mean, it's as simple as that, like somebody accessing my Bitcoin wallet, which I don't have any, but anyway, it's a problem. Somebody accessing my command terminal, it's a problem. So it's always is somebody accessing information that shouldn't be accessed by that person. So how do you stop that is the first question you need to ask.

And then What's the way to respond to that? It's another one more concern. I am more concerned about the response time rather than the response itself, because there's always a workaround that can be as simple as patch version of the library or can be as complicated as. You know, like rolling back to a previous version.

But the concern is companies that are releasing once every six months or once a quarter that has a multi day, multi week release process. That's a big problem. Because they are utterly exposed.

Brilliant. And so I think we'll end it there. So do you have any more thoughts on the future of supply chain

security?

Nope. Scan your dependencies, make sure that you can react in your system in less than a day. And, you know, be vigilant. That's the only three things that you can really do. Other than that, keep doing. Yeah,

well, and thanks for watching. This is Cloudsmith's monthly webinar and we'll see you again next month.

Thank you. Bye bye. Bye. Thanks David. Bye.