From the history of supply chain security threats to security development and deployment we've covered everything you’ve always wanted to know about the software supply chain but were afraid to ask. Dan Lorenc, Founder/CEO, Chainguard, Paddy Carey, Senior Staff Engineer, Cloudsmith, Adil Leghari, Solutions Architect Manager, Cloudsmith and Dan McKinney, Developer Relations, Cloudsmith, gathered for a fireside chat to cover your most burning questions:
- What got us here? Types of attacks – availability, dependencies, development tools, and more.
- Why is this a hard problem? Software supply chain challenges and considerations
- What's being done? A look at OSS projects and initiatives that were born out of the SSC security need.
- What are we doing, and how can you help? What have Cloudsmith & ChainGuard been working on to make these issues easier to tackle, and what you and your organization can do to help.
Read the transcript of the full webinar below:
Introductions:
Adil Leghari: Hello, everyone. Good morning, afternoon, good evening. Depending on where you are in the world, on behalf of my colleagues here at Cloudsmith and Guard, I want to welcome you to our discussion entitled “Everything You Wanted to Know about Securing the Software Supply Chain, But Were Afraid to Ask”. So this is going to be an open discussion or conversation, so feel free to hop in, you know, with questions if you want to. We do have a Q&A section specifically in Zoom for that, so it helps if you throw your questions in there because it will.
Definitely, everybody will get the benefit of the answer if we answer it on the call. In addition to that, of course, like, you know, we're all here and I want to welcome you. But I think over the last few years, software supply chain security specifically has become top of mind as one of these concerns for almost every organization, right, that's producing and consuming software. So we are the tech community have to try and understand this. But if we're going to wrap our heads around it, we really need to have conversations like these, you know, to improve our collective understanding of where we came from and sort of where we're going. So hopefully, by the end of this talk, we'll leave with not just a bunch of fear, but hopefully a little hope as well about the future of supply chain security and software.
So let's go through the agenda real quick here. I will first off just do a quick little speaker introduction for all of us. Everybody will do their own introduction and then we'll give you a little background on what we do with our organizations, with Cloudsmith and Chainguard as well. And then we'll get into the sort of the meat of the torque where we're talking about, you know, some of the things that we want to bring up today. As and again, this will be open, sort of like a fireside chat. We're just discussing stuff. And so feel free to ask questions as well and help my fellow speakers feel free to hop in at any point.
We'll be touching on points like what got us here, specifically when we talk about previous tipping point events, sort of attacks, and other things that precipitated our big concern about the software supply chain. And then we'll touch on why this is such a hard problem to solve because there are definitely some challenges there. Each attacking each different compromise or issue that comes up really has different origins and different techniques and methods. So it can be quite tricky, sort of a moving target.
And then we'll talk about specifically what's being done where we highlight a few open-source projects in the space, specifically around the efforts that Dan Lorenc has done over at Chainguard and also at his time at Google and with open source. So thanks for being here with us, Dan. And then also we'll specifically talk a little bit about at the end about what are we doing at organizations like Cloudsmith and Chainguard, and also how you as an organization can help and contribute to the same efforts? Sound good?
OK, well, without further ado, let's introduce ourselves a little bit. So I'm Adil Leghari, I'm a solution architect manager here at Cloudsmith. I'm a sys admin turned solution architect turned manager. I spent the better part of two decades here wrangling Windows boxes and Linux boxes. So over the past five years, I've become an active member of the PowerShell and the devops automation community, and I'm a speaker and author and super passionate about PowerShell, dev ops and automation, yada yada yada over to you, Paddy.
Paddy Carey: I'm an engineer here at Cloudsmith and I've been an engineer for a long time, starting in the tech and e-commerce and finance, and eventually find myself deep in package manager land, which is how I ended up here at the Cloudsmith. I've worked on at everything here, from core package management functionality, securing supply chains and my own to get our platform team part of the responsibility there is securing our own supply chain.
So hopefully that's a relevant experience for me to talk about today.
Dan McKinney: Hi, everyone, yes, I'm Dan McKinney, and I work in developer relations at Cloudsmith and I've had a somewhat tortuous path through the developer relations. So I've worked in systems engineering and manufacturing, actually another type of supply chain as well. And yes, it's really my role to sort of engage with the community and educate the community, which is why this is a perfect forum to do this today.
I'm also an absolutely crazy and retro computing and video game collector, and I work as a DJ at the weekends. But yes, it brings a little bit of color to the proceedings. So nice. Thanks, Dan.
Dan Lorenc: Yeah, second Dan today. My name is Dan Lorenc. Currently, I'm a founder and CEO at a new startup called Chainguard. But before that, I was at Google for about nine years, where I got to spend the last five or six years working almost completely in Open-Source and started worrying about open-source supply chains. So I've been involved in a bunch of different open source projects there and continue to be in products like sig store and salsa products or SLSA. So we have to talk about some of those later today.
Adil Leghari: Absolutely. Thank you and thank you on behalf of all of us. Thank you for your efforts and for keeping things open for us. Awesome. OK, well, let's talk a little bit about each other as organizations, about Cloudsmith. I'm going to let Dan take this one.
Dan Mckinney: Yeah, thanks. So Cloudsmith is a cloud-native package management solution. So given how essential software artifacts and packages, in general, are to our software supply chain, it's very high on our agenda to secure that supply chain. So we have been around a lot longer than a lot of people think.
If they're only hearing of Cloudsmith, we've been around and this will be our sixth year. So it's a very exciting time for us because as you said earlier, it'll, you know, secure software supply chains are really at the forefront of people's minds at the moment. And it's so close to our hearts as well. So it's a really exciting time to be part of that. So that's who we are.
Dan Lorenc: Yeah. So yeah, we've not been around for very long. We've been around for about five months. Chainguard here. We got started in October of 2021, and we were trying to hold a bunch of to invest in open source to help people make secure make their software supply chain secure by default. We've seen time and time again that if you make developers do things differently if you make them change the way they work, security is always an afterthought. It's not what you're thinking about when you're in that development loop.
But we believe that if we do this the right way, and if you take a couple of steps back and think about the problem a little bit, we can actually make secure software development a better and easier way and have it work that way by default. So that's what we're focused on doing. We're still just getting started. So thanks a lot for having us here and giving me some time to talk to everybody.
Adil Leghari: Dan and I must say, you're being very humble. You guys are slowly picking up the who's who of Kubernetes and other SLSA supply chain projects. So good for you over there. We eagerly await your announcements.
All right. So let's hop in a little bit. And so now what I'm going to do is I'm going to take the slides off the screen because I think this is a discussion. So let's open it up a little bit and talk about what got us here, right?
What got us here?
SolarWinds
Adil Leghari: The scary elephant in the room is all the attacks and everything that went on right? But I think there are a couple of events that will probably highlight that have helped shape the industry as far as the software supply chain. And I think one of the first ones, of course, is the SolarWinds attack, right? So that was December 2020. SolarWinds, if you don't know, it was a very prominent company, that makes software for network infrastructure management. So in December of 2020, they announced that their Orion software had been compromised as a result of the software supply chain attack.
And so this is funny because this attack very much was managed to burn the word "software supply chain" into our collective consciousness, right? Because everybody now knows that word, that phrase specifically, because its malicious code was inserted into the Orion software early enough and that supply chain, it remained undetected. So what happened was that pushed out with the other updates to the Orion software and slowly got out to consumers. And you know, we're not talking about small consumers here, we're talking about big folks like Microsoft and Intel, etc. So little did they know that they were installing those malicious payloads along with the monthly updates.
That's really serving to highlight part of our problem, that up until this point, developers were largely ignoring these parts of the pipeline. This is sort of a tipping point event for that sort of alertness. Now, Dan Lorenc, I know I've heard you talk about this before, but I think you were still a Google working on some open-source projects at this point, right? And this is sort of one of those moments, those tipping points.
Dan Lorenc: Yes, this is the kind of right after I switched from Google Cloud to Google's kind of core security groups, I finally been successful in convincing some people at Google that this was a big enough risk to take seriously is like an existential issue for the internet and for Google. And it was still kind of slow, though, until, you know, the kind of news broke about this attack and the amount of people affected and everything. Almost overnight, we went from a "why is no one listening to us?" to "why haven't we been doing this for years?".
And was quickly followed up with, the stuff I think you're going to shout about in a minute with, you know, the U.S. government looking around to announce executive orders and regulations and new compliance and everything. So yeah, it has become an increasingly hot topic over the last couple of years, unfortunately, because of attacks and not because we were as an industry ahead of the game. But it tends to work that way, unfortunately.
Adil Leghari: Yeah, it's not a problem until it's everybody's problem, right?
So, yeah, that definitely was one of those moments, I think for everyone in the industry. I was working at the time, I believe at Chocolatey software and you know, we kept on getting that question again and again where suddenly folks were like "hey, you know, we need to secure software supply chains?" And I'm like, "OK, that's great". I mean, we have kind of been preaching this stuff for a little bit, but now suddenly we're getting all of this attention, right? So I think I think a lot of folks perked up their ears here and sort of your C-level staff to hire folks who are normally not as concerned suddenly started tuning in and asking these questions.
Dan Lorenc: Yeah, I've seen pictures of board decks now with this on a slide from every CSO out there, so it's something everybody's trying to rapidly come up with an answer to.
Adil Leghari: I know over the past year everyone's learned the acronym SBOM, and it's kind of funny because it's like, really, we've been talking about this for a while. So I think, yeah, leading onto that timeline was right.
So in February 2021, I think was one of these announcements, where Alex Birsan, who is a security researcher, showed that he was able to compromise supply chains as well. And he was luckily a white hat hacker. So we didn't have the benefit of stress or the negative effects of that. But of course, it had been copycat attacks since then, right?
I want to highlight this one specifically because I think he did this basically by, you know, sort of subverting the logic in the actual process of some of the normal supply chain, right? So as developers, a lot of times we want to use dependencies, right? Obviously, we don't want to reinvent the wheel every time. And so this is specifically referred to as dependency confusion. And it's similar to sort of typosquatting as another type of attack where, you know, essentially what he did was he introduced public packages into widely used public registries that shared the name of private packages for a lot of big organizations. And we're talking 35 different companies. We're talking names like Microsoft, Apple, PayPal, Tesla, you know, like it wasn't a small attack, but basically, he was able to very simply do that by creating packages on public registries with similar names.
So it highlighted another thing where it's the functionality of the package registries in the package management system by default are to trust the public registries and pull those down. But you know, it's the whole "who watches the watchmen" at that point, we're not sure. I mean, everybody had the overnight to kind of figure out. Wait a minute. We have to actually change the logic of how we do this stuff. Right, and I think Dan McKinney, you were mentioned. Yeah?
Dan McKinney: I mean, I think the real problem until there was that, I mean, essentially this was a problem with like configuration, the way people had configured the package manager clients to, you know, be able to hit public sources in preference to their private internal repositories. But there were no red flags like, you know, it went ahead. It wasn't that it was, you know, in contrast to SolarWinds, which was a breach of their build system. This was normally running build systems that just were, you know, configured incorrectly but didn't shout about it when something unexpected happened. So, because pipelines are so automated now, that's what enabled that, you know, the automation kicked in, packages get pulled in, things get built. And there was no point where something shouted to everybody.
And why is why was there no point when that happened? Kind of like Dan said earlier that it's not the default and it's not easy enough to have that default stance of, you know, something unexpected or there's a problem or there's something we suspect may be a problem, and maybe we should have eyes on this. The heavy automation sort of drove that. And look, I'm a huge fan of automation. I mean, obviously, automation is what enables, you know, companies to move with a real, high velocity and deliver software quickly.
And that's fantastic as an industry. But it has to come with checks and balances because, you know, automation run amok well will lead to this kind of problem. That's, I think, a core difference between dependency confusion and the SolarWinds attack. Both were supply chain breaches, right? And both resulted in malicious or potentially malicious software getting out into the hands of users. But there's a subtle difference there. The lack of visibility on that process and the lack of alerting and automatic red flags, is what really caused the problem.
What is the "software supply chain"?
Adil Leghari: Mm-Hmm. OK. So this is great because I'm going to pick on you a little bit and say so because I love you. So we have a Q&A question. So while you're at this talking about this specific software? Moussa asks, "starting from the basics, can you please define what is meant by software supply chain?" So defining our terms, of course, where everybody wants to be on the same team?
Dan McKinney: And I'll have a go with that then, yes, I'll have to go with that. And well, look, I'm a big fan of doing this as well, so apologies to everybody, but I love to draw analogies, right? So and I worked for five years in manufacturing. It was a silicon wafer and fab, actually, so I was very familiar with the term, a "bill of materials", right? And what goes into a final built product that's very easy to see in manufacturing, right? Because you know, your suppliers, your raw materials, they come in, they go through a production line process and the other end comes a finished product.
And that's the same for software supply chains, right? You have raw materials that you ingest almost. And those are the open source libraries and projects that you're building upon and you're using as components in your finished product. Unlike the manufacturing industry, the software industry sort of hasn't reached that level of maturity yet of manufacturing. So for manufacturing, for raw materials, and for supplies that you bring in, there are certificates of conformance. Everything goes through a quality analysis. You look at a company like Apple, you know, they audit their suppliers heavily right back to the, you know, the mine that the rare earth metals were mined out of, you know, and they have all of that traceability on the supply chain from raw materials right through to the finished product that you hold in your hand.
And as an industry, the software industry kind of needs to get there. Now a problem with that is that it depends so heavily on open source, everybody uses open source and open source is fantastic. We're a great supporter of open source at Cloudsmith. So we don't have those centralized industry bodies that do quality control and conformance checks like manufacturing dollars, which is exactly why we need what Dan said for the software industry. We need zero trust because we don't have those centralized armies and organizations to prove the provenance of an open source library or to give that traceability all the way back to the source. So that's what a software supply chain is. It's everything that goes into making up your product and, of course, dependency of dependencies. It's just like the raw materials in a physical supply chain. You know where a screw can be traced back all the way through the metal that was used to make it all the way back to the main that the metal came from. And that's like the chain of dependencies and so on.
Adil Leghari: Exactly. And I will save some of the problems is with that for the later parts of this conversation with their natural points, and we can discuss that too. So just to summarize a lot of what you're saying, which was awesome, is the idea that this bill of materials that people keep talking about - software build materials as SBOM stands for software build of materials.
And you're not only trying to understand where code comes from, but it's sort of each step in that process, right? What dependencies did I use? What image was this built on? What factory? If you want to say that. What repository was used? What commit was specifically added this code? And all these sorts of details kind of like. I know Adolfo from Chainguard says that sometimes, right? It's a product packing slip on the side of a UPS package. The details, you know, this was checked by this person at this point went through these waypoints and came from this factory and all that was detailed. So that's kind of where we're getting out of the software supply chain.
Dan McKinney: And Adil as well, even the environments, even the tooling that was used to build an artifact that's part of that as well. You know, it's the complete bill of materials and not just the raw materials themselves, but the environment that they went through. You know, the versions of the tools even down to, you know, the build host that was used for a particular pipeline run. It's all it's all relevant.
Executive Order 14028
Adil Leghari: Absolutely. So, OK, the last date I'm going to highlight real quick is May 12, 2021, and the Office of the President, of course, on the state-issued Executive Order 14028. There won't be a quiz later if you don't even know. But it was on improving the nation's cybersecurity, specifically. And so this really highlighted that this was a real issue that the government was willing to throw its weight behind, right?
And they really talked about removing barriers to the sharing of threat information and, you know, modernizing some of these practices in cybersecurity that had kind of fallen by the wayside, but only about actually establishing government-run cyber safety review boards and details like that. So, I think this was a big point for the industry, for sure, Dan Lorenc and I think you definitely were involved in some of this or your projects were referred to in some of this, right?
The Secure Software Development Framework (SSDF)
Dan Lorenc: Yeah, yeah. No, it was a kind of fast couple of months from the attack to kind of the analysis allegations coming from earlier the year before. I'm out to get this executive order that was kind of just the start of that road to right the executive directive on how to produce software. Right? It was a directive to a bunch of other agencies within the government to start collecting industry feedback and use that industry feedback to shape similar recommendations that get applied in purchasing decisions and that kind of thing.
So we're still seeing kind of that process unfold and actually two or three weeks ago on this, the National Institute of Standards and Technology in the U.S. finally published - the not final version they're going to keep iterating on it - but the, you know, a final version of the SSDF, The Secure Software Development Framework, which is really a great read and if you haven't seen it yet. But this is kind of the culmination of talking to people and industry, people in your government, people in the public sector, and private sector listening, what the best practices were today and then leading all of those into a, you know, checklist that you can run through.
And, you know, just an almost like a self-assessment for your organization. I'd highly recommend taking a walk through it if you haven't yet, it's broken out in a couple of different sessions. We've got something on the Chainguard blog about it too. But this is basically, you know, the industry set of best practices here for both consumer and secure dependencies and then also operating build systems securely in your supply chain. There's some stuff in there that I think will surprise most people because, you know, there's a lot of stuff which is basic, you know, brush your teeth style hygiene that everybody knows they need to do. And then there's a couple of things in there that are kind of pushing the envelope a bit. And I think you're going to catch people by surprise later this year if you don't get a handle on it now.
Adil Leghari: So what do you want to mention any of those?
Dan Lorenc: Oh, man. Yeah, well, I mean, there's one funny one that I missed, but there's one funny one that I'm curious to see how it plays out, but it requires, you know, vendors to try to comply with the SSDF. Do you have to also apply that recursively to their vendors, which makes perfect sense? Yeah. But yeah, it's going to be very slow until the first person tries to meet this. And then all of a sudden it's going to cost the industry because we're all in a supply chain together. Most of us are somewhere in the middle, right? So long before it starts, well, creeping up on people.
Adil Leghari: That's the tough part. Right is its providence all the way down. Right? You really need to get folks to buy into it. So I think if once large organizations start hopping on that bandwagon, I think the industry, you know, just by sheer momentum alone is going to be forced to comply to a lot of the stuff. So as organizations, what you're going to see is if you're selling products and software, you're going to start seeing more and more security. You're going to start seeing more and more from your customers and consumers wanting details about where, you know, the entire chain of trust all the way down to your tier dependency, even if they are open source, right? So they're not necessarily having to reveal the entire sausage and how it's made, but at least all of the different components that went into it.
Open Source Security
Dan Lorenc: One of my other favorite parts, though not, you know, easy or hard or just I love seeing it called out. I mean, just talking about open source, the guidance kind of doesn't require, but it recommends reusing well-tested well-developed components, including open source. We're talking about open source security. There are a lot of misconceptions that open source is worse than closed source or anything like that. And you know, it's not open source software, it's just software the same as closed source software. And I think by in large, most people agree that you know, in general, open source software is or can be more secure than proprietary software.
It doesn't mean every single library on GitHub is, but I think on average the stuff you're using, you're in a much better spot if you're using, well tested, well-reviewed, well-maintained, open source libraries. And so the net framework actually calls that out and recommends it as a best practice rather than everybody trying to rip off the open source out of their supply chains. Right? That would be a mess. Yeah, set things back even further.
Adil Leghari: So you bring up a really great point then, and this is one of my little bugbears, I keep hearing this in the industry. A lot of folks and I've heard organizations, even on sales calls or other places, talk about, you know, you can be as secure as you want as an organization. But if you're consuming open source packages, you know you could be vulnerable. And I just feel like a little bit fearmongering right at the end of the day, it's really that the open-source code is in every supply chain. Let's be real. There are pieces of it everywhere. And that's the tried and tested one, you know? So why aren't we backing that as an industry and supporting those efforts? More so because that's where we all collectively, as humans benefit a little bit more from the improvements. Right, Dan?
Dan McKinney: Absolutely, absolutely. I couldn't agree more, really. In fact, I do think that frequently, you know, open source software effectively has more eyes on, you know, in the process of its development. You know, by its very nature, it's probably wanted to sort of audit and check so. And yeah, no, I totally agree with that. And I know even, well, Paddy probably knows better than me. But you know, within Cloudsmith, we, of course, use open-source software to build Cloudsmith, and we open source as much of our own stuff as we can and we contribute where we can. So, you know, I think that's something that we would definitely agree with.
Adil Leghari: Paddy did you want to say anything about it?
Paddy Carey: Yeah, Cloudsmith, like a lot of other companies, wouldn't exist were it not for open-source, you know, being able to trust that software, to be able to trust that the things you were building on top of is crucial. And it's also crucial for our customers. Our customers need to be able to trust that our supply chain is secure in the same way, we need to be able to trust the supply chains of our providers, by its cloud providers and even their hardware suppliers. It has to be secure the whole way down.
And I guess we'll get there eventually, right? Yes, it's got a long, hard road to get there, so I was moving in the right direction.
Dan McKinney: That's exactly right. I mean, this is a journey right there and then we're all sort of on together and, you know, reunified by our use of open source and that there is no quick fix. You know, one size fits all solution here, but it's about figuring out what the bits of that solution look like. You know, no, nobody can do it on their own.
Adil Leghari: And I think that's where Dan Lorenc, you also mentioned, you know, this is security by default, where there are no quick wins. But we need to make it easier and simpler for folks to be able to consume security as a part of the supply chain, right? We don't want it to be like this thing where it's like an afterthought.
Why is this a hard problem to solve?
Adil Leghari: So for those speaking of journeys, for those of us following along with the actual agenda, we are now into the why is this a hard problem to solve? Right? So I think a lot of this is it comes from a challenge, like you mentioned before, about the software supply chains and pipelines, right? We have complex pipelines now. So the more that we bring automation into which we want, we are also, you know, introducing probably, possibly a little less large on some of our code with the automation and pipelines that kick-off.
Complex dependency challenges
And also some complex dependencies. We depend on software, which depends on software. It depends outside of the radar. So that's where I think this can be a tricky, tricky problem, right?
Dan Mckinney: Yeah, I absolutely. And, you know, we've all seen this. You can pull in one dependency that maybe has 100 dependencies of its own, maybe more. Sometimes it gets pretty ridiculous at stages. And that's what that's only really one part of it, really. You know, it's a hard problem on many fronts. It really is.
Adil Leghari: Yeah. And I think on top of that, I mean, of course, every attack that we hear about right, SolarWinds , Kaseya, Log4Shell, that are out there...was it the DevOps security report that mentioned, I think attacks have been up this past year by 600 percent or so? So there's definitely a lot of exploitation going out there in the software supply chain.
And, you know, historically, Dan Lorenc, a lot of the stuff that's been put out there. I mean, I know I've worked on code before where I tried to sign things and different packages or even different pieces of code in PowerShell. But it's historically very difficult to sign, get a signing key or be able to sign anything you know, when you either have to pay a lot of money or try and try to jump through a lot of hoops with the servers and, you know, like signing mechanisms. So I think it's safe to say that these mechanisms used to be a little bit clunky, right?
Dan Lorenc: Yeah, absolutely. I think, you know, not for any real hard reason or anything if there just wasn't demand for it, really, I think. If you sign a package in the force and nobody verified it, did you really sign it at all before announcing? So it's complicated to get started. Like if nobody's asking for signatures or anything like that, then you're not going to do them. And if nobody is used to asking, then nobody's going to provide them and you kind of get stuck in the cycle. So you have to kind of break out of that somehow.
And I think the attacks we've seen have kind of proven the need for it. And so we're seeing some pretty rapid adoption and Sigstore and a lot of these other projects just because of that. Now, everybody, it's where people are aware of it. Software producers are trying to do the right thing. And software consumers are trying to do the right thing now, too. So it's kind of a great time to be in the field trying to build tools to make that easier.
Adil Leghari: Yeah, no, I agree. It's funny because I liken it to and this is probably a poor analogy, but it was sort of like a Rube Goldberg game sort of contraption to try and find things. You have to make sure a server was available. And historically, with cosigning, I know back when I was at Chocolatey, you have to pay for some of these signing certificates, right?
What is being done?
Open Source Projects
Adil Leghari: So, yeah, I think this is good because, like this, the conversation's just naturally flowing this way. But like here, let's talk about what's being done. Let's talk about some of these big open source projects. So I think the natural starting point to this is Sigstore, right? Do you want to reference that?
Dan Lorenc: Yeah, sure. I can tell a quick story there. We're studying cosigning and why it wasn't being adopted in different ecosystems for a while within the OSSF and Google. And we saw kind of a similar thing play out across all the ecosystems where, you know, the tooling wasn't great. People proposed adding signatures to Python or npm or Ruby or something like that.
And it's just really complex and you start thinking about key distribution and who's going to verify what and what is the signature even mean on these conversations just kind of stalled out because there wasn't much of, you know, motivation. But then on the flip side, there was all these kind of walled gardens and other commercial ecosystems where signing was happening all the time, right? If you publish an Android app or, you know, Windows drivers or something like that, you need to sign them and so in those worlds 100 percent of everything you signed just because you had to.
And the way those toolchains worked and everything was with kind of traditional certificate authority hierarchies the same way TLS works in your browser or open source alternatives were most commonly using something like PGP and Web of Trust, which was kind of finicky and hard to get right. And so when we started looking into the commercial code signing stuff it, you know, just kind of gave us all these flashbacks to what the internet used to be like, you know, five, six years ago.
If you remember trying to set up like an Apache server or something and get the, you know, TLS, you'd have the green shield next to your browser rather than the Red X, right, you know you'd have to find them, you'd have to pay them, you know, a couple of hundred dollars, they'd do some verification. You'd have to send them something on official letterhead for your company, right?
Adil Leghari: Do you remember the little logo that used to be in a corner of web pages - "Geo Trust" or "Web of Trust"?
Dan Lorenc: Yeah, yeah, encrypted. And you just put that there. That's not how that works.
But yeah. And so, yeah, you pay in that email, you a certificate and you copy it into the right directory and rename it and run these arcane commands. And if you got it all right, then you get the green shield and then you probably forgot all about it. A year later, the certificate expired and you'd have to remember how to do the whole thing again.
Adil Leghari: Because everybody was complaining, right?
Dan Lorenc: Yeah. And then Let's Encrypt came along and kind of changed it completely, right? They set up a free certificate authority as a public betterment group and got accepted by the browsers and all of us, by automating everything. Instead of certificates being valid for a year with these kinds of intrusive checks on your identity. They made a whole new standard called Acme, where you could prove that you had control of a domain. And you know, the certificates are only good for a couple of weeks or if something bad happened, it would automatically expire. They do these continuous checks. They got this going. And then within a couple of years, it went from like 50 percent adoption to like 99 percent of the web. And it shows, yep, people do really want to do the right thing. If you put it in their tooling and you don't have to think about it anymore.
Sigstore
I mean, with Kubernetes clusters you can just put like, you know, an entry somewhere, then you don't have to worry about anything else anymore. So kind of on one line change versus what people had to do before. So we kind of tried to copy that model with Sigstore. So instead of having people buy things, manage keys or worry about losing them, we set up a free certificate authority. And this is all backed by, you know, transparency logs and some new technology we also copied from the web ecosystem.
And now you don't have to worry about keys, and you don't have to worry about, you know, validating each other's stuff in person, and you can just kind of get them and sign things on the fly and do it automatically inside of GitHub. And so we're seeing a huge amount of adoption there, which is great because it actually gives us a way to kind of trace packages back to where they came from with cryptographic proofs and let people sign things with email addresses instead of having to keep things on the smart cards and everything like this.
Adil Leghari: Yeah, for sure. And I think Sigstore was a big turning point, right? And that's sort of one of the big projects that are coming out of this. And do you want to highlight co-sign at all? Because I think that's the big one for folks with Docker images.
Cosign
Dan Lorenc: Sure. Yeah. We, you know, Sigstore supports all sorts of artifact types. You can sign firmware, you can sign, you know, executables, jars, all the stuff. But I'm one of the first ones we started with was containers in the overall OCI ecosystem because, yeah, there's always language package managers. And there's also been this movement to start using OCI registries to start storing all these other things with efforts like, you know, the ArtifactsProject and stuff.
So we said, hey if we started hearing more people start moving things into these registries, it's kind of a double win. Plus, the containers, you know, kind of critical, they're critical to mass production and they're kind of the last mile anyway. So the Cosign tool there is focused on container signing and all of these other little ecosystems that have developed around storing other things in container registries.
So for Cosign, it's https://github.com/sigstore/cosign, you can download it, it's in most package managers and gets started signing right away, whether it's on your laptop or in a GitHub action pipeline or and whatever CICD system you happen to be using.
Dan McKinney: I'm sure in the next bit about what we are individually doing to help this. I'm sure probably has something to say about cosign because it's on our agenda. But I love the analogy there between Sigstore and Let's Encrypt because I'm old enough as well to remember the silly dance that you had to do to get a certificate and set it up. And I couldn't believe the first time I use Let's Encrypt how simple it was. I mean, it was awesome. The DNS verification and the next thing, everything was working. And I remember having thought at that point that, yeah, that this is the way it should be. It should be this straightforward and. That's what we need to get to with artifacts and packages. I mean, it absolutely is.
And look, you know, we already support, you know, on Cloudsmith for artifacts that are stored on the Cloudsmith repository. We already support signing them with the GPG key and things like that. But as Dan said again, that's all well and good, all the artifacts are signed, but it's when people come to consume them, you know, are they bothering to verify? Is anybody even looking at it? You know, that's the other side of the two. Also, is it easy to do that?
It's all down to ease of use, and it almost you make it so easy that it's automatic that the people why wouldn't you do it then is the question. You know, because it just it almost you make it difficult to not do right rather than difficult to do. And then you people have to go out of their way to not do so by them. And we'll talk more about that next.
Adil Leghari: I mean, there's no official agenda, by the way. So for the record, we can flow back and forth. So I very much think we're now in what are we doing specifically as organizations to help? And you know what every organization should do to support these efforts? So Paddy?
Paddy Carey: I just want to add to the previous example that the thing that always sticks out in my head of how awkward things used to be as a teenager, getting into Linux, figuring out how to build Debian packages, figured out how to sign them with my own GBG key. But nobody trusted them. And I remember I was 17, I had to travel 90 miles to a key signing party, show my government I.D. to everyone else. And then they had to verify that I looked like my I.D. and then maybe they would trust me. Maybe enough people would trust me that suddenly my signatures would be signed or verified, and it was just this horrible, long-winded process. I remember once someone I knew you personally for three or four years at the time refused to sign my GBG key because I didn't look like my I.D.
You know, some of these open source communities were really strict how this stuff works. Yeah, it was just a difficult process. And by the time it was all done, I had no appetite to continue doing it anymore because it actually.
Dan McKinney: Yeah, exactly, that's a fantastic story, by the way, and you know, I've worked Paddy for a couple of years. I've never heard that story. That's awesome. That's a really good story.
How do we know this is Paddy now? Yeah, I know you got your ID. Can we see your I.D., please? No, that's it. I mean, that's a great example. It's an awesome example.
What are we doing?
Adil Leghari: And I think this is also part of this whole thing about how not only us specifically at Cloudsmith and Chainguard but how all organizations can help these efforts, right? And I think I think a big part of this is is, you know, now with Sigstore and Cosign gaining popularity and a lot of these languages now at package registries coming on board as well, right?
We talk about, you know, Shopify is great open source efforts with Ruby Gems. The work they're doing on there, I'd really want to give kudos to those folks because they're leading the charge. And, you know, Python folks and then Maven central over there. We have that like sort of I keep referring to it as a tipping point, but it's sort of like momentum, right from pushing forward.
Dan Lorenc: Feels like a ball rolling down a hill, and it's just going too fast to stop now.
Adil Leghari: Yeah, yeah. Yeah, now we've got the momentum,
Dan McKinney: And the thing is that that is what we need as well. We need to have that momentum because of course, the last thing that anybody wants here is for these, and Don mentioned it earlier with walled gardens and things, the last thing we want is for more proprietary stuff to come along. It will never get by in an adoption. We need the projects to be opened, the tooling open. And of course, as vendors, I mean, Cloudsmith is a vendor, a software vendor, we provide a service, but what we need to do is support those projects, integrate them with our offering.
It's fine to be a vendor that builds value on top of those things in terms of analytics and reporting and all, that's superb! But the core, you know, the core of Sigstore and Cosign that needs to be widely adopted and to be widely adopted, it needs to be open and everybody needs to have to buy in to it. We are very anti-vendor lock-in at Cloudsmith anyway and for something as important as this, it needs to be open so that all the vendors can jump on board with it.
And that's what's going to give it the adoption, right? It's going to make it that secure by default, really, you know?
Adil Leghari: This is very much my soapbox here at Cloudsmith, for those who know me, I think a big part of what I see is this sort of shared ecosystem of providence, right? And so folks like, you know, our colleagues over at Sonotype with Maven Central or, you know, the folks at Shopify with the Ruby Gems stuff. This is the stuff I love to highlight because it's vendors, you know, that are proprietary and they're doing their own stuff, but they're working in the open source spaces, right?
And they're developing sort of out loud and in the open and being transparent about some of these, you know, different tooling and different organizations, different projects that are coming through open source and they're putting their weight behind that right. And I think this is a challenge, right? I think a lot of folks like Dan, said, you can build value on top of this stuff.
And I understand, you know, as an organization, you're beholden to your stakeholders. You have to make money, absolutely. But I mean, I'll be honest, I'm going to call out our industry a little bit here and say, I feel like, you know, the security by default thing isn't always adopted, right? It's often an additional feature that you have to buy on top.
So I want to say that our commitment here at Cloudsmith and something I'm going to continue to push for internally is that the mechanisms for security exist for the free plans for the open source plans. You know, the communities that we're supporting that comes in by default, you don't have to pay extra for that. You shouldn't have to. And we should support that ecosystem of vendors, right? And we should support those open source projects to continue doing. I'll get off my soapbox now!
Dan McKinney: Well, yeah, that's quite the soapbox, Adil, I like it! But no, I think you're right and I agree that you know, as obviously as a vendor, as a software business, and that is what you should do, should build value on top. And that's fine. And in fact, that's exactly the right way to go about it. But the basics really need to be open source software, available to everyone, auditable by everyone, needs to be out there in the community. It's the only way to get real widespread adoption, you know?
Of course, the attacks that happened shone a light on it, but those alone wouldn't really push that forward, you know, so has to be adopted by the community in general. And I know, look, actually Paddy definitely knows more about this than I do, right? Because I'm not an engineer at Cloudsmith. Paddy is. What do you think is sort of coming down the pipeline at Cloudsmith internally is worth highlighting here?
Paddy Carey: I know our engineering teams are currently working on, I guess, in broad terms to meeting users where they. Secure defaults and everything are great, but we need to make sure that the packaging ecosystems that our users are working and whether that's containers or enterprises using Maven repositories or Python dependencies, whatever they're doing, whatever open source telling they're using in the rest of the ecosystem, we need to make sure that all the signatures they generate, the attestations they generate, we can work with those.
We don't want to create a walled garden. We need to make sure that whatever the ecosystem is doing that we support. So that includes supporting SBOM standards, it's about supporting all of these open standards, making sure that whatever way our customers, our users are working to secure their supply chains, that were there to meet them when we're ready to do that.
Not every user is. And that's why the UX of these tools and the UX of these processes are so important. If we're going to convince these people to make the move and secure their own supply chains, we need to help them to do that. And it's yes, it's just about that working with all those tools. So when Ruby Gems implements Sigstore support, we need to be there to meet them with that.
When PyPI implements their support for force signatures when Conda supports the update framework, we need to be there and sort of integrating those things as they happen.
Dan McKinney: Yes. And I mean, I know that we have, you know, support for Cosign with our OCI registry coming down the line. And I know that kind of stuff. But it's not just that we're going to provide this kind of stuff for users of Cloudsmith, right?
I'm sure, on the platform team, you guys are eyeing this stuff up keenly internally as well for our own build pipelines?
Paddy Carey: Absolutely. It's something that came up earlier. If we want our customers to have secure supply chains, we're part of their supply chain. We need to be secure and our supply chain needs to be secure as it's turtles all the way down.
So we need to be able to trust the software that we're deploying so that when we make attestations about the software that our users are shipping via Cloudsmith, we know that they're secure and we need, we need to trust our cloud supplier, our cloud provider supply chain. They need to be able trust their providers so we can really only do the work for ourselves and push others to do that work. It's really just about utilizing all of those tools internally as well and exposing the parts of those processes that allow the users to verify that we're doing the right thing as well.
It's one thing for us to say that we're secure. We need to be able to prove to the users and the people building on top of our platform that we are actually secure.
Adil Leghari: Yeah, absolutely. And I think that that's a great way to sum it up.
I think I will say, Paddy, I think hopefully we're going to see ourselves on the friends of Sigstore repo soon?
And I don't know if Dan Lorenc, you'd like to say anything about what you are doing in the space, but I know that there's stuff that you were going to announce very soon, right?
Dan Lorenc: Oh, no, I don't remember! And I'll just say I love the way you talked about all the different ecosystems because you were using the word "when". And I think that's awesome! Instead of "if" you're saying "when Sigstore store comes to Ruby" and "when it goes to PyPI" and "when the update framework goes to Conda". I just love that like in the last year, I don't think you could have used that word. I think you would have it would have been like a maybe if future, you know, maybe someday type of thing, but now it really is kind of like inevitable.
Adil Leghari: Yeah, it's funny how that like a year ago, you couldn't even imagine these conversations happening.
Dan McKinney: So I will just add one thing I know we've only got 10 minutes left, and there are a few questions in the Q&A that we're probably going to want to address. Yeah.
I just wanted to throw this in because this is something again, I'm not a guy, and this is only the second time I've made an apology in this discussion, right? So I will make an analogy again with the manufacturing industry because the entire focus of this talk has been on threats to the supply chain, right? When you say that a lot of people think of malicious packages, impersonated packages, you know, a bad guy, right? A bad person, you know, you know, corrupting your process.
But also don't lose sight of the fact that availability is a threat to your supply chain, right? Now, it's not often that a package becomes unavailable, but it has happened, right? And everybody remembers the chaos when left pad was removed from npm, right? You know, this is absolutely not the focus of supply chain security, but it's an aspect I'm that is that in that analogy that I used in manufacturing what the manufacturers do, they build up inventory. They have a warehouse, right?
So that's something that we do think about in Cloudsmith, right? Cloudsmith repositories, you can pull in packages from upstream, you can cache them so that if the upstream source disappears, you have those local cached copies of packages your built pipelines won't feel.
Adil Leghari: Important difference, though, I will mention private is trusted first, not public.
Dan McKinney: Yes, of course. Of course, private is trusted first, and again, this does not absolve you from all those packages needing to be verified and signed over all of that good stuff that we've been spending the time talking about.
But it's very easy to lose sight of the fact that you know your availability and the reliability is just, you know, a part of a secure software supply chain as well, just the way it's the part of a secure physical supply chain. You know, if a supplier disappears and you need to have a plan in place for that, and that's just it's just worth thinking about. But I absolutely agree it's secondary to the threats that are being addressed by Sigstore and Cosign and vendors like also adding support for those totals definitely secondary to those.
But it's not something I hear people talk about much or bring up in these discussions. So I just wanted to put it out there before we lost sight of that.
Paddy Carey: So I think in general terms, though, Dan, it's a good point that there's no silver bullet here. There's no one fix. There are a hundred things here that are a hundred different parts of the supply chain, 100 moving parts that all need care and attention as we work to secure our supply chain.
So signing alone, great, it's not enough. You know, caching dependencies from upstream. great it's not enough. Not all of these things to work together and the entire community to come together and work to address all these problems are knocking them down one by one, and we'll get more secure over time. But it's going to be a journey to get there.
Adil Leghari: Absolutely. And I think that that's part of the challenge, right, is I think people look at this problem and often security folks will come up and list all the different things that you need to have in order in order for something to be trusted, right, as a package or any sort of artifact that you have up in the registry. And I think the challenge there, Paddy specifically is I feel like, yes, absolutely. We have a lot of different things and you need to secure it all the way down. But we have to start somewhere, right? And I think that that's the important thing to not lose sight of.
We have to make these micro-steps along the way in order to get the whole thing secure. And I think you can't let a lot of folks get a little bit overwhelmed with all the different things to do and then just throw up their hands up in the air. And I think that that's, you know, hopefully, projects like Sigstore and Cosign and other open source efforts and getting together as a community will help us to make these things not difficult anymore, you know?
Dan McKinney: Yeah. Well, I mean, the SLSA framework is a great example of that because each level one and the SLSA frameworks is pretty easy to achieve. You know, it's it's pretty simple step to get to that. Obviously getting up to level four. Yeah, it is a lot more you need to do. But it's giving, you know, accessibility. You know that you can get something in place and you can feel good about it and you can feel good that you've taken the first steps on that path, you know, so you don't jump to SLSA level four from nothing. That would be, you know, it would be a really monumental thing o try and do. It would put you off if you tried to do that, so you start off, as Paddy says, little steps along this journey, but we need to do it together. Nobody can run off on their own on this. We need to do it together as an industry, you know?
Adil Leghari: Yeah. And so I guess this is a good transition naturally into one of the questions that was up, which I think they were starting to answer, which is, you know, which are two terms on here Cosign and SLSA and he wanted a little bit, maybe just a little bit more definition and discussion around it. We can definitely provide links in the chat if that's helpful. But if you want to spend a little bit of time touching on Cosign and SLSA specifically.
Dan Lorenc: I'll just stress that SLSA is a security level, I cannot remember what a stands for homeland security levels for software artifacts or something like that. And it is, you know, framework one levels one through four today and still kind of under, you know, heavy development as we iterate and people actually try to work up their way through that ladder.
Like you said, level four, it's pretty hard for I'm not sure if anybody's really done it yet, so I'm not sure your feedback as we get to those higher levels. One was designed purposely to be easy so people can kind of get on the train and get a badge setup in their repo and show what they're doing.
The way I kind of think about it is a level one is like, don't do the build on a machine under your desk or your laptop or something like that. That's really all you have to do is use a build system.
Level two is, you know like proves that somehow some publishing like providence or something like that, some people can verify that you're using a build system.
Level three is, you know, like use a secure build system, maybe not just everything qualifies today. It'd be great if you know all these built system operators do increase that security.
Then level four is, you know, change your whole build process to make it hermetic and not full dependencies from the internet and, you know, perfect to keep themselves off. So, you know, kind of far out there, the more secure you can possibly be. So I think there's a pretty big jump from three to four. So that's kind of my take looking at it. So we might end up, you know, adding an intermediate one or something and there shifting some things around.
Yeah, it's just really pretty easy to understand and you get value out of each level as you go.
Adil Leghari: Awesome. Yeah, no, that's a great, great summary of that, I think. And so along those lines, similarly, when we talk about Sigstore and Cosign, Cosign specifically. I think Sigstore, you can think of it kind of like your umbrella, right? Just being able to sign software in different ways and Cosign is specific for sort of OCI container images. So that's a good way to understand it.
Yeah, and we can drop the links in the chat as well.
So there's one last thing I'll do is there's a meaty question here which we kind of need to get to. And I think it's kind of one of those things which is open-ended a little bit because I think I'd like feedback from all of you on it. And I think it definitely this is part of the stuff that we need to work together as a, as an organization or as a community to do all of these organizations.
So many organizations nowadays rely on distributing their software from sandbox containers or images. What could be the best practice as a customer who is consuming these kinds of distributions from the vendor because these containers and images or sandboxes are an entry point for supply chain compromise, Any good practices for the customers to conduct tests on these binaries?
Adil Leghari: He is talking along the lines of hey, as consumers of these products, these container images, what can we be doing as the best practices when we're on the consumption step of this to make sure that we're checking the boxes? So it's more a best practice sort of conversation but go ahead, Dan.
Dan Lorenc: Yeah, I can start. I mean, there are probably 37 different threat models of wanting to worry about here, but I break them down into two categories first. You know, two high-level ones. The first one is, you know, you're consuming this kind of black box thing from a vendor. You know, it might have some vulnerabilities inside of it. And if you run it, then you become vulnerable. So it's kind of like, you know, if you're consuming about software, it doesn't matter if it's signed, you know, exactly where it came from and you have all the metadata, right? It's still bad software. Not bad, bad. Like, you know, it might be vulnerable, might be old, that kind of thing. And so you want to do your scans on it. You want to make sure the code is running in an environment of least privilege, all that stuff. So if it is compromised, right, it's not running his route with, you know, network access and access to your database and all that stuff.
The second one is, yeah, did this actually get to me from the distributor correctly without being tampered with? Those kinds of things. And so that's checking the signatures just to provide them checking the hashes as they provide asking them to if they don't, depending on your relationship with the vendor. and making sure that you know an attacker in the middle can't change it if you're driving from the correct place and that hasn't been tampered with along the way where somebody could insert some kind of targeted malware. So at a high level, those are kind of the two big categories I look at and kind of need to treat them a little bit differently.
Adil Leghari: Paddy do want to add to that at all?
Paddy Carey: Really what he said, I think that covers it, really. It's about it's about providence. On the other side is just good old-fashioned traditional ops stuff. It's like, you know?
Adil Leghari: Well, exactly, yes. If you're using this sort of tooling, right? I think as a consumer, even, you know, build the expectation with your vendors, ask them to be able to support that. Like if they don't currently have a mechanism for having an SBOM alongside an image right, then ask them for it, put in the feature requests.
This the sort of, you know, desire, demand from the industry is going to what's going to push a lot of vendors that way. We've heard it a lot from this side and believe me, we hear you and we're working on that and we want to continue to support those efforts. So yeah, that's what you can do to be a part of that. But to have that build that expectation, especially with your cloud vendors, that it should be by default, you should make the stuff available to us. And it doesn't matter how much I'm paying you for it. Just like I'm a consumer of yours, you should be giving this stuff by default and available to democratize it. Sort of secure it.
Paddy Carey: It's a rising tide, lifts all lifts, all boats. Things right? That's better for open source. It gets better for everyone.
Adil Leghari: Absolutely. So I think we are running up against time a little bit, but I want to thank my colleagues here, Paddy, Dan and Dan for all of your wonderful insights.
I look forward to some of the stuff we're doing over here at Cloudsmith and also eagerly watching what's going on in Chainguard to see what's happening there in this space. So folks, again, thank you so much for all your time and for joining us on this talk here today. And this doesn't happen without all of you and all of our efforts together. So remember that and thanks for sharing. And hopefully, you left with a little bit more information around making this big issue, a sort of tough issue a little bit more accessible for everyone.
