Future is Cloud-native & Your Organization Should Be Too | Ciara Carey | Conf42 Cloud Native 2022

In 2022, your entire tech stack is likely in the cloud - so why aren’t your software packages? Whether you’re currently on-premise, have your own in-house solution, or have a bit of a hybrid setup, watch Ciara Carey's Conf42 Cloud-Native Keynote on-demand to explore:

What does it mean to be truly cloud-native?

How being cloud-native helps Cloudsmith succeed and can help you succeed too!

Getting started: A look at the cloud-native ecosystem and what to consider when setting up your pipeline

The Future is Cloud-native & Your Organization Should Be Too | Conf42 Cloud-Native Keynote [On-demand Session]

With the increased focus on software supply chain security, the question arises- what implications does that have for CI/CD processes and DevOps pipelines?

The Future is Continuous: Integration, Packaging and Delivery - DevOps Institute SKILup Day CI/CD

Join Dan McKinney, Developer Relations at Cloudsmith for an introduction to Continuous Packaging, and to discuss what it takes to secure build and deployment pipelines at The 2022 DevOps Institute SKILup Day CI/CD.

Check out the amazing graphic which summarizes Dan's talk!

Webinar: The Future is Continuous: Integration, Packaging and Delivery

Webinar: To NuGet and Beyond: NuGet Ecosystem & Upstream Support at Cloudsmith

Watch our To NuGet & Beyond webinar with Adil Leghari, Dan McKinney, Paddy Carey and Mikey Lombardi.

We break down the NuGet ecosystem in Cloudsmith's first ever hosted webinar! Read the video transcript below, and make sure to read our blog article: 

Good afternoon, good evening, depending on where you are in the world. We appreciate you taking the time out of your day to join us here and welcome to our webinar entitled To NuGet and Beyond. Now we here Cloudsmith know that as a package format, NuGet is a core technology for the Microsoft developer ecosystem, and we want to ensure that we're fully meeting the needs of our customers when it comes to NuGet packaging.

And we also want to make sure that we're treating our Windows users like First-Class citizens on our platform. So to that end, today we'll be talking about some exciting new improvements we've made to our NuGet repository platform, here at Cloudsmith, but also our new support for NuGet Upstreams. Sound good?

OK, so before we get ahead of ourselves, some introductions are in order. Firstly, who the heck are we? So, Dan, I'ma let you go first.

 Yeah, thanks very much, Adil. So yes, hi everybody. I'm Dan McKinney, I work in developer relations here at Cloudsmith. So it's my role to sort of educate and help and assist in any way that I can. So hoping to get some good questions today and to show you some of the sort of unique features of Cloudsmith and how we've brought those features to the NuGet community and the NuGet ecosystem in general.

Awesome, awesome. So I'm Adil Leghari, I'm a solutions architect, manager here at Cloudsmith. I'm assistant turned solution architect and turned manager, and I've spent the better part of two decades here wrangling windows boxes.

So over the past five or so years, I've gotten more active in the PowerShell & DevOps community. I'm a speaker and author and a blah blah blah blah blah, you know that sort of thing. But I think more important for today, I've also spent the past three years in the packaging management space, here Cloudsmith and also previously a Chocolaty; hello ChocoFam. So I guess you could say I know a thing or two about new packaging. Both of us are very pleased to have joining us today as our surprise guest speaker, my dear friend and colleague in the PowerShell Dev Community, Michael Lombardi, Mikey, over to you.

Hey, how's it going? So I have been in the PowerShell Windows Dev Ops space for around roughly a decade and spent a lot of time doing talks and trying to get people to write docs and script code a little bit more cleanly, elegantly.

And then in the last couple of years of my entire professional life, doing nothing but open source development and documentation, writing previously at Puppet, currently at Microsoft, a huge amount of fun to maintain Chocolaty packages and PowerShell modules. So packaging has stayed near and dear to me because it's the best way to get what I wrote or what other people wrote to any system. Everything else is just so much pain.

**Adil Leghari:**Yeah, no, for sure. By now, I think also you're doing some game dev stuff, right? Which you'll be looking at in the future for packaging?

Yeah, doing some interesting work in Golang to build some CLI apps and frameworks and tools. And I'm very excited actually to use Cloudsmith to deliver those things in a much less frustratingly, complex way. I don't want to figure out packaging from the ground up if I can help it. And the raw feeds seem like they're going to solve exactly my problem for me. So I'm very excited about that.

Awesome. We're excited to have you, Mikey. I just want to put a caveat upfront that your views and opinions do not reflect those of your employer.

But whereas Dan and I we are, we are speaking for our employers.

Now, for those of you who may not be familiar with Cloudsmith and what we do here, Dan, why don't you walk us through a quick overview?

Yes, absolutely. Well, I would hope that we do have some Cloudsmith users, but for those that aren't yet Cloudsmith users, Cloudsmith is a global cloud native and universal package management as a service package management solution. And so what we really provide is the ability to sort of build a single source of truth for all your software, artifacts, images, packages.

And although this is a very NuGet-focused webinar, you know, we are fully universal. We support over 28 package formats in multi-format repositories as well, so we are a completely global distributed, web application. There's nothing to install. You don't have to patch anything or update anything. It's a managed platform. We have over 225 points of presence on our package delivery network, as we call it, so great for distributed teams or software distribution even.

An so just to clarify, if I were just a random user, a free user even, if I upload any package up into Cloudsmith, it is available in 225 different points of presence.

Yeah, exactly. You know, it's cached at the edge of the package delivery network. So, you know, performant global access. You don't have to worry about replication or clustering or high availability or anything like that. That's the whole managed platform for you.

Well, of course, we are heavy on security, lots of really, you know, granular access controls and permissions and the ability to restrict and allow access and a very fine-grained manner. And for private repositories, of course, you can have public and open source repositories as well. And yes, as I said, universal. So, you know, we love all package formats equally, but with a special focus on NuGet today.

So actually, Adil, I think this is a very good point now that I've just explained, yes?

For our first poll, so really, we'd like to know more about our users or people who are potentially considering using Cloudsmith. What is your need? What problem are you trying to solve in package management? What is your use case? So we have some options in our poll. You know, A to distribute software externally or B to manage growing complexity within an organization which is very popular now.

Actually, we see a lot of this or a very hot topic at the moment, software supply chain security to supply security or software supply chain. Or maybe you just want to centralize package management on one sort of platform. And for those of us, for those of you who have just joined the webinar and missed the previous slide, what is Cloudsmith? That's a perfectly legitimate answer in this case. So yes, and we'll leave that up just for a second and people can answer and we'll look at them at the end.

SSounds good, Dan, and I do want a plug because we have this here, and I think Alan, our CEO, I'm sure is watching, did remind us yesterday that we have all these great resources and we sometimes don't even plug them enough internally. So let's do that now. So I will mention that securing the software supply chain is one of those big SBOM buzzwords that are out there now. We did try and demystify this a little bit. We had a webinar last week with the Linux Foundation and Dan Lorenc over at Chainguard, who's of Sigstore and Cosign fame.

So feel free to go to our website and check that out as well. It's available under resources.

But yeah, generally, we talk a lot about software supply chain security, which is a very hot topic of conversation, right, Mikey?

Yeah, just a little bit of a nightmare to catch up to for everybody who pretended that it didn't matter forever.

So that's kind of like we talked to not talk specifically about everything you wanted to know about the supply chain, but we're afraid that so I'm plugging another webinar during this webinar. But hey, it's all good content, right? So just frames of reference for you for later.

OK, so let's talk you through a little bit of the agenda now that you know who we are and what Cloudsmith is just you know what you're getting yourself into today. So specifically, what I'll talk about, first of all, is what is NuGet, as a packaging format, as a technology, and how it enables folks on the .Net dev ecosystem.

And then we'll talk about specific NuGet flavors, so the communities and tooling around different use cases for the technology. So in NuGet, that's mainly three groups the C#/.NET folks using Visual Studio, the PowerShell dev community and of course, the Chocolaty dev community as well. So we'll highlight use cases, and I'll also focus a little bit on the differences because the packages are very discrete and different.

There's some complexity in there that we want to help and solve. So that's the next point. We will talk about how Cloudsmith helps solve and make simple some of these complexities for you in the NuGet ecosystem and some of the solutions we have around that. Then I will demo and show off a little bit of this actually live for you so you can understand some of the stuff that we've been working on here, especially to cater to .NET dev specifically. So I'll show off some of the easy stuff you can do with the package installs and stuff like that.

But I'll also touch on, you know, some of the work that we're doing with PowerShell modules, et cetera, making the Cloudsmit CLI available and in places like the Chocolaty Community Repository. So then after that, Dan is going to really dive into, but wait, there's more! All the additional platform features that you get, are just parts of being a Cloudsmith customer. So there's a lot more there that we can touch on, but we'll try and limit it to the time that we have here today, and then we'll leave it open for a Q&A at the end as well because I'm sure there'll be some questions. And will have Mikey chiming in with anything coming from a Cloudsmith customer or user perspective as well.

 Any time he sees fit as well, so sound good? Let's get into the meat of the talk show. So first of all, what is NuGet, right? NuGet is the official package management system the .NET development community, right? Those are NuGet packages. And so that includes a platform and tooling to help .NET developers create, publish and consume and share this reusable code. Why reinvent the wheel if you don't have to? A lot of NuGet devs will build,

let's say you write some code to essentially solve a problem, and then now you can have that library packaged as a NuGet package and other developers now, when they need to solve that same problem, they can just pull down your library, install that NuGet package and then they're ready to go with the tools available in that package to make their development and workflow a lot more efficient. So it encourages the reusability of code and the functionalization of those things. And so the smallest unit of that is a NuGet package.

So a little background for you. NuGet was created in 2010 under the name “NuPack”, I believe, according to Rob from Chocolatey, I believe it was “Nu” before that? He mentioned it once. So I think it was “Nu”, firstly, that it was talked about but referred to as “NuGet” or “NuPack” earliest and then also then moving to NuGet as a format by the Outercurve Foundation. The Outercurve Foundation is now very much a precursor to the .NET Foundation set up by Microsoft.

Some people, you know a thing or two about, Mikey. So of course, NuGet packages themselves, what do they contain? What is the NuGet package? So it's actually kind of very universal and multi-use format. So it can be very different depending on the type of package you have, and we'll highlight some of that here. But the main components sort of to boil it down, are a library sort of like a compiled piece of code, or a binary, that actually stores some of the functions and other features of the other that you're trying to use.

So in the case of .NET/C#, it can be a DLL in the case of a Chocolatey package that can be a binary itself, EXEs or MSIs. And along with that is related files so often describing how to use or install the package itself in the case of Chocolate Factory, this is actually a PowerShell script that does the actual installation upgrade or uninstall of the package. So that's part of related files there.

And then we have a manifest file, the Nuspec, if you will, right?

This is the metadata of about the package version title author information like good details to have there, including dependency format as well if you have any dependencies for that package are defined here in the nuspec. OK, so let's talk about NuGet flavors and communities. And again, I'm going to get into some of this stuff, and I just want to do a quick call out here that there's no quiz later, so you don't have to remember all of this. But a big part of what is NuGert, NuGet flavors and communities, tooling, we tackle a lot of this in a blog that I published last week.

So that blog will go and cover this in detail, too as well. If you don't want to remember all of it now, I'll do a quick summary for you, for the different communities,

 So let's summarize some of these use cases, right? So there are three big groups here, firstly, let's talk about the .NET developer community, now this is largely C# folks, but also F# to a small extent. The VB, you know, the hangers-on that are out there still running, VB. So the organization behind this is very much the .NET Foundation now as in 

, of course, and Microsoft that sponsor the .NET Foundation.

And so in addition to that, of course, the language of choice for this is largely C#, so C# devs that are working, you know, in Visual Studio, et cetera, but of course, some F# and VB too. The tool of choice is ubiquitous here, so Visual Studio is the tool that is used most popularly as an IDE an “integrated development environment”, so Visual Studio is what he was to write that code and has some integration built right in for NuGet packaging.

And of course, NuGet CLI as well. So NuGet.exe can be used by devs when they're looking to do something or preferring to do it from the command line or also when they need it in a pipeline. So it's not interactive.

The public repository of choice, of course, is the NuGet Gallery. The NuGet Gallery is probably the largest public repository of NuGet packages. It contains almost 300,000 NuGet packages. So for the sake of our discussion today, it is the largest. So that's a lot of packages that are up there.

And so these packages now have moved to NuGet format V3, just a heads up. So NuGet v3 is the newer format. 

 moved from V2 to V3, largely citing concerns around scaling and the availability of 

. So they had to solve some problems there, and so they upgraded to NuGet V3. And so that's really a re-artifacting of the whole format. So, very much not compatible with the other ones in that sense.

So in addition to that, largely the composition of these are our libraries, so compiled code that other developers can use so they can quickly import that individual studio and then it's ready to go for them. They can just start using that from the repository and start using the IntelliSense and the functions right from that code built into Visual Studio.

The second group is a group we know very well, right, Mikey? So the PowerShell developer community, right?

So this is not a “formal group”, per se. I mean, of course, PowerShell team shout out to Mike Greene and Steve Lee over at Microsoft do work over there and do push the technology. But largely it's the community of folks here in the PowerShell community, and we show up a lot on Slack and Discord and places like that. And of course, Microsoft is behind this technology. The “Shell Father” himself, Jeffrey Snover, who wrote it back in the day or who invented the format.

It's worth calling out that the language is open source and has been for half a decade now. It's Microsoft, but it is all developed out in the open and has been open source for a while. Which is great because people like me and you get to go and bother them with our stuff all the time.

Yeah, we can happily submit our suggestions and we get a response back, say we await your PR, right?

I will call that out a little bit here in the tools of choice too. Obviously, the language is PowerShell, but but largely here, you know, there's two main ways that folks interact in the PowerShell format. One, is powershell.exe. So the original Windows PowerShell does come bundled in, and for a lot of sysadmins, of course, they know on every Windows OS, right? So it's in the Windows ecosystem, it comes built into the box, so it's easy to use.

But largely Mikey, like you called out PowerShell, the pwsh.exe is now cross-platform and open source. So this has made it almost ubiquitous. Now it's across all technologies and a lot of OSS adoption has increased, once it was open source there.

The open source version of Microsoft PowerShell is the very popular format now and ever growing and where a lot of the code now, the functionality lands, right? According to Microsoft, I believe Windows PowerShell is a feature complete, right? It’s just more bug fixes and security updates. So all this new development is occurring on PowerShell cross-platform.

What are we referring to is as now? Should we call it Microsoft PowerShell officially?

Yeah. Okay, let's let's stick with that. And so, that's obviously increased for PowerShell devs, the availability and the accessibility of the code. Right now, of course, the big, big public repository is the PowerShell Gallery. The PS Gallery hosts a lot of the packages, over 9600 unique packages.

The great thing about it is that it's an easy format to be able to push up there. Once you do have your manifest and other things you find, you can push any PowerShell module up there. Now we know, Mikey, you and I, that sometimes what will have to end up happening, at no fault of the team, is the fact that availability can sometimes be an issue, right? Which is where it's helpful to have your own internal or private repositories, which we can get into.

Yes, totally, and running services is hard, and calling out to public services from every server is maybe not your favorite thing in the World. Certainly, not your security team's favorite thing.

Exactly. And it helps to have that level of vetting and internal verification, which an internal repository affords. So we'll touch on tha int a little bit. Now, just a quick point here, so PowerShell modules themselves, they are just files, scripts and manifests in a folder essentially once they get on to the box in the endpoint.

So NuGet here, is actually used as a delivery mechanism, so it's what's used to package the actual module itself. So when it's uploaded into a NuGet repository and when it's pulled down from a NuGet repository is when nuget.exe is used. So NuGet actually does under the hood pack and unpack that package for you into a PowerShell module. So it's a delivery mechanism, so it doesn't actually live in that format. And importantly, this NuGet V2 not NuGet V3.

It's also great for dependency management. That’s really hard to figure out on your own. And version pinning and all that stuff. So you get that for free just by using the NuGet package format.

Exactly, that version pining and dependency is one of the great features of NuGet for sure.

So great segway, here to the Chocolatey Dev Community.

This is a community, Mikey, also near and dear to our hearts, fellow choco-fam from back in the day. So it's definitely alive and well. And a lot of sysadmins now use Chocolatey as an easy way to be able to install any package under the sun on all your endpoints. I know, Stevie is probably in the chat right now, he is a Senior SA and Manager over at Chocolatey. He's helped me a lot with the PS Cloudsmith module, so shout out to Stevie, I appreciate all you do. For sure he is very active with this and he's a big, big advocate for it.

To NuGet and Beyond Webinar [On-demand Session]

Everything You Wanted to Know About Securing the Software Supply Chain but Were Afraid to Ask

From the history of supply chain security threats to security development and deployment we've covered everything you’ve always wanted to know about the software supply chain but were afraid to ask. Dan Lorenc, Founder/CEO, 

, Paddy Carey, Senior Staff Engineer, Cloudsmith, Adil Leghari, Solutions Architect Manager, Cloudsmith and Dan McKinney, Developer Relations, Cloudsmith, gathered for a fireside chat to cover your most burning questions:

 Types of attacks – availability, dependencies, development tools, and more.

Software supply chain challenges and considerations

A look at OSS projects and initiatives that were born out of the SSC security need.

 What have Cloudsmith & ChainGuard been working on to make these issues easier to tackle, and what you and your organization can do to help.


Read the transcript of the full webinar below:


Hello, everyone. Good morning, afternoon, good evening. Depending on where you are in the world, on behalf of my colleagues here at Cloudsmith and Guard, I want to welcome you to our discussion entitled “Everything You Wanted to Know about Securing the Software Supply Chain, But Were Afraid to Ask”. So this is going to be an open discussion or conversation, so feel free to hop in, you know, with questions if you want to. We do have a Q&A section specifically in Zoom for that, so it helps if you throw your questions in there because it will.

Definitely, everybody will get the benefit of the answer if we answer it on the call. In addition to that, of course, like, you know, we're all here and I want to welcome you. But I think over the last few years, software supply chain security specifically has become top of mind as one of these concerns for almost every organization, right, that's producing and consuming software. So we are the tech community have to try and understand this. But if we're going to wrap our heads around it, we really need to have conversations like these, you know, to improve our collective understanding of where we came from and sort of where we're going. So hopefully, by the end of this talk, we'll leave with not just a bunch of fear, but hopefully a little hope as well about the future of supply chain security and software.

So let's go through the agenda real quick here. I will first off just do a quick little speaker introduction for all of us. Everybody will do their own introduction and then we'll give you a little background on what we do with our organizations, with Cloudsmith and Chainguard as well. And then we'll get into the sort of the meat of the torque where we're talking about, you know, some of the things that we want to bring up today. As and again, this will be open, sort of like a fireside chat. We're just discussing stuff. And so feel free to ask questions as well and help my fellow speakers feel free to hop in at any point.

We'll be touching on points like what got us here, specifically when we talk about previous tipping point events, sort of attacks, and other things that precipitated our big concern about the software supply chain. And then we'll touch on why this is such a hard problem to solve because there are definitely some challenges there. Each attacking each different compromise or issue that comes up really has different origins and different techniques and methods. So it can be quite tricky, sort of a moving target.

And then we'll talk about specifically what's being done where we highlight a few open-source projects in the space, specifically around the efforts that Dan Lorenc has done over at Chainguard and also at his time at Google and with open source. So thanks for being here with us, Dan. And then also we'll specifically talk a little bit about at the end about what are we doing at organizations like Cloudsmith and Chainguard, and also how you as an organization can help and contribute to the same efforts? Sound good?

OK, well, without further ado, let's introduce ourselves a little bit. So I'm Adil Leghari, I'm a solution architect manager here at Cloudsmith. I'm a sys admin turned solution architect turned manager. I spent the better part of two decades here wrangling Windows boxes and Linux boxes. So over the past five years, I've become an active member of the PowerShell and the devops automation community, and I'm a speaker and author and super passionate about PowerShell, dev ops and automation, yada yada yada over to you, Paddy.

 I'm an engineer here at Cloudsmith and I've been an engineer for a long time, starting in the tech and e-commerce and finance, and eventually find myself deep in package manager land, which is how I ended up here at the Cloudsmith. I've worked on at everything here, from core package management functionality, securing supply chains and my own to get our platform team part of the responsibility there is securing our own supply chain.

So hopefully that's a relevant experience for me to talk about today.

 Hi, everyone, yes, I'm Dan McKinney, and I work in developer relations at Cloudsmith and I've had a somewhat tortuous path through the developer relations. So I've worked in systems engineering and manufacturing, actually another type of supply chain as well. And yes, it's really my role to sort of engage with the community and educate the community, which is why this is a perfect forum to do this today.

I'm also an absolutely crazy and retro computing and video game collector, and I work as a DJ at the weekends. But yes, it brings a little bit of color to the proceedings. So nice. Thanks, Dan.

: Yeah, second Dan today. My name is Dan Lorenc. Currently, I'm a founder and CEO at a new startup called Chainguard. But before that, I was at Google for about nine years, where I got to spend the last five or six years working almost completely in Open-Source and started worrying about open-source supply chains. So I've been involved in a bunch of different open source projects there and continue to be in products like sig store and salsa products or SLSA. So we have to talk about some of those later today.

: Absolutely. Thank you and thank you on behalf of all of us. Thank you for your efforts and for keeping things open for us. Awesome. OK, well, let's talk a little bit about each other as organizations, about Cloudsmith. I'm going to let Dan take this one.

Yeah, thanks. So Cloudsmith is a cloud-native package management solution. So given how essential software artifacts and packages, in general, are to our software supply chain, it's very high on our agenda to secure that supply chain. So we have been around a lot longer than a lot of people think.

If they're only hearing of Cloudsmith, we've been around and this will be our sixth year. So it's a very exciting time for us because as you said earlier, it'll, you know, secure software supply chains are really at the forefront of people's minds at the moment. And it's so close to our hearts as well. So it's a really exciting time to be part of that. So that's who we are.

Yeah. So yeah, we've not been around for very long. We've been around for about five months. Chainguard here. We got started in October of 2021, and we were trying to hold a bunch of to invest in open source to help people make secure make their software supply chain secure by default. We've seen time and time again that if you make developers do things differently if you make them change the way they work, security is always an afterthought. It's not what you're thinking about when you're in that development loop.

But we believe that if we do this the right way, and if you take a couple of steps back and think about the problem a little bit, we can actually make secure software development a better and easier way and have it work that way by default. So that's what we're focused on doing. We're still just getting started. So thanks a lot for having us here and giving me some time to talk to everybody.

 Dan and I must say, you're being very humble. You guys are slowly picking up the who's who of Kubernetes and other SLSA supply chain projects. So good for you over there. We eagerly await your announcements.

All right. So let's hop in a little bit. And so now what I'm going to do is I'm going to take the slides off the screen because I think this is a discussion. So let's open it up a little bit and talk about what got us here, right?

The scary elephant in the room is all the attacks and everything that went on right? But I think there are a couple of events that will probably highlight that have helped shape the industry as far as the software supply chain. And I think one of the first ones, of course, is the SolarWinds attack, right? So that was December 2020. SolarWinds, if you don't know, it was a very prominent company, that makes software for network infrastructure management. So in December of 2020, they announced that their Orion software had been compromised as a result of the software supply chain attack.

And so this is funny because this attack very much was managed to burn the word "software supply chain" into our collective consciousness, right? Because everybody now knows that word, that phrase specifically, because its malicious code was inserted into the Orion software early enough and that supply chain, it remained undetected. So what happened was that pushed out with the other updates to the Orion software and slowly got out to consumers. And you know, we're not talking about small consumers here, we're talking about big folks like Microsoft and Intel, etc. So little did they know that they were installing those malicious payloads along with the monthly updates.

That's really serving to highlight part of our problem, that up until this point, developers were largely ignoring these parts of the pipeline. This is sort of a tipping point event for that sort of alertness. Now, Dan Lorenc, I know I've heard you talk about this before, but I think you were still a Google working on some open-source projects at this point, right? And this is sort of one of those moments, those tipping points.

Yes, this is the kind of right after I switched from Google Cloud to Google's kind of core security groups, I finally been successful in convincing some people at Google that this was a big enough risk to take seriously is like an existential issue for the internet and for Google. And it was still kind of slow, though, until, you know, the kind of news broke about this attack and the amount of people affected and everything. Almost overnight, we went from a "why is no one listening to us?" to "why haven't we been doing this for years?".

And was quickly followed up with, the stuff I think you're going to shout about in a minute with, you know, the U.S. government looking around to announce executive orders and regulations and new compliance and everything. So yeah, it has become an increasingly hot topic over the last couple of years, unfortunately, because of attacks and not because we were as an industry ahead of the game. But it tends to work that way, unfortunately.

 Yeah, it's not a problem until it's everybody's problem, right?

So, yeah, that definitely was one of those moments, I think for everyone in the industry. I was working at the time, I believe at Chocolatey software and you know, we kept on getting that question again and again where suddenly folks were like "hey, you know, we need to secure software supply chains?" And I'm like, "OK, that's great". I mean, we have kind of been preaching this stuff for a little bit, but now suddenly we're getting all of this attention, right? So I think I think a lot of folks perked up their ears here and sort of your C-level staff to hire folks who are normally not as concerned suddenly started tuning in and asking these questions.

Yeah, I've seen pictures of board decks now with this on a slide from every CSO out there, so it's something everybody's trying to rapidly come up with an answer to.

 I know over the past year everyone's learned the acronym SBOM, and it's kind of funny because it's like, really, we've been talking about this for a while. So I think, yeah, leading onto that timeline was right.

So in February 2021, I think was one of these announcements, where Alex Birsan, who is a security researcher, showed that he was able to compromise supply chains as well. And he was luckily a white hat hacker. So we didn't have the benefit of stress or the negative effects of that. But of course, it had been copycat attacks since then, right?

I want to highlight this one specifically because I think he did this basically by, you know, sort of subverting the logic in the actual process of some of the normal supply chain, right? So as developers, a lot of times we want to use dependencies, right? Obviously, we don't want to reinvent the wheel every time. And so this is specifically referred to as dependency confusion. And it's similar to sort of typosquatting as another type of attack where, you know, essentially what he did was he introduced public packages into widely used public registries that shared the name of private packages for a lot of big organizations. And we're talking 35 different companies. We're talking names like Microsoft, Apple, PayPal, Tesla, you know, like it wasn't a small attack, but basically, he was able to very simply do that by creating packages on public registries with similar names.

So it highlighted another thing where it's the functionality of the package registries in the package management system by default are to trust the public registries and pull those down. But you know, it's the whole "who watches the watchmen" at that point, we're not sure. I mean, everybody had the overnight to kind of figure out. Wait a minute. We have to actually change the logic of how we do this stuff. Right, and I think Dan McKinney, you were mentioned. Yeah?

 I mean, I think the real problem until there was that, I mean, essentially this was a problem with like configuration, the way people had configured the package manager clients to, you know, be able to hit public sources in preference to their private internal repositories. But there were no red flags like, you know, it went ahead. It wasn't that it was, you know, in contrast to SolarWinds, which was a breach of their build system. This was normally running build systems that just were, you know, configured incorrectly but didn't shout about it when something unexpected happened. So, because pipelines are so automated now, that's what enabled that, you know, the automation kicked in, packages get pulled in, things get built. And there was no point where something shouted to everybody.

And why is why was there no point when that happened? Kind of like Dan said earlier that it's not the default and it's not easy enough to have that default stance of, you know, something unexpected or there's a problem or there's something we suspect may be a problem, and maybe we should have eyes on this. The heavy automation sort of drove that. And look, I'm a huge fan of automation. I mean, obviously, automation is what enables, you know, companies to move with a real, high velocity and deliver software quickly.

And that's fantastic as an industry. But it has to come with checks and balances because, you know, automation run amok well will lead to this kind of problem. That's, I think, a core difference between dependency confusion and the SolarWinds attack. Both were supply chain breaches, right? And both resulted in malicious or potentially malicious software getting out into the hands of users. But there's a subtle difference there. The lack of visibility on that process and the lack of alerting and automatic red flags, is what really caused the problem.

 Mm-Hmm. OK. So this is great because I'm going to pick on you a little bit and say so because I love you. So we have a Q&A question. So while you're at this talking about this specific software? Moussa asks, 

"starting from the basics, can you please define what is meant by software supply chain?" 

So defining our terms, of course, where everybody wants to be on the same team?

 And I'll have a go with that then, yes, I'll have to go with that. And well, look, I'm a big fan of doing this as well, so apologies to everybody, but I love to draw analogies, right? So and I worked for five years in manufacturing. It was a silicon wafer and fab, actually, so I was very familiar with the term, a "bill of materials", right? And what goes into a final built product that's very easy to see in manufacturing, right? Because you know, your suppliers, your raw materials, they come in, they go through a production line process and the other end comes a finished product.

And that's the same for software supply chains, right? You have raw materials that you ingest almost. And those are the open source libraries and projects that you're building upon and you're using as components in your finished product. Unlike the manufacturing industry, the software industry sort of hasn't reached that level of maturity yet of manufacturing. So for manufacturing, for raw materials, and for supplies that you bring in, there are certificates of conformance. Everything goes through a quality analysis. You look at a company like Apple, you know, they audit their suppliers heavily right back to the, you know, the mine that the rare earth metals were mined out of, you know, and they have all of that traceability on the supply chain from raw materials right through to the finished product that you hold in your hand.

And as an industry, the software industry kind of needs to get there. Now a problem with that is that it depends so heavily on open source, everybody uses open source and open source is fantastic. We're a great supporter of open source at Cloudsmith. So we don't have those centralized industry bodies that do quality control and conformance checks like manufacturing dollars, which is exactly why we need what Dan said for the software industry. We need zero trust because we don't have those centralized armies and organizations to prove the provenance of an open source library or to give that traceability all the way back to the source. So that's what a software supply chain is. It's everything that goes into making up your product and, of course, dependency of dependencies. It's just like the raw materials in a physical supply chain. You know where a screw can be traced back all the way through the metal that was used to make it all the way back to the main that the metal came from. And that's like the chain of dependencies and so on.

 Exactly. And I will save some of the problems is with that for the later parts of this conversation with their natural points, and we can discuss that too. So just to summarize a lot of what you're saying, which was awesome, is the idea that this bill of materials that people keep talking about - software build materials as SBOM stands for software build of materials.

And you're not only trying to understand where code comes from, but it's sort of each step in that process, right? What dependencies did I use? What image was this built on? What factory? If you want to say that. What repository was used? What commit was specifically added this code? And all these sorts of details kind of like. I know Adolfo from Chainguard says that sometimes, right? It's a product packing slip on the side of a UPS package. The details, you know, this was checked by this person at this point went through these waypoints and came from this factory and all that was detailed. So that's kind of where we're getting out of the software supply chain.

 And Adil as well, even the environments, even the tooling that was used to build an artifact that's part of that as well. You know, it's the complete bill of materials and not just the raw materials themselves, but the environment that they went through. You know, the versions of the tools even down to, you know, the build host that was used for a particular pipeline run. It's all it's all relevant.

Adil Leghari: Absolutely. So, OK, the last date I'm going to highlight real quick is May 12, 2021, and the Office of the President, of course, on the state-issued Executive Order 14028. There won't be a quiz later if you don't even know. But it was on improving the nation's cybersecurity, specifically. And so this really highlighted that this was a real issue that the government was willing to throw its weight behind, right?

And they really talked about removing barriers to the sharing of threat information and, you know, modernizing some of these practices in cybersecurity that had kind of fallen by the wayside, but only about actually establishing government-run cyber safety review boards and details like that. So, I think this was a big point for the industry, for sure, Dan Lorenc and I think you definitely were involved in some of this or your projects were referred to in some of this, right?

The Secure Software Development Framework (SSDF)

Yeah, yeah. No, it was a kind of fast couple of months from the attack to kind of the analysis allegations coming from earlier the year before. I'm out to get this executive order that was kind of just the start of that road to right the executive directive on how to produce software. Right? It was a directive to a bunch of other agencies within the government to start collecting industry feedback and use that industry feedback to shape similar recommendations that get applied in purchasing decisions and that kind of thing.

So we're still seeing kind of that process unfold and actually two or three weeks ago on this, the National Institute of Standards and Technology in the U.S. finally published - the not final version they're going to keep iterating on it - but the, you know, a final version of the SSDF, The Secure Software Development Framework, which is really a great read and if you haven't seen it yet. But this is kind of the culmination of talking to people and industry, people in your government, people in the public sector, and private sector listening, what the best practices were today and then leading all of those into a, you know, checklist that you can run through.

And, you know, just an almost like a self-assessment for your organization. I'd highly recommend taking a walk through it if you haven't yet, it's broken out in a couple of different sessions. We've got something on the Chainguard blog about it too. But this is basically, you know, the industry set of best practices here for both consumer and secure dependencies and then also operating build systems securely in your supply chain. There's some stuff in there that I think will surprise most people because, you know, there's a lot of stuff which is basic, you know, brush your teeth style hygiene that everybody knows they need to do. And then there's a couple of things in there that are kind of pushing the envelope a bit. And I think you're going to catch people by surprise later this year if you don't get a handle on it now.

 So what do you want to mention any of those?

 Oh, man. Yeah, well, I mean, there's one funny one that I missed, but there's one funny one that I'm curious to see how it plays out, but it requires, you know, vendors to try to comply with the SSDF. Do you have to also apply that recursively to their vendors, which makes perfect sense? Yeah. But yeah, it's going to be very slow until the first person tries to meet this. And then all of a sudden it's going to cost the industry because we're all in a supply chain together. Most of us are somewhere in the middle, right? So long before it starts, well, creeping up on people.

 That's the tough part. Right is its providence all the way down. Right? You really need to get folks to buy into it. So I think if once large organizations start hopping on that bandwagon, I think the industry, you know, just by sheer momentum alone is going to be forced to comply to a lot of the stuff. So as organizations, what you're going to see is if you're selling products and software, you're going to start seeing more and more security. You're going to start seeing more and more from your customers and consumers wanting details about where, you know, the entire chain of trust all the way down to your tier dependency, even if they are open source, right? So they're not necessarily having to reveal the entire sausage and how it's made, but at least all of the different components that went into it.

 One of my other favorite parts, though not, you know, easy or hard or just I love seeing it called out. I mean, just talking about open source, the guidance kind of doesn't require, but it recommends reusing well-tested well-developed components, including open source. We're talking about open source security. There are a lot of misconceptions that open source is worse than closed source or anything like that. And you know, it's not open source software, it's just software the same as closed source software. And I think by in large, most people agree that you know, in general, open source software is or can be more secure than proprietary software.

It doesn't mean every single library on GitHub is, but I think on average the stuff you're using, you're in a much better spot if you're using, well tested, well-reviewed, well-maintained, open source libraries. And so the net framework actually calls that out and recommends it as a best practice rather than everybody trying to rip off the open source out of their supply chains. Right? That would be a mess. Yeah, set things back even further.

 So you bring up a really great point then, and this is one of my little bugbears, I keep hearing this in the industry. A lot of folks and I've heard organizations, even on sales calls or other places, talk about, you know, you can be as secure as you want as an organization. But if you're consuming open source packages, you know you could be vulnerable. And I just feel like a little bit fearmongering right at the end of the day, it's really that the open-source code is in every supply chain. Let's be real. There are pieces of it everywhere. And that's the tried and tested one, you know? So why aren't we backing that as an industry and supporting those efforts? More so because that's where we all collectively, as humans benefit a little bit more from the improvements. Right, Dan?

Absolutely, absolutely. I couldn't agree more, really. In fact, I do think that frequently, you know, open source software effectively has more eyes on, you know, in the process of its development. You know, by its very nature, it's probably wanted to sort of audit and check so. And yeah, no, I totally agree with that. And I know even, well, Paddy probably knows better than me. But you know, within Cloudsmith, we, of course, use open-source software to build Cloudsmith, and we open source as much of our own stuff as we can and we contribute where we can. So, you know, I think that's something that we would definitely agree with.

Paddy did you want to say anything about it?

 Yeah, Cloudsmith, like a lot of other companies, wouldn't exist were it not for open-source, you know, being able to trust that software, to be able to trust that the things you were building on top of is crucial. And it's also crucial for our customers. Our customers need to be able to trust that our supply chain is secure in the same way, we need to be able to trust the supply chains of our providers, by its cloud providers and even their hardware suppliers. It has to be secure the whole way down.

And I guess we'll get there eventually, right? Yes, it's got a long, hard road to get there, so I was moving in the right direction.

 That's exactly right. I mean, this is a journey right there and then we're all sort of on together and, you know, reunified by our use of open source and that there is no quick fix. You know, one size fits all solution here, but it's about figuring out what the bits of that solution look like. You know, no, nobody can do it on their own.

And I think that's where Dan Lorenc, you also mentioned, you know, this is security by default, where there are no quick wins. But we need to make it easier and simpler for folks to be able to consume security as a part of the supply chain, right? We don't want it to be like this thing where it's like an afterthought.

So for those speaking of journeys, for those of us following along with the actual agenda, we are now into the why is this a hard problem to solve? Right? So I think a lot of this is it comes from a challenge, like you mentioned before, about the software supply chains and pipelines, right? We have complex pipelines now. So the more that we bring automation into which we want, we are also, you know, introducing probably, possibly a little less large on some of our code with the automation and pipelines that kick-off.

And also some complex dependencies. We depend on software, which depends on software. It depends outside of the radar. So that's where I think this can be a tricky, tricky problem, right?

Yeah, I absolutely. And, you know, we've all seen this. You can pull in one dependency that maybe has 100 dependencies of its own, maybe more. Sometimes it gets pretty ridiculous at stages. And that's what that's only really one part of it, really. You know, it's a hard problem on many fronts. It really is.

Yeah. And I think on top of that, I mean, of course, every attack that we hear about right, SolarWinds , Kaseya, Log4Shell, that are out there...was it the DevOps security report that mentioned, I think attacks have been up this past year by 600 percent or so? So there's definitely a lot of exploitation going out there in the software supply chain.

And, you know, historically, Dan Lorenc, a lot of the stuff that's been put out there. I mean, I know I've worked on code before where I tried to sign things and different packages or even different pieces of code in PowerShell. But it's historically very difficult to sign, get a signing key or be able to sign anything you know, when you either have to pay a lot of money or try and try to jump through a lot of hoops with the servers and, you know, like signing mechanisms. So I think it's safe to say that these mechanisms used to be a little bit clunky, right?

 Yeah, absolutely. I think, you know, not for any real hard reason or anything if there just wasn't demand for it, really, I think. If you sign a package in the force and nobody verified it, did you really sign it at all before announcing? So it's complicated to get started. Like if nobody's asking for signatures or anything like that, then you're not going to do them. And if nobody is used to asking, then nobody's going to provide them and you kind of get stuck in the cycle. So you have to kind of break out of that somehow.

And I think the attacks we've seen have kind of proven the need for it. And so we're seeing some pretty rapid adoption and Sigstore and a lot of these other projects just because of that. Now, everybody, it's where people are aware of it. Software producers are trying to do the right thing. And software consumers are trying to do the right thing now, too. So it's kind of a great time to be in the field trying to build tools to make that easier.

 Yeah, no, I agree. It's funny because I liken it to and this is probably a poor analogy, but it was sort of like a Rube Goldberg game sort of contraption to try and find things. You have to make sure a server was available. And historically, with cosigning, I know back when I was at Chocolatey, you have to pay for some of these signing certificates, right?

 So, yeah, I think this is good because, like this, the conversation's just naturally flowing this way. But like here, let's talk about what's being done. Let's talk about some of these big open source projects. So I think the natural starting point to this is Sigstore, right? Do you want to reference that?

 Yeah, sure. I can tell a quick story there. We're studying cosigning and why it wasn't being adopted in different ecosystems for a while within the OSSF and Google. And we saw kind of a similar thing play out across all the ecosystems where, you know, the tooling wasn't great. People proposed adding signatures to Python or npm or Ruby or something like that.

And it's just really complex and you start thinking about key distribution and who's going to verify what and what is the signature even mean on these conversations just kind of stalled out because there wasn't much of, you know, motivation. But then on the flip side, there was all these kind of walled gardens and other commercial ecosystems where signing was happening all the time, right? If you publish an Android app or, you know, Windows drivers or something like that, you need to sign them and so in those worlds 100 percent of everything you signed just because you had to.

And the way those toolchains worked and everything was with kind of traditional certificate authority hierarchies the same way TLS works in your browser or open source alternatives were most commonly using something like PGP and Web of Trust, which was kind of finicky and hard to get right. And so when we started looking into the commercial code signing stuff it, you know, just kind of gave us all these flashbacks to what the internet used to be like, you know, five, six years ago.

If you remember trying to set up like an Apache server or something and get the, you know, TLS, you'd have the green shield next to your browser rather than the Red X, right, you know you'd have to find them, you'd have to pay them, you know, a couple of hundred dollars, they'd do some verification. You'd have to send them something on official letterhead for your company, right?

Do you remember the little logo that used to be in a corner of web pages - "Geo Trust" or "Web of Trust"?

Yeah, yeah, encrypted. And you just put that there. That's not how that works.

But yeah. And so, yeah, you pay in that email, you a certificate and you copy it into the right directory and rename it and run these arcane commands. And if you got it all right, then you get the green shield and then you probably forgot all about it. A year later, the certificate expired and you'd have to remember how to do the whole thing again.

Because everybody was complaining, right?

Yeah. And then Let's Encrypt came along and kind of changed it completely, right? They set up a free certificate authority as a public betterment group and got accepted by the browsers and all of us, by automating everything. Instead of certificates being valid for a year with these kinds of intrusive checks on your identity. They made a whole new standard called Acme, where you could prove that you had control of a domain. And you know, the certificates are only good for a couple of weeks or if something bad happened, it would automatically expire. They do these continuous checks. They got this going. And then within a couple of years, it went from like 50 percent adoption to like 99 percent of the web. And it shows, yep, people do really want to do the right thing. If you put it in their tooling and you don't have to think about it anymore.

Securing The Software Supply Chain Linux Foundation Webinar [On-demand Session]

Continuous Software Pipelines: Why Enterprises Are Going Cloud-Native Dev Week Enterprise Open Talk

Your entire tech stack is likely in the Cloud - so why aren’t your software packages?

Whether you’re currently on-premise, have your own in-house solution or have a bit of a hybrid set up, join us in this session to explore:

- Why enterprise organizations are making the move from on-premise solutions to completely Cloud-Native ones

- What this means for improving, scaling, and securing their CI/CD pipelines

- What the benefits of this are over cloud-hosted

- How to easily set up a secure, cloud-native software pipeline in 60 seconds.

Continuous Software Pipelines: Why Enterprises Are Going Cloud-Native 2021 Dev Week Cloud Keynote

Why are enterprise organizations making a move from on-premise solutions to completely cloud-native? What does that mean for improving, scaling, and securing their CI/CD pipelines? And what exactly is continuous packaging, anyway? Join Dan McKinney in this DeveloperWeek Cloud session he answers all of these questions, helping attendees understand the true difference between cloud-hosted and cloud-native, how to get started with migrating to a cloud-native solution, and the true benefits of being entirely within the cloud.

Securing End to End Software Delivery With Cloudsmith & Buildkite

Why would an enterprise organization move from an on-premises solution to one that's completely cloud-native? What does that mean for improving, scaling and securing their CI/CD pipelines? And what exactly is continuous packaging, anyway?

Combining Cloudsmith’s continuous packaging with Buildkite’s integration and delivery for a true continuous software pipeline in the cloud

Applying shift-left security principles across the whole value stream

Buildkite Test Analytics, a new suite of features that helps identify, track and fix problems in your test suites

Securing End to End Software Delivery With Cloudsmith & Buildkite [On-demand Session]

UnblockConf21 – Buildkite Webinar with Cloudsmith

Join Ciara Carey, DevRel at Cloudsmith, and Nick Rycar from Buildkite, as they explore what it means to combine Cloudsmith’s continuous packaging with Buildkite’s integration & delivery to truly have a continuous software pipeline in the cloud.

"Build It Yourself, They Said. It Will Be Worth It, They Said" Dev Week Enterprise Keynote Session

“We’ll build it ourselves!” We’ve all heard it, seen it, and likely been directly impacted by the decision to build a custom, in-house solution rather than use an existing one. Whether it’s a CI/CD tool, artifact management solution, or even the entire DevOps tech stack, it’s a common misconception that building it internally is easier, cheaper, and faster. When, in fact, the complete opposite is true! Modern development processes demand high levels of availability and performance, so building a custom system that provides both of these isn’t a trivial undertaking. So why, then do enterprises continue to build in-house? And what exactly are the benefits of purchasing an existing solution? And how does this impact the quality of their overall organization? Join Cloudsmith’s Dan McKinney in this session as he answers all of these questions while helping attendees understand the true difference between building and buying DevOps solutions, how to make the best decision for your organization, and the benefits of an existing solution made to solve the problems you face.

Cloudsmith blog