What were the trends for DevOps tools and culture in 2022? I talked to Aisling Conroy, Senior Manager of Competitive Intelligence at CircleCI and Luke Kilpatrick Director of Developer Experience at LinearB, to review the year and discuss key learnings and trends at our December webinar.

In 2022 did we get any better at developing software?
- Did we make any headway at automating human frailty out of building software?
- Did AI make a difference in building software?
- Does the focus on multi-cloud mean the death of on-prem?
- How healthy is open source after the big shock of Log4Shell?

2022 started with a huge amount of ambition from tech companies which did so well during the pandemic. As the year went on, the belt tightened, and so did plans; as Aisling put it, "everyone has a plan until you get punched in the face." She notes although the grand plans may not have been executed, we got a lot done, but there is still much more to do.

Some of the 2022 trends Luke and Aisling identified included:

Platform Engineering dedicated to improving workflow

AI tools to augment the software developer

I'm going to sum up 2022 DevOps as "the year of the workflow", where we have dedicated people and tools focused on fixing and improving workflows.

Both Linear B and CircleCI are trying to speed up developer workflow.

Aisling tells us that at CircleCI, they want an optimal-size pipeline run of around 10 mins. She sees pipelines on the CircleCI platform become more advanced as customers build more advanced applications than ever, but in some cases, software delivery cycle times have stagnated. "Teams have gone from once a quarter to once a month, which is great, but ideally, we want teams to be able to move at the speed of their creativity."

To improve speed, Aisling tells us there is a focus on tooling, better integrations, interoperability, and spending less time on the manual stuff to become more efficient.

Luke notes that LinearB's customers tend to do a great job of optimizing the ops pipeline but if there is a problem, there are still humans involved. At LinearB, they have done a lot of research on the developer team workflow. They have looked at 10's of thousands of code reviews (PR's) and other areas where there were bottlenecks and found a massive bottleneck around code reviews. So they took this information and tried to help fix the problem. This is what lead to gitStream. Like most things in computing, humans are usually the slowest part of the system. If you have an amazing CI/CD system using CircleCI or other tools, your deployment and delivery problems get reduced. There has been a ton of amazing work done here in the past 10 years. However, checking quality and making sure good code is getting merged is still hard to do. The gold standard is still a peer review of the code.

The biggest slowdown is a lot of small changes don't need this review and a lot of the big ones need to get reviewed more. So LinearB made a tool to help, and best of all, it's free!

Dedicated Experts in Workflows- Platform Engineering

Luke notes that the previous years in DevOps have been focused on the ops side, and a lot still is, but we saw a lot of great stuff coming out to help the developer side of the house. There is also a shift to Developer Experience improvement, also called platform engineering. This really became a big thing in 2022.

Luke likened a platform engineer to a sous chef working alongside a high-end chef. Most high-end chefs are only as good as their sous chef.

Having dedicated experts at a senior level reviewing and improving pipelines and workflow is a big trend for 2022.

The panel agreed there was a big focus on identifying and improving flaky tests. CI tools are highlighting and running metrics on your testing suite. Rerunning flaky tests or turning them off is not good enough. Companies are expected to do more with less, and robust test suites are an avenue for this.

AI tooling for software development is having a moment- from GitHub's CoPilot, CircleCI acquisition of Ponicode to the recent surprise of ChatGPT, which forced StackOverflow to ban answers from the AI tool.

At the start of the year, Aisling was quite bullish on AI, and she saw it being used to help developers.

She notes that this is an example where we all got very excited at the start of 2022, and we all thought we'd be living in a different world by the end of the year. We've seen many companies make efforts in the AI area over the past year, and it's been good progress, but actual progress has not come close to where we anticipated it to be.

But as we see AI tooling moving to where we write code, like CoPilot and Ponicode, it is bound to change how we develop software in the coming years.

Boring Programming Languages are making waves

I love comparing languages- the loyalty people feel towards their chosen programming language and the ecosystems it creates are the closest software comes to football.

y on the best and worst programming Languages for developer workflow. LinearB's metric dashboard has used its data on thousands of dev teams and hundreds of thousands of pull requests to see how languages compare with respect to PR lifespans.

It turns out, that Java and C# have a significantly shorter PR lifespan compared to other languages.

Luke tells us that Java and C# are more mature languages that have great tooling integrated into their workflows.

This year the discussion around hybrid cloud has shifted to multicloud. I wondered if this meant that on-prem is just yesterday's news - case closed?

Both Luke and Aisling disagreed!

Aisling says that there is always going to be some users that will need on-prem, maybe in niche industries like mining, aerospace etc. The trend has been steady in 2022 and will continue in 2023. MultiCloud will continue to gain more traction. In fact, Google had this as a prediction in their GoogleNext conference in September. There is talk of US legislation specifying that large corporations must follow a multi-cloud strategy. But there are still certain types of computing that customers might want to have in-house. Apple M1 is an example of this, though I think we'll see a lot of progress in the coming year in cloud companies being able to provide a growing variety of configurable compute options.

Luke, as a former Nutanix guy, believes on-prem will always have its place, but cloud tech will be part of it and moving loads from on-prem to public will become easier and faster. The tools for doing this are getting better. At the moment for large organizations, on-prem tends to be cheaper in the long run. Luke sees companies using a burst model where you can use infrastructure out to the cloud for big bursts of traffic.

MultiCloud strategy is definitely a 2022 trend, but it hasn't killed off on-prem just yet.

2022 started with teams scrambling to find out if they were affected by the 

 vulnerability found in a commonly used open source package called Log4J. It has become clear to most teams this year how dependent they are on open source software and that to truly secure software supply chains, a coordinated effort is needed from the whole software community.

For me, 2022 was the year where tooling began to help teams see what open source software is being used in their products and to help secure software supply chains. Open source tools like Sigstore, Grype, Trivy and Syft have made big waves this year.

Security by obscurity is thought less of:

GitStream is currently being giving it away from free

If you Open Source something, you better reach a certain standard

Aisling saw a trend in open source to improve interoperability in open source tools with Open Policy Agent tools. She was also impressed with how many open source software delivery tools matured, including Argon and Flux. Both Luke and Aisling see open source in a much healthier place at the end of 2022.

Concentration on workflows from a tooling and human level, appreciation of open source and glimmers of what AI can do to improve software development.

Join us on January 31st for our next webinar, which will continue the State of DevOps discussion - A Look Ahead at 2023. Registration information is coming soon.

Recap: 2022 State of DevOps Webinar

Security into your software supply chain means visibility into how an artifact is built, packaged and signed.

A considerable part of the reason supply chain attacks are such a threat is that we are missing information to decide if the software we develop and use is safe.

Audit logs, SBOMs, automation, and package management are all essential aspects to bring visibility into your software supply chain so you can answer questions like:

Who wrote the code, who built that code, and who approved the code?

What 3rd party dependencies do I use in my software, how trustworthy are they, and am I up to date?

How do I trace my build from source to deployment?

More than inspecting, scanning, and monitoring historical data, this information can be ingested by observability tools that can help you gain insights, identify unusual behavior and possibly prevent a software supply chain attack.

I’ve listed 10 ways to make your software pipelines more transparent and generate the information needed to gain your observability insights:

Automate your builds using a CI/CD tool like Jenkins, CircleCI, or BuildKite. Make sure all the other tools you use in your build process, including your package repositories, tests and code scanners, facilitate automation using APIs, CLIs, and webhooks.

Having your build steps codified with no manual steps means they can be logged with timing and authentication information. You can gather unusual login attempts, policy changes, or build patterns. Using in-toto attestations at each build step can make this information verifiable and thus more trustworthy.

Audit logs capture all actions the system, users, and service accounts perform for compliance and troubleshooting purposes.

All your build tools should produce audit logs, including your CI/CD, packaging, deployment, and cloud infrastructure tools.

Audit logs can be forwarded to the Security Incident and Event Management (SIEM) for visibility, tracking, and managing of security incidents. SIEM tools also provide next-generation detection, analytics, and response.

It's much more difficult for people and scanners to figure out what is installed on a system when software is installed using scripts instead of a package manager.

A software package, artifact, or image is the output of building software- it groups files containing your software along with the metadata about the software and dependencies in a well-defined format.

A package manager is a software tool that uses the power of automation to create, upload, install, upgrade, and configure software packages. Most software languages, containers, or operating systems have a corresponding package manager, e.g., NPM, NuGet, Maven, or RPM.

Dependencies installed with a package manager ensure consistency and transparency in how software is included in your product and give a straightforward way to access your package's metadata, including version information, license information, dependencies, environment state, etc.

Let's make it easy to figure out what is on our system by installing software with a package manager.

4 Versioning Software and Version Control

Consistently versioning your software product releases, e.g., 

, lets you keep track of all the changes and progress you make. For consumers of your software, the version numbers should inform them of when it's imperative to update and if there are backward compatibility issues.

Mistakes in build can be a hassle, but if at all possible and avoid republishing your builds.

For visibility into your supply chain, we need to know all the 3rd party dependencies in your software. Some package managers, such as pip and npm, do their best to resolve package information look-ups when explicit guidance isn’t provided. This means that a different dependency can be included in your software depending on when you build it e.g. if a dependency has a new update.

If you pin your dependencies to an exact version in your manifest files, no matter when you build your code, the same top-level dependencies should be pulled in, making your build more transparent and consistent.

You need to know when your dependencies have updates for visibility into your software build.

The previous step recommended pinning your builds, but if you have many dependencies, it can be very difficult to update each one manually.

Several update tools help you check for updates in your code base, including Renovate and Dependabot (GitHub).

 is an automated tool that gives developers a score on how safe their dependencies are. Understanding the trustworthiness of your dependencies is vital to improving the security posture of your project.

Signing a package/artifact/image lets people know that this package was signed by a person or organization and proves that the code was not tampered with after it was signed.

There are issues with signing, including a lack of understanding of the signing process, cost, complexity of managing keys, implementation of code signing pipelines, and the lack of verification of signatures.

Recent developments in signing to reduce the complexity of signing and remove the cost barrier to signing with the 

 project have invigorated efforts to sign artifacts.

A Software Bill of Materials (SBOM) lists all the components in your software. The notion of the SBOM was popularized by the Whitehouse when it was namechecked in an 

 last year on improving software security.

An accurate SBOM would improve the transparency of your supply chain, which could allow for:

Earlier identification of potentially vulnerable systems

Support informed decisions for 3rd party dependencies

Incentivize secure software development practices.

 is still evolving, but hopefully, it will be integrated into build tools, making this process accurate and automatable.

It is no longer just about the published artifact at the end of your software pipeline. Instead, it is about providing the tools to ensure artifact integrity across the supply chain.

Information demonstrating the supply chain provenance of a built artifact includes a combination of the steps above, including signing, build attestations, and SBOMs.

Provenance allows you to know and prove the origin of your software.

Monitor and Observe your Software Supply Chain

Some of this information isn’t so easy to generate but will help us monitor and observe our software supply chain, which will ultimately help secure it. This information will help us:

Monitor software supply chain more effectively

Enable visibility for DevOps, security analysts, and developers into the entire build process

Find and connect events in our build and trace them back to the source

Detect and mitigate against vulnerabilities earlier

Detect and alert for unusual login activity or build activities

Observability tools are promising to answer questions relating to the information that your system is generating, even questions that are not currently anticipated- what could this do for our software supply chain?

10 ways to make your software pipeline more observable

Everything You Wanted to Know About Securing the Software Supply Chain but Were Afraid to Ask

From the history of supply chain security threats to security development and deployment we've covered everything you’ve always wanted to know about the software supply chain but were afraid to ask. Dan Lorenc, Founder/CEO, 

, Paddy Carey, Senior Staff Engineer, Cloudsmith, Adil Leghari, Solutions Architect Manager, Cloudsmith and Dan McKinney, Developer Relations, Cloudsmith, gathered for a fireside chat to cover your most burning questions:

 Types of attacks – availability, dependencies, development tools, and more.

Software supply chain challenges and considerations

A look at OSS projects and initiatives that were born out of the SSC security need.

 What have Cloudsmith & ChainGuard been working on to make these issues easier to tackle, and what you and your organization can do to help.


Read the transcript of the full webinar below:


Hello, everyone. Good morning, afternoon, good evening. Depending on where you are in the world, on behalf of my colleagues here at Cloudsmith and Guard, I want to welcome you to our discussion entitled “Everything You Wanted to Know about Securing the Software Supply Chain, But Were Afraid to Ask”. So this is going to be an open discussion or conversation, so feel free to hop in, you know, with questions if you want to. We do have a Q&A section specifically in Zoom for that, so it helps if you throw your questions in there because it will.

Definitely, everybody will get the benefit of the answer if we answer it on the call. In addition to that, of course, like, you know, we're all here and I want to welcome you. But I think over the last few years, software supply chain security specifically has become top of mind as one of these concerns for almost every organization, right, that's producing and consuming software. So we are the tech community have to try and understand this. But if we're going to wrap our heads around it, we really need to have conversations like these, you know, to improve our collective understanding of where we came from and sort of where we're going. So hopefully, by the end of this talk, we'll leave with not just a bunch of fear, but hopefully a little hope as well about the future of supply chain security and software.

So let's go through the agenda real quick here. I will first off just do a quick little speaker introduction for all of us. Everybody will do their own introduction and then we'll give you a little background on what we do with our organizations, with Cloudsmith and Chainguard as well. And then we'll get into the sort of the meat of the torque where we're talking about, you know, some of the things that we want to bring up today. As and again, this will be open, sort of like a fireside chat. We're just discussing stuff. And so feel free to ask questions as well and help my fellow speakers feel free to hop in at any point.

We'll be touching on points like what got us here, specifically when we talk about previous tipping point events, sort of attacks, and other things that precipitated our big concern about the software supply chain. And then we'll touch on why this is such a hard problem to solve because there are definitely some challenges there. Each attacking each different compromise or issue that comes up really has different origins and different techniques and methods. So it can be quite tricky, sort of a moving target.

And then we'll talk about specifically what's being done where we highlight a few open-source projects in the space, specifically around the efforts that Dan Lorenc has done over at Chainguard and also at his time at Google and with open source. So thanks for being here with us, Dan. And then also we'll specifically talk a little bit about at the end about what are we doing at organizations like Cloudsmith and Chainguard, and also how you as an organization can help and contribute to the same efforts? Sound good?

OK, well, without further ado, let's introduce ourselves a little bit. So I'm Adil Leghari, I'm a solution architect manager here at Cloudsmith. I'm a sys admin turned solution architect turned manager. I spent the better part of two decades here wrangling Windows boxes and Linux boxes. So over the past five years, I've become an active member of the PowerShell and the devops automation community, and I'm a speaker and author and super passionate about PowerShell, dev ops and automation, yada yada yada over to you, Paddy.

 I'm an engineer here at Cloudsmith and I've been an engineer for a long time, starting in the tech and e-commerce and finance, and eventually find myself deep in package manager land, which is how I ended up here at the Cloudsmith. I've worked on at everything here, from core package management functionality, securing supply chains and my own to get our platform team part of the responsibility there is securing our own supply chain.

So hopefully that's a relevant experience for me to talk about today.

 Hi, everyone, yes, I'm Dan McKinney, and I work in developer relations at Cloudsmith and I've had a somewhat tortuous path through the developer relations. So I've worked in systems engineering and manufacturing, actually another type of supply chain as well. And yes, it's really my role to sort of engage with the community and educate the community, which is why this is a perfect forum to do this today.

I'm also an absolutely crazy and retro computing and video game collector, and I work as a DJ at the weekends. But yes, it brings a little bit of color to the proceedings. So nice. Thanks, Dan.

: Yeah, second Dan today. My name is Dan Lorenc. Currently, I'm a founder and CEO at a new startup called Chainguard. But before that, I was at Google for about nine years, where I got to spend the last five or six years working almost completely in Open-Source and started worrying about open-source supply chains. So I've been involved in a bunch of different open source projects there and continue to be in products like sig store and salsa products or SLSA. So we have to talk about some of those later today.

: Absolutely. Thank you and thank you on behalf of all of us. Thank you for your efforts and for keeping things open for us. Awesome. OK, well, let's talk a little bit about each other as organizations, about Cloudsmith. I'm going to let Dan take this one.

Yeah, thanks. So Cloudsmith is a cloud-native package management solution. So given how essential software artifacts and packages, in general, are to our software supply chain, it's very high on our agenda to secure that supply chain. So we have been around a lot longer than a lot of people think.

If they're only hearing of Cloudsmith, we've been around and this will be our sixth year. So it's a very exciting time for us because as you said earlier, it'll, you know, secure software supply chains are really at the forefront of people's minds at the moment. And it's so close to our hearts as well. So it's a really exciting time to be part of that. So that's who we are.

Yeah. So yeah, we've not been around for very long. We've been around for about five months. Chainguard here. We got started in October of 2021, and we were trying to hold a bunch of to invest in open source to help people make secure make their software supply chain secure by default. We've seen time and time again that if you make developers do things differently if you make them change the way they work, security is always an afterthought. It's not what you're thinking about when you're in that development loop.

But we believe that if we do this the right way, and if you take a couple of steps back and think about the problem a little bit, we can actually make secure software development a better and easier way and have it work that way by default. So that's what we're focused on doing. We're still just getting started. So thanks a lot for having us here and giving me some time to talk to everybody.

 Dan and I must say, you're being very humble. You guys are slowly picking up the who's who of Kubernetes and other SLSA supply chain projects. So good for you over there. We eagerly await your announcements.

All right. So let's hop in a little bit. And so now what I'm going to do is I'm going to take the slides off the screen because I think this is a discussion. So let's open it up a little bit and talk about what got us here, right?

The scary elephant in the room is all the attacks and everything that went on right? But I think there are a couple of events that will probably highlight that have helped shape the industry as far as the software supply chain. And I think one of the first ones, of course, is the SolarWinds attack, right? So that was December 2020. SolarWinds, if you don't know, it was a very prominent company, that makes software for network infrastructure management. So in December of 2020, they announced that their Orion software had been compromised as a result of the software supply chain attack.

And so this is funny because this attack very much was managed to burn the word "software supply chain" into our collective consciousness, right? Because everybody now knows that word, that phrase specifically, because its malicious code was inserted into the Orion software early enough and that supply chain, it remained undetected. So what happened was that pushed out with the other updates to the Orion software and slowly got out to consumers. And you know, we're not talking about small consumers here, we're talking about big folks like Microsoft and Intel, etc. So little did they know that they were installing those malicious payloads along with the monthly updates.

That's really serving to highlight part of our problem, that up until this point, developers were largely ignoring these parts of the pipeline. This is sort of a tipping point event for that sort of alertness. Now, Dan Lorenc, I know I've heard you talk about this before, but I think you were still a Google working on some open-source projects at this point, right? And this is sort of one of those moments, those tipping points.

Yes, this is the kind of right after I switched from Google Cloud to Google's kind of core security groups, I finally been successful in convincing some people at Google that this was a big enough risk to take seriously is like an existential issue for the internet and for Google. And it was still kind of slow, though, until, you know, the kind of news broke about this attack and the amount of people affected and everything. Almost overnight, we went from a "why is no one listening to us?" to "why haven't we been doing this for years?".

And was quickly followed up with, the stuff I think you're going to shout about in a minute with, you know, the U.S. government looking around to announce executive orders and regulations and new compliance and everything. So yeah, it has become an increasingly hot topic over the last couple of years, unfortunately, because of attacks and not because we were as an industry ahead of the game. But it tends to work that way, unfortunately.

 Yeah, it's not a problem until it's everybody's problem, right?

So, yeah, that definitely was one of those moments, I think for everyone in the industry. I was working at the time, I believe at Chocolatey software and you know, we kept on getting that question again and again where suddenly folks were like "hey, you know, we need to secure software supply chains?" And I'm like, "OK, that's great". I mean, we have kind of been preaching this stuff for a little bit, but now suddenly we're getting all of this attention, right? So I think I think a lot of folks perked up their ears here and sort of your C-level staff to hire folks who are normally not as concerned suddenly started tuning in and asking these questions.

Yeah, I've seen pictures of board decks now with this on a slide from every CSO out there, so it's something everybody's trying to rapidly come up with an answer to.

 I know over the past year everyone's learned the acronym SBOM, and it's kind of funny because it's like, really, we've been talking about this for a while. So I think, yeah, leading onto that timeline was right.

So in February 2021, I think was one of these announcements, where Alex Birsan, who is a security researcher, showed that he was able to compromise supply chains as well. And he was luckily a white hat hacker. So we didn't have the benefit of stress or the negative effects of that. But of course, it had been copycat attacks since then, right?

I want to highlight this one specifically because I think he did this basically by, you know, sort of subverting the logic in the actual process of some of the normal supply chain, right? So as developers, a lot of times we want to use dependencies, right? Obviously, we don't want to reinvent the wheel every time. And so this is specifically referred to as dependency confusion. And it's similar to sort of typosquatting as another type of attack where, you know, essentially what he did was he introduced public packages into widely used public registries that shared the name of private packages for a lot of big organizations. And we're talking 35 different companies. We're talking names like Microsoft, Apple, PayPal, Tesla, you know, like it wasn't a small attack, but basically, he was able to very simply do that by creating packages on public registries with similar names.

So it highlighted another thing where it's the functionality of the package registries in the package management system by default are to trust the public registries and pull those down. But you know, it's the whole "who watches the watchmen" at that point, we're not sure. I mean, everybody had the overnight to kind of figure out. Wait a minute. We have to actually change the logic of how we do this stuff. Right, and I think Dan McKinney, you were mentioned. Yeah?

 I mean, I think the real problem until there was that, I mean, essentially this was a problem with like configuration, the way people had configured the package manager clients to, you know, be able to hit public sources in preference to their private internal repositories. But there were no red flags like, you know, it went ahead. It wasn't that it was, you know, in contrast to SolarWinds, which was a breach of their build system. This was normally running build systems that just were, you know, configured incorrectly but didn't shout about it when something unexpected happened. So, because pipelines are so automated now, that's what enabled that, you know, the automation kicked in, packages get pulled in, things get built. And there was no point where something shouted to everybody.

And why is why was there no point when that happened? Kind of like Dan said earlier that it's not the default and it's not easy enough to have that default stance of, you know, something unexpected or there's a problem or there's something we suspect may be a problem, and maybe we should have eyes on this. The heavy automation sort of drove that. And look, I'm a huge fan of automation. I mean, obviously, automation is what enables, you know, companies to move with a real, high velocity and deliver software quickly.

And that's fantastic as an industry. But it has to come with checks and balances because, you know, automation run amok well will lead to this kind of problem. That's, I think, a core difference between dependency confusion and the SolarWinds attack. Both were supply chain breaches, right? And both resulted in malicious or potentially malicious software getting out into the hands of users. But there's a subtle difference there. The lack of visibility on that process and the lack of alerting and automatic red flags, is what really caused the problem.

 Mm-Hmm. OK. So this is great because I'm going to pick on you a little bit and say so because I love you. So we have a Q&A question. So while you're at this talking about this specific software? Moussa asks, 

"starting from the basics, can you please define what is meant by software supply chain?" 

So defining our terms, of course, where everybody wants to be on the same team?

 And I'll have a go with that then, yes, I'll have to go with that. And well, look, I'm a big fan of doing this as well, so apologies to everybody, but I love to draw analogies, right? So and I worked for five years in manufacturing. It was a silicon wafer and fab, actually, so I was very familiar with the term, a "bill of materials", right? And what goes into a final built product that's very easy to see in manufacturing, right? Because you know, your suppliers, your raw materials, they come in, they go through a production line process and the other end comes a finished product.

And that's the same for software supply chains, right? You have raw materials that you ingest almost. And those are the open source libraries and projects that you're building upon and you're using as components in your finished product. Unlike the manufacturing industry, the software industry sort of hasn't reached that level of maturity yet of manufacturing. So for manufacturing, for raw materials, and for supplies that you bring in, there are certificates of conformance. Everything goes through a quality analysis. You look at a company like Apple, you know, they audit their suppliers heavily right back to the, you know, the mine that the rare earth metals were mined out of, you know, and they have all of that traceability on the supply chain from raw materials right through to the finished product that you hold in your hand.

And as an industry, the software industry kind of needs to get there. Now a problem with that is that it depends so heavily on open source, everybody uses open source and open source is fantastic. We're a great supporter of open source at Cloudsmith. So we don't have those centralized industry bodies that do quality control and conformance checks like manufacturing dollars, which is exactly why we need what Dan said for the software industry. We need zero trust because we don't have those centralized armies and organizations to prove the provenance of an open source library or to give that traceability all the way back to the source. So that's what a software supply chain is. It's everything that goes into making up your product and, of course, dependency of dependencies. It's just like the raw materials in a physical supply chain. You know where a screw can be traced back all the way through the metal that was used to make it all the way back to the main that the metal came from. And that's like the chain of dependencies and so on.

 Exactly. And I will save some of the problems is with that for the later parts of this conversation with their natural points, and we can discuss that too. So just to summarize a lot of what you're saying, which was awesome, is the idea that this bill of materials that people keep talking about - software build materials as SBOM stands for software build of materials.

And you're not only trying to understand where code comes from, but it's sort of each step in that process, right? What dependencies did I use? What image was this built on? What factory? If you want to say that. What repository was used? What commit was specifically added this code? And all these sorts of details kind of like. I know Adolfo from Chainguard says that sometimes, right? It's a product packing slip on the side of a UPS package. The details, you know, this was checked by this person at this point went through these waypoints and came from this factory and all that was detailed. So that's kind of where we're getting out of the software supply chain.

 And Adil as well, even the environments, even the tooling that was used to build an artifact that's part of that as well. You know, it's the complete bill of materials and not just the raw materials themselves, but the environment that they went through. You know, the versions of the tools even down to, you know, the build host that was used for a particular pipeline run. It's all it's all relevant.

Adil Leghari: Absolutely. So, OK, the last date I'm going to highlight real quick is May 12, 2021, and the Office of the President, of course, on the state-issued Executive Order 14028. There won't be a quiz later if you don't even know. But it was on improving the nation's cybersecurity, specifically. And so this really highlighted that this was a real issue that the government was willing to throw its weight behind, right?

And they really talked about removing barriers to the sharing of threat information and, you know, modernizing some of these practices in cybersecurity that had kind of fallen by the wayside, but only about actually establishing government-run cyber safety review boards and details like that. So, I think this was a big point for the industry, for sure, Dan Lorenc and I think you definitely were involved in some of this or your projects were referred to in some of this, right?

The Secure Software Development Framework (SSDF)

Yeah, yeah. No, it was a kind of fast couple of months from the attack to kind of the analysis allegations coming from earlier the year before. I'm out to get this executive order that was kind of just the start of that road to right the executive directive on how to produce software. Right? It was a directive to a bunch of other agencies within the government to start collecting industry feedback and use that industry feedback to shape similar recommendations that get applied in purchasing decisions and that kind of thing.

So we're still seeing kind of that process unfold and actually two or three weeks ago on this, the National Institute of Standards and Technology in the U.S. finally published - the not final version they're going to keep iterating on it - but the, you know, a final version of the SSDF, The Secure Software Development Framework, which is really a great read and if you haven't seen it yet. But this is kind of the culmination of talking to people and industry, people in your government, people in the public sector, and private sector listening, what the best practices were today and then leading all of those into a, you know, checklist that you can run through.

And, you know, just an almost like a self-assessment for your organization. I'd highly recommend taking a walk through it if you haven't yet, it's broken out in a couple of different sessions. We've got something on the Chainguard blog about it too. But this is basically, you know, the industry set of best practices here for both consumer and secure dependencies and then also operating build systems securely in your supply chain. There's some stuff in there that I think will surprise most people because, you know, there's a lot of stuff which is basic, you know, brush your teeth style hygiene that everybody knows they need to do. And then there's a couple of things in there that are kind of pushing the envelope a bit. And I think you're going to catch people by surprise later this year if you don't get a handle on it now.

 So what do you want to mention any of those?

 Oh, man. Yeah, well, I mean, there's one funny one that I missed, but there's one funny one that I'm curious to see how it plays out, but it requires, you know, vendors to try to comply with the SSDF. Do you have to also apply that recursively to their vendors, which makes perfect sense? Yeah. But yeah, it's going to be very slow until the first person tries to meet this. And then all of a sudden it's going to cost the industry because we're all in a supply chain together. Most of us are somewhere in the middle, right? So long before it starts, well, creeping up on people.

 That's the tough part. Right is its providence all the way down. Right? You really need to get folks to buy into it. So I think if once large organizations start hopping on that bandwagon, I think the industry, you know, just by sheer momentum alone is going to be forced to comply to a lot of the stuff. So as organizations, what you're going to see is if you're selling products and software, you're going to start seeing more and more security. You're going to start seeing more and more from your customers and consumers wanting details about where, you know, the entire chain of trust all the way down to your tier dependency, even if they are open source, right? So they're not necessarily having to reveal the entire sausage and how it's made, but at least all of the different components that went into it.

 One of my other favorite parts, though not, you know, easy or hard or just I love seeing it called out. I mean, just talking about open source, the guidance kind of doesn't require, but it recommends reusing well-tested well-developed components, including open source. We're talking about open source security. There are a lot of misconceptions that open source is worse than closed source or anything like that. And you know, it's not open source software, it's just software the same as closed source software. And I think by in large, most people agree that you know, in general, open source software is or can be more secure than proprietary software.

It doesn't mean every single library on GitHub is, but I think on average the stuff you're using, you're in a much better spot if you're using, well tested, well-reviewed, well-maintained, open source libraries. And so the net framework actually calls that out and recommends it as a best practice rather than everybody trying to rip off the open source out of their supply chains. Right? That would be a mess. Yeah, set things back even further.

 So you bring up a really great point then, and this is one of my little bugbears, I keep hearing this in the industry. A lot of folks and I've heard organizations, even on sales calls or other places, talk about, you know, you can be as secure as you want as an organization. But if you're consuming open source packages, you know you could be vulnerable. And I just feel like a little bit fearmongering right at the end of the day, it's really that the open-source code is in every supply chain. Let's be real. There are pieces of it everywhere. And that's the tried and tested one, you know? So why aren't we backing that as an industry and supporting those efforts? More so because that's where we all collectively, as humans benefit a little bit more from the improvements. Right, Dan?

Absolutely, absolutely. I couldn't agree more, really. In fact, I do think that frequently, you know, open source software effectively has more eyes on, you know, in the process of its development. You know, by its very nature, it's probably wanted to sort of audit and check so. And yeah, no, I totally agree with that. And I know even, well, Paddy probably knows better than me. But you know, within Cloudsmith, we, of course, use open-source software to build Cloudsmith, and we open source as much of our own stuff as we can and we contribute where we can. So, you know, I think that's something that we would definitely agree with.

Paddy did you want to say anything about it?

 Yeah, Cloudsmith, like a lot of other companies, wouldn't exist were it not for open-source, you know, being able to trust that software, to be able to trust that the things you were building on top of is crucial. And it's also crucial for our customers. Our customers need to be able to trust that our supply chain is secure in the same way, we need to be able to trust the supply chains of our providers, by its cloud providers and even their hardware suppliers. It has to be secure the whole way down.

And I guess we'll get there eventually, right? Yes, it's got a long, hard road to get there, so I was moving in the right direction.

 That's exactly right. I mean, this is a journey right there and then we're all sort of on together and, you know, reunified by our use of open source and that there is no quick fix. You know, one size fits all solution here, but it's about figuring out what the bits of that solution look like. You know, no, nobody can do it on their own.

And I think that's where Dan Lorenc, you also mentioned, you know, this is security by default, where there are no quick wins. But we need to make it easier and simpler for folks to be able to consume security as a part of the supply chain, right? We don't want it to be like this thing where it's like an afterthought.

So for those speaking of journeys, for those of us following along with the actual agenda, we are now into the why is this a hard problem to solve? Right? So I think a lot of this is it comes from a challenge, like you mentioned before, about the software supply chains and pipelines, right? We have complex pipelines now. So the more that we bring automation into which we want, we are also, you know, introducing probably, possibly a little less large on some of our code with the automation and pipelines that kick-off.

And also some complex dependencies. We depend on software, which depends on software. It depends outside of the radar. So that's where I think this can be a tricky, tricky problem, right?

Yeah, I absolutely. And, you know, we've all seen this. You can pull in one dependency that maybe has 100 dependencies of its own, maybe more. Sometimes it gets pretty ridiculous at stages. And that's what that's only really one part of it, really. You know, it's a hard problem on many fronts. It really is.

Yeah. And I think on top of that, I mean, of course, every attack that we hear about right, SolarWinds , Kaseya, Log4Shell, that are out there...was it the DevOps security report that mentioned, I think attacks have been up this past year by 600 percent or so? So there's definitely a lot of exploitation going out there in the software supply chain.

And, you know, historically, Dan Lorenc, a lot of the stuff that's been put out there. I mean, I know I've worked on code before where I tried to sign things and different packages or even different pieces of code in PowerShell. But it's historically very difficult to sign, get a signing key or be able to sign anything you know, when you either have to pay a lot of money or try and try to jump through a lot of hoops with the servers and, you know, like signing mechanisms. So I think it's safe to say that these mechanisms used to be a little bit clunky, right?

 Yeah, absolutely. I think, you know, not for any real hard reason or anything if there just wasn't demand for it, really, I think. If you sign a package in the force and nobody verified it, did you really sign it at all before announcing? So it's complicated to get started. Like if nobody's asking for signatures or anything like that, then you're not going to do them. And if nobody is used to asking, then nobody's going to provide them and you kind of get stuck in the cycle. So you have to kind of break out of that somehow.

And I think the attacks we've seen have kind of proven the need for it. And so we're seeing some pretty rapid adoption and Sigstore and a lot of these other projects just because of that. Now, everybody, it's where people are aware of it. Software producers are trying to do the right thing. And software consumers are trying to do the right thing now, too. So it's kind of a great time to be in the field trying to build tools to make that easier.

 Yeah, no, I agree. It's funny because I liken it to and this is probably a poor analogy, but it was sort of like a Rube Goldberg game sort of contraption to try and find things. You have to make sure a server was available. And historically, with cosigning, I know back when I was at Chocolatey, you have to pay for some of these signing certificates, right?

 So, yeah, I think this is good because, like this, the conversation's just naturally flowing this way. But like here, let's talk about what's being done. Let's talk about some of these big open source projects. So I think the natural starting point to this is Sigstore, right? Do you want to reference that?

 Yeah, sure. I can tell a quick story there. We're studying cosigning and why it wasn't being adopted in different ecosystems for a while within the OSSF and Google. And we saw kind of a similar thing play out across all the ecosystems where, you know, the tooling wasn't great. People proposed adding signatures to Python or npm or Ruby or something like that.

And it's just really complex and you start thinking about key distribution and who's going to verify what and what is the signature even mean on these conversations just kind of stalled out because there wasn't much of, you know, motivation. But then on the flip side, there was all these kind of walled gardens and other commercial ecosystems where signing was happening all the time, right? If you publish an Android app or, you know, Windows drivers or something like that, you need to sign them and so in those worlds 100 percent of everything you signed just because you had to.

And the way those toolchains worked and everything was with kind of traditional certificate authority hierarchies the same way TLS works in your browser or open source alternatives were most commonly using something like PGP and Web of Trust, which was kind of finicky and hard to get right. And so when we started looking into the commercial code signing stuff it, you know, just kind of gave us all these flashbacks to what the internet used to be like, you know, five, six years ago.

If you remember trying to set up like an Apache server or something and get the, you know, TLS, you'd have the green shield next to your browser rather than the Red X, right, you know you'd have to find them, you'd have to pay them, you know, a couple of hundred dollars, they'd do some verification. You'd have to send them something on official letterhead for your company, right?

Do you remember the little logo that used to be in a corner of web pages - "Geo Trust" or "Web of Trust"?

Yeah, yeah, encrypted. And you just put that there. That's not how that works.

But yeah. And so, yeah, you pay in that email, you a certificate and you copy it into the right directory and rename it and run these arcane commands. And if you got it all right, then you get the green shield and then you probably forgot all about it. A year later, the certificate expired and you'd have to remember how to do the whole thing again.

Because everybody was complaining, right?

Yeah. And then Let's Encrypt came along and kind of changed it completely, right? They set up a free certificate authority as a public betterment group and got accepted by the browsers and all of us, by automating everything. Instead of certificates being valid for a year with these kinds of intrusive checks on your identity. They made a whole new standard called Acme, where you could prove that you had control of a domain. And you know, the certificates are only good for a couple of weeks or if something bad happened, it would automatically expire. They do these continuous checks. They got this going. And then within a couple of years, it went from like 50 percent adoption to like 99 percent of the web. And it shows, yep, people do really want to do the right thing. If you put it in their tooling and you don't have to think about it anymore.

Securing The Software Supply Chain Linux Foundation Webinar [On-demand Session]

Securing End to End Software Delivery With Cloudsmith & Buildkite

Why would an enterprise organization move from an on-premises solution to one that's completely cloud-native? What does that mean for improving, scaling and securing their CI/CD pipelines? And what exactly is continuous packaging, anyway?

Combining Cloudsmith’s continuous packaging with Buildkite’s integration and delivery for a true continuous software pipeline in the cloud

Applying shift-left security principles across the whole value stream

Buildkite Test Analytics, a new suite of features that helps identify, track and fix problems in your test suites

Securing End to End Software Delivery With Cloudsmith & Buildkite [On-demand Session]

We are excited to announce that Cloudsmith has secured $15 million of funding in our recent Series A round. This latest round will help us continue to build best-in-class technology for today’s software engineers and their organizations by evolving cloud-native package management and providing a secure, single source of truth for all software artifacts and assets.

Cloudsmith was built to tackle the complexity of managing software assets, a key challenge for any company to secure their supply chains. Especially in 2021.

With the Cloudsmith platform, we enable our customers across the globe to track and control the distribution of any software asset and provide the tooling to minimize the risks of utilizing open-source software.

It’s an industry-wide issue and our investors, led by Tiger Global, understand the need for a fully managed cloud-native solution that Cloudsmith provides.

Shaping the Future of Software Supply Chains

CI/CD and DevOps are part of modern software supply chains, but a genuinely holistic view from source to delivery is required. A solution that powers global infrastructure at scale and provides criticality of performance, observability, and isolation. A logistics-based smart CDN that offers controls and insights; a software-aware Package Distribution Network (PDN).

As engineers ourselves, we understand that ownership of how our customers’ software artifacts, containers, and packages are pulled in, stored, and distributed is vital. It's the central pinch-point that everything should pass through, and going forwards, as supply chain attacks increase, every single opening in your supply chain is a vector for attack.

Our complement of continuous packaging, which is the glue between CI and CD in the supply chain, positions packaged assets as first-class traceable entities in the supply chain's bill of materials.

With support for 24+ formats, 225 PoPs, 24/7 engineer-driven support, and an ever-evolving partner ecosystem with leading tools like Buildkite, HashiCorp, Snyk, and CircleCI, our mission is to evolve the cloud-native supply chain, making it simple for organizations to secure their software delivery at scale.

Venture-Backed by Top Software Automation and DevOps Firms

Our Series A round is led by Tiger Global, with participation from Shasta, Sorenson, Amaranthine, Leadout Capital, and previous investors Frontline, MMC, and Techstart.

Scott Johnston (CEO of Docker), Sarah Friar (CEO of Nextdoor), Yvonne Wassenaar (CEO of Puppet), Shane Curran (CEO of Evervault), Matt Klein (Lyft, creator of Envoy), Yun-Fang Juan (ex-Facebook), Cian O'Maidin (CEO of NearForm), Babak Hamadani (Primeset, ex-Twitter) and Kit Merker (Nobl9) also participated in the round, raising our total capital to $15 million.

We are also incredibly lucky to have these awesome Angel investors who participated in the round that truly understand and believe in the Cloudsmith mission.

Our new funds will be used to grow the Cloudsmith team, deliver on our product roadmap, and continue to expand into international markets.

To every Cloudsmither, partner, customer, and advisor of Cloudsmith, thank you for your continued support. We honestly couldn’t have done this without each and every one of you.

Thank you for choosing us and believing in our product and our mission. We may be five years in, but this is only just the beginning.

We’re beyond excited for this next phase of Cloudsmith, and we’re currently hiring across all departments.

 and come help us revolutionize cloud-native supply chains!

Cloudsmith Raises $15m in Series A to Evolve the Future of Software Supply Chains

Deep in the woods, where trees are black and the air is thick, steam rises wistfully across the damp ground. A single dirt track, barely wide enough to pass, scars the terrain for what seems like an endless number of miles. It winds its way through the mountains and valleys, across a rickety bridge over a cavernous ravine, before plunging back into darkness, the trees bending over as if to grasp those passing through. Finally, in a small clearing, a lonely decrepit wooden cabin reveals itself.

The cabin sits unobtrusively in its setting, a small ranch porch makes it almost look inviting. The surrounding trees sway and creak in the wind as if whispering a foreboding message of warning. Evil lurks all around it.

But it is deep within that the problem lies.

In the rotting floor, a trap door reveals a rickety staircase which leads ominously down into the blackness below. A single candle is enough to illuminate a narrow claustrophobic path, deep into the tight bowels of the cellar. The stale air is suffocating. Down the passage, through an opening in the rock-built foundations, a small room houses an old oak desk sitting dusty and alone. The world has forgotten it. In the bottom drawer, locked with a key long lost, is a book wrapped in a thick, almost rigid cloth. Within this book are the incantations to unlock a devastating demonic possession.

A public software repository is the cabin. The book is the dependency you're not tracking. The aimless, unwitting temporary residents are the developers and processes used to build your software supply chain. Without knowledge. Without protection. Bad things will happen.

And without realizing it, the incantations are spoken, and the demonic possession is unleashed. You’ve opened a door, a conduit to another place. A scary and unrelenting place. Filled with demons that want your soul. And a vessel to infect the rest of your world. A host to do terrible, horrible things. Things that no one wants or expects. Control is lost.

So fire up the chainsaw and load your shotgun… tackle the demons head on.

Cloudsmith blog

Recap: 2022 State of DevOps Webinar

10 ways to make your software pipeline more observable

Securing The Software Supply Chain Linux Foundation Webinar [On-demand Session]

Securing End to End Software Delivery With Cloudsmith & Buildkite [On-demand Session]

Cloudsmith Raises $15m in Series A to Evolve the Future of Software Supply Chains

Dead Evil: A Software Supply Chain Possession

The Internet of Things - Why Cloudsmith Plays a Crucial Role

Modern Tech Stacks need Multi-Format Repositories

Go Cloud-Native or Go Home

SolarWinds and the Secure Software Supply Chain

Cloud-Hosted or Cloud-Native? Discover Why Cloudsmith Was Born in the Cloud

Cloudsmith, the JFrog Bintray Alternative & Replacement