Hey everybody, thanks for tuning in today to Cloudsmith's monthly webinar. Today's topic is on how do mature DevOps teams manage security. So before we get started, let's go through a few housekeeping notes. We have prizes to give out. So we have Two free lunches and two free prize packs to give away at the end of the webinar.

So be sure to watch till the end to win a chance, to be in a chance to win. We're also streaming on Twitter, on LinkedIn, on YouTube, as well as our webinar platform. Please post questions to whatever platform you're using. The wonderful Hilary will be monitoring those channels and giving them back to me.

We're going to be holding one or two polls. And again, you post in the. your platform or tweet or chat, and we will be looking for those questions, your answers to your polls. So we have two really amazing guests today for our talk. So let's bring them on stage. Hey everyone.

Hey Nigel. Hey. Hey Jack. So today we have Nigel Kirsten and Jack Chester. Nigel Kirsten is the field CTO at Puppet by Perforce. And he's the author of Puppet's Much Loved Report on State of the DevOps. And we also have Jack Chester. He's the... A senior staff software developer at Shopify. He's an author of a book, K native in action on building services applications.

And he's the chair of a OpenSF software repository working group and heavily involved in Ruby's open source community. So thanks for coming today. Hey, and Nigel, so you have released 10, I think that's the full 10 reports, every, that's a lot on the state of the DevOps. So how was that? It's been a

pretty massive effort over the years, and I have to say, you know, I'm sort of the last person standing, so to speak, but the people who, if you're going to talk about the history of the state DevOps report, who I think had a bigger impact than you also, Alana Brown.

Who's since moved on and now works at remote. com. It was her idea in the first place and she really drove it for a number of years. I was co author. And then when Dr. Nicole Forsgren came on for four years, I think it was there. She really brought a level of statistical rigor and research to the whole project.

But there's been so many people. Turnbull, Jez Humble, Michael Stonkey. We've had so many great authors over the years. But last year for us was a really big one because it was 10 years and Suddenly made me realize how long I'd been messing around in this industry. I had to look up the term for this the other day of semantic satiation.

You know, when you say a word over and over and over again, and it stops, it loses all meaning. I think DevOps and DevSecOps are kind of like that. Yeah. You say DevOps 20 times and it doesn't mean anything anymore. You try

And so what's, when's number 11 going to be out or are you still going to be as big a part to it?

Yeah,

I'm, I'm guiding it at the moment. We've got a fantastic researcher, Ronan Keenan, who's taken on sort of the bulk of the work and is working with some research firms for us. We're trying to do something a bit different this year because I think, and we can chat about this, this topic goes forever, but basically I think.

DevOps is now such a big big field, it's very difficult in a single report to sort of come up with interesting, useful findings. You have the folks at the beginning of their journey, the folks who are very much post DevOps, the folks who've moved on, who've tried it, who it doesn't work, who it does work.

And trying to do all of that in a single report, I think you just end up producing a book every year. So this year we're just focused on platform engineering. Yeah. I can't write a book every single year. It gets to you.

We feel the same. Yeah. Yeah.

Writing a book is hard. Yeah. Absolutely.

JAck tell us about how you recently, I know you're focusing on open source security and I'm a long time listener of your working group um, software repositories, and I'm just wondering how you got into that and how, what was your journey

to.

Yeah, I've been, I've been co chairing or deputy chairing. I don't know how you want to describe it. With Dustin Ingram from PyPI from the Python Software Foundation. How did I get into it? I used to work for a company called Pivotal. whIch, you know, I really enjoyed my time there. And one of the things I worked on at Pivotal was what was called Pivotal Network.

It was our distribution point for all our software products, which we needed to do legally. And some of the products got installed in cages under armed guard. bEcause they were fairly sensitive sort of operations. Is this like an euphemism or really? No, this is really a thing that happened. Like the, the, the software, like the USB stick would be walked under arm guard.

Like it was, it was that kind of a place. Okay. And I, I suddenly thought there are people in the world who would be very interested. In getting inside those cages through our software. anD that was one of those Oh, expletive moments. And that, that led to my interest. That was, that was sort of like the, the lightning bolt that led me down the path to where I am today.

Oh, cool. So our topic today is how do mature DevOps teams manage software security? So I thought my first question I'll pose to Nigel, like, like I know we're saying DevOps means nothing anymore, but what does DevOps mean? Is it just automation and cloudy stuff? It's just tools, right?

Yeah, I mean, this is, I think it's a tough one.

And you talk to folks like Patrick Dubois, who very much coined the term. And there was a deliberate, it was very deliberate that there wasn't a clear definition here is exactly the reductive definition we have of what we're trying to do here. Because in many ways, if we tried that, if you look back at the early days, it was basically a bunch of sysadmins going, how do we actually be agile?

How do we actually take Agile in spirit and apply it to operations? Oh, look, we have all of these cultural problems, all of these accountability, you know, ownership doesn't match authority, all of these things. And I think we had had such a vibrant, interesting, exciting space emerge because it wasn't really tightly defined and you'd turn up to DevOps days and you could get a talk about just about anything.

But then we hit the enterprise. And I think the lack of a definition meant that honestly, like shyster vendors stepped in and started going, we do DevOps here is DevOps in a box or consultants coming along in a similar way to agile and safe and various, you know, sort of permutations like that. that I don't think are particularly true to the original spirit.

So, as far as what it actually means to me, I take a pretty big tent approach. It's a loose collection of practices, technical and cultural, to get over organizational boundaries inside organizations so that we can ship software with less stress and better. Like, and that sounds really vague and could apply to just about anything, but every time I try and narrow it down much more than that, I ended up cutting out something

I think is important.

Yeah, it was like, I've worked on those teams where it's like every six months. I think that was a normal kind of thing. You would release something and it would be very stressful. Something would go wrong and then you'd have to roll back. And it was, it was a high stress moment in a big, long journey. So I think that moving away from that is, is a good thing.

You think a

bridge call over the weekend?

Yes, no one ever wants to do those calls.

Yeah. So and so how does that how do you see security? Is security becoming. Bring, being brought into it more. It wasn't like at the start, we tried to merge development and operations. And now we're like, not just now, but now we're like, Oh, security are kind of still a bit siloed.

Let's bring them back into this tent. And is that how you see it, Jacque?

Yes. And unfortunately, and just because of the sort of the economics of the situation, it's going to be siloed for a while in a lot of ways. There's just not that many cybersecurity folks to go around. So a lot of organizations either deliberately without thinking about it or out of regret wind up with a central security team that acts as a gatekeeper.

Which we know from our DevOps days is, is an anti pattern. tHe other thing I see as an anti pattern is again, very much like the experience of DevOps went through the evolution is the idea that there's a box of software you can install and today you have security and that's not true at all. And it's, it's.

It's a pity that we have to go through this evolution, but I'm, I'm hopeful that we'll come out the other side with something better.

And I know it's not a box, but is there like some nice tools

that can like, give you a bit of a log up? Yeah, it is important to think carefully about your tooling. Dan Lorenz, who's the CEO of a company called Chainguard says a lot, and I agree with him.

I had a similar sort of motto once upon a time, which is that build is production. The, the systems where you are building the software are as sensitive and you know, risk dense as production itself. Because as I said, you know, like if someone gets into the bucket of bits, you are in a world of hurt.

And a lot of the time people Historically have underrated that risk. And so the bucket of bits has been the fastest path into production to attack the build system itself or the the artifact system itself as well. So in terms of your software, you should think carefully about, about those systems and securing them and hardening them and applying all the security practices you have now to them.

bUt I think there's, there's sort of like two great tributaries of risk or two great tributaries of security risk that you can think about flowing into, into the river as it were. And one of them is that build system, upstream dependency, you know, risks that come from the outside of the organization and then risks that come from the inside and the really big one is making sort of, you know, unintentional errors in your software that lead to a vulnerability.

That one, I think doesn't get as much time as it needs to. Because it's hard again to install something, or it's hard to have a, you know, a checklist that says I have now secured myself against security errors.

Yeah, actually, one of the times where you probably as a developer have the most power over security is when you're bringing in these dependencies.

Like what, like, what is it that you should consider when you're like considering bringing in a brand new dependency? Like what are the, you could like, what is the checklist? Should you have a checklist or are you, can you, or should you be able to test anything on your developer?

Well, yes and no. So that that's an emerging field right now is, is people producing these checklists.

There's even a startup called socket. dev who have, have sort of automated the checklist for NPM at least. Yes, I would broadly say like, take the things you're already doing. So is this project lively? You know, is it active? Are people still contributing? Do they respond quickly to problems? You also want to look at security practices.

Like, do they have MFA enabled on the repository accounts that they use? But also you want to make sure little things like, are you accidentally installing a different dependency from the one you thought? Are you making a typo? So double check that you're getting the package you expect to get. lIttle things like those can add up to a lot.

But I think we're in the early days of having, having a strong story about how to pick dependencies with a security point of view.

It's funny, something you said there, Jacques, I wanted to jump on it because I think one of the things that's underpinning all of this is how hard it is, you know, software development is a team sport.

The teams keep getting bigger and bigger and bigger with different roles. And it's often really hard as an individual practitioner to actually make a good decision, whether you're locally or globally optimizing. And I think that's what a lot of this stuff comes down to. It's like your, your job that you're being measured on is to ship some software, implement some features, resolve some bugs or whatever.

And. If everyone just goes for the shortest possible path to get there, you end up in a situation where the environment they're operating in becomes more fragile, more error prone, more insecure. And yet we're just not very good as human beings working in large groups. How do you surface the right kinds of things to make a decision between local and global optimization?

I don't have a solution

here. No, if you have a solution, then I urge you to Put your name in for a Nobel prize in economics. Exactly. Because that would be a pretty big breakthrough.

Yeah. And there's so many points in the software life cycle, like the source code, the CI CD system, the artifact depository.

The dependencies, the external dependencies on public repos and then all the tooling you use as well. You're like, you're scripting your environmental variables. Like it's just, there's just a lot.

There is, there is. And that's, that's one of the hard things about being a software developer is that there's so much to know about so many topics that it's hard to be an expert in everything.

I, again, I wish, I wish I had the solution where I could just, you know, do a sort of an Isaac Asimov thing and you play a tape and that puts a memory in your head, you can, you can tell how dated that story is.

That was a really good 10

years ago. Yeah, exactly. You put in, you put in the reel to reel and some blinking lights and there you go.

But I think there's, there's still a lot of value in. Creating a minimal level of awareness of the possible issues. You don't have to necessarily know the solutions. You just have to know A, that there might be a problem here and B, where you can get help.

Yeah, absolutely. And I know both of you are, are, have talked about how cultural change and how people are actually And focusing on people is, is a great way to get better security.

Do you want to talk about cultural change in DevOps and how to, how to get your DevOps processes? Really nice and secure using culture, Nigel.

Sure. So I think there's a, there's a bunch of things to, I think, unpack there. One is that, you know, DevOps and, you know, a lot of the most significant tech movements we've had of how we build software.

These are grassroots movements, these weren't by people at the top of the hierarchical pyramid inside organizations, these are people who are down at the bottom. And so, it's easy to sort of go we have a cultural problem. And one of the things we found out from last year's State Development Support when we did a bunch of qualitative and quantitative research was that, Organizations with lots of what we would call cultural problems talk about culture all the time.

But organizations that don't have many of those sorts of problems, They stopped using the word culture because it's not, it's not actionable. And it's actually encourages a weird kind of form of helplessness inside organizations. Like if you're an individual developer and you're like, ah, well, our culture doesn't allow for people to just make those decisions.

Everyone goes, ah, you know, it's like an earthquake. What are you going to do about it? You know, you just sort of wait for it to move on. But organizations that actually implemented these sort of changes and had fewer cultural problems. Somewhat paradoxically don't talk about culture. They talk about specific things.

We have a problem with ownership, we have a problem with making decisions quickly, we have a problem with documenting tribal knowledge or ancestral knowledge around a code base. Like all of these things are quite actionable. Yeah. And one of the things I found really interesting last year with the Team Topologies authors, Manuel and Matt, who, if you haven't read Team Topologies, it's one of the best organizational design books ever around tech.

And their definition they came down to was, stop talking about culture. Talk about what you need to do to ship software quickly with low cognitive load and stress on individuals. If you actually look at those things and identify them, then they start becoming things people feel like they can do something about.

So that was a really long winded way of saying, I think culture is massively important, but you've got to go at least one level below and go, what is it we're trying to actually achieve here? Like, let's not just say culture and throw up our hands. But let's go, what's the problem and how

are we going to fix it?

And then you can make like incremental changes and get better and better and better until you've just...

And people, people can tell if they're making a difference, as you say, when they're working incrementally. One of the things I found really frustrating when I worked at Google was there was this ineffable phrase, Googliness.

And people would go, well, that's not very googly. And you're like, I don't actually know exactly what you mean. And I'm pretty sure you're just using this as a weapon to get your point of view across.

You need to get that at the Google o meter. Exactly. How googly is this?

And so on that, do you guys think that metrics are important to improve software security?

Is it like part of improving your DevOps? Like, I would say it's like

metrics. Yeah. Sorry to cut you off. No, no. But to answer the question as, as I see it, metrics are essential. They are not enough. And as we all know If you govern purely by the metrics, two things happen. One, anything that's not in the metrics, you will ignore.

And two, if what you're doing is like a control loop where you have a little controller. You think of yourself as a little controller. You've got your sensors, which are the metrics coming in. And then you've got the actuator, which is you doing stuff to the system. It turns out that if you want to improve the difference between the target and what's actually happening now, the easiest thing to do is to fiddle with the sensor, right?

It is much easier to. to gain the metrics than to actually improve the system. So you need to be aware of that. And the reason that that's important is that if you tie punishment and reward to metrics, they will be immediately gained to within an inch of their life. So those would be the two cautions I'd give about metrics.

Yeah.

That's very human to like change the, change the measuring system.

It's a really good example of this, where they tried to incentivize all of the tellers to getting everyone to open up bank accounts. And instead what they end up finding out was that all of these tellers on mass were doing the sort of natural optimization there, which is just going, okay, let's open up lots and lots of accounts with people, whether it was a good idea or not.

Oh yeah. I remember when I was in I was in Curry's it's like a electronic store and I was a cashier and I had to really get my metrics up. On selling insurance, I think on product, but I didn't see what the product was. And I was like, this is my, this customer is getting my two cents. Would you like insurance on your product?

And she just looked at me and goes like, no, it

was a cleaner bag.

We're good. You know, I asked the question.

Yeah, I would say use, use metrics to sense the environment. But as I said, beware tying punishment and reward. Like if it didn't work for the Soviet Union who had unlimited authority to try to make it work, an unlimited supply of men and women with guns and dogs to try and make a metrics governing system work, then it's not going to work for you.

Right. So, so use with caution.

Yeah. I think that's a good example. So to kind of cut you off here, it's like, cause I get asked this a lot about the big four metrics that came out of. The work we did with the Dora folks and that they ran with, you know, the mean time to recovery, change failure rate, et cetera, et cetera, deployment frequency.

And it is horrifying what people out there in the real world have done with these metrics. They're a sane collection of four metrics that pull in different directions, so you can't optimize one too much at the cost of the other. But you literally get teams inside enterprises competing on how to improve all of these things.

And, you know, exactly as Jacques was saying, like, you can improve deployment frequency and mean change failure rate by deploying more often and not being as good at measuring it, looking for errors. And so you'd get these teams optimizing for 1%, 2%, 3 percent improvements in these metrics. And sort of losing sight of the biggest picture but to, I guess, bring this back to a security lens.

The thing I often talk to folks when they're trying to do DevSecOps inside organizations at the start of this journey is like, how quickly can you push a change to production and know that it's actually gone out? Because If you can't do that quickly, if you can't respond to something, push out a fix to it or a change of any kind and know whether it worked or not.

Like, that is just the 101 sort of substrate. And you can spend all this time optimizing all sorts of other policies and processes, but if you can't... Create change in your environment quickly and reliably and be able to see the results of that change. Like, stop caring about DevSecOps and all this thing.

Just fix all those things first. Yeah,

the worst time to find out that you can't deploy to production quickly and safely is in the middle of a security incident or an outage.

Absolutely.

Yeah, I'm sure people have found out that recently with Log4Shell.

And on Log4Shell is like do you see critical vulnerabilities and updating your software, all the dependencies as in having the process for that being really important or What do you, yeah, do you, so actually on that, we have a poll. So the question is, do you pin your bills or do you update to the latest?

So this is sort of, this question comes up with, it's mostly around vulnerabilities. Well, not, there's loads of good reasons to update, but with respect to security, when you if you update to latest, you'll get all the fixes, but if you pin your bills, you're not going to be Tricked into updating to a bad version.

so, and so we see here, there's, there's most, it's kind of half and half, but most people prefer to update to the latest. So, 24 percent said, 49 percent update to the latest. 40 percent say pin my bills and the rest are, it's not important to me. So I think, and I, I don't really feel like this question is solved.

Yeah. So I, in Cloudsmith, we always say we recommend to pin your bills, but like, if there's a critical vulnerability, it'd be great if you're updated as this. Quickly as possible. So I totally see the other side. So we like to say pin your bills, but then use tooling like the Panda bot or I think Renovate is it, to give you a prompt, an alert, a PR to, with an update to the latest, and that'll kind of quicken that cycle.

So what do you guys think on that topic?

This one's a bit of a hornet's nest. I'll let Jack answer this more in more detail, but I'd say. At a high level, the way I feel that is, some of it depends on scale. If, if you're like two developers who own the whole system that you're in, like, you know, in a very small startup, the answer's very different to if you're a multinational bank with regulations and, you know, hundreds and hundreds of teams interacting with each other.

I do think, you know, the big problem with auto updating to latest all the time is when are you creating that artifact that, are you testing some, like, are you creating something that's going to be tested in a test environment? Are you going to be able to reproduce that artifact again? I think there's some nuance here and it involves, you know, probably doing a mixture of both, but choosing when in your software delivery lifecycle you do each of those activities.

The, the most depressing answer from experts is there's nuance

it depends.

Well, on the one hand, and on the other hand um, I I'm broadly in the camp that you should pin your dependencies in source code and update them automatically. I don't like mystery dependencies showing up in production without warning and without a record.

That makes me deeply uncomfortable personally, but I recognize that it's a hassle. We are sort of like in, I don't know, like not quite the prehistory, but we're definitely at least no further than the bronze age in terms of dealing with this stuff. We have technology, but it goes blunt easily and causes a lot of, a lot of hassle.

And we just need to learn to grow the muscle to do it. And that's just going to take a lot of time and be sporadic and uneven. But I do agree with Nigel's point that there's sort of minimum standards of hygiene you need to reach first. You need to have good testing and CI in place. You need to have smooth the road to production from source code changes.

Those are the same capabilities you will need. to automate upgrades. I will put an asterisk here about like the trade off and risks between waiting to upgrade versus upgrading too soon. And Sonatype have released their eighth state of the supply chain report a few days ago. It's worth reading. They do fantastic, fantastic research.

Their position is that you should hang back a little. You know, one on two versions behind the pace or maybe some amount of time I think would be a better way to do it on the theory that if you're right at the bleeding edge, you will, you will get cut from time to time and that it's not worth the risk.

I'm kind of on the fence about that. I think that the incidence of a vulnerability existing is far higher than the incidence of a supply chain attack being successful.

Yeah. So balancing risks. What about general bugs too? Cause this is the one that always gets me. Like there's nothing I find more frustrating than if you're developing something using a bunch of libraries or frameworks and you keep beating your head against the wall going, why is this not working?

It should be working. And then you upgrade a dependency and you're like, ah, it was actually a bug all along. I think. There's something to be said to staying on latest generally leads to a better experience.

Oh, it's also because upgrading is, is not just like a linear function of the number of things you have to upgrade.

It's exponential, right? Cause there are interactions between the dependencies. So the longer you lead it, you know, like the, the, the larger that sort of Cartesian join of doom gets. So you want to, you want to keep close to the edge if you can at Shopify, for example we have the monolith, which is the main application, but probably the largest rails app in the world.

And we keep that on rails edge once a week, once a week, we upgrade to what is literally in the main repo rails. Like we're not waiting to point releases or anything like that. We're keeping up with it because we know that the upgrade pain is just too large. If we hang back for a year, like it would just, just be catastrophic.

And I can, I can sort of look back at the earlier history of the company through, through documents and, and, and get, get commits. And I can see that pain and I can see why we did it.

Yeah. I just got off a call with a customer who's still on Red Hat 4 and is unlikely to ever get off because they, they, they left it too long and now they have to stay.

Slides, little bit of history that they have to work around. Are

you muted? Are you unmuted?

No, the webinar gods are against us. No, still muted.

This is how, you know, it's live everyone. Yeah.

So I think one of the interesting things while, while working out her audio is a lot of this conversation around security issues and software supply chains, it's often feels kind of one sided in terms of companies that are getting an awful lot of software sort of for free from volunteer maintainers who have been.

You know, every time one of these vulnerability comes out, it's like everyone has the pitchforks out for the maintainers who are like, you know, I was doing this out of the goodness of my heart. And I was maintaining that stupid backwards compatible feature because you all protested against it. I think something has to change about the producer consumer relationship with open source.

Like there's a general assumption that it's software of a certain quality. Everyone should try and write good software, but. Something feels out of kilter in society about the promises and commitments that people expect.

There's a, there's a really fascinating paper that just is, is currently in pre print SSRN, the Social Science Research Network.

It's a pre print server called Tragedy of the Digital Commons. Which, which is like written for a law journal, but goes into kind of like the economics of it, you know, like the law and economics kind of situation. And she makes exactly the same point, which is that large software companies in particular are free riding off the community.

In a big way her argument is that the, the sort of the ambient costs of security risks should be pushed back onto those companies to bear because they're the ones who are best able to bear it. Yeah. I totally

agree. Big tech, big tech loves open source, like sharks love fish, you know,

can you guys hear me now?

Sorry about that. Yes. Yes. But my laptop, the battery is gone. But anyway, I saw that legal letter that one of the log for J mod maintainers received. And it was just like, Oh, for the love of God, like he's like doing this for free. And you're like telling him giving him a legal letter to update and like from a company that's using his.

Code for free. It's, it definitely doesn't sit well, doesn't seem morally right or something.

I mean, I'm, I'm kind of in an interesting position here because I'm, I've been one of the champions for introducing MFA requirements for for software repositories, you know, where the, where the authors need to have MFA enabled because their packages are so widely used.

And in a sense that's imposing a cost, you know, it's imposing, imposing additional effort on the package maintainers who didn't, didn't ask for it. Right. And, and I do feel bad about that, but I then have to sort of take the utilitarian stance that the end consumers are far more numerous and for them the consequences are far more serious if, if there's a compromise.

It's, it's a tricky, it's a tricky thing, but I think the, the difference there is that like the end consumers can just involve other, you know, random open source developers who, who didn't expect something nasty to come down the pipe, as well as the companies who can bear the cost and should contribute back.

Yeah.

Yeah. I saw there was PyPI, they have some stats on who has converted to 2FA. It's, it's not like. Super impressive. It's like 20 percent of people that will eventually be asked, be forced to have 2FA, have turned on 2FA. I, is it, maybe some of them don't know about it or some of them don't want to do it and they'll just wait till they have to.

It won't be a big deal.

That's, that's largely what happened in, in Ruby. I, I know some of those authors because they work at Shopify and they said, yeah, we agree with logic. We're just not going to do it until you make us do it. Cause it's just, it's just work, right. It's an additional thing to do.

Yeah.

Yeah. And what about like, I've seen, I saw a list of things that maybe open source maintainers can do to be more secure, but it was like a lot of stuff for someone to do. It was like add scorecards to their repo. There was like, there was just a ton of stuff to do. Oh, do a course. Like, I just can't imagine if you're doing this in your spare time, that like a lot of people are going to do it.

Especially

when people often got into this because it was fun, you know, like, Hey, I solved a problem in a fun, interesting way, and I want to share that with the world. And I think, I don't know if software licenses are the way, or some kind of opt in system, but I feel like there's got to be a way to distinguish between, Hey, everyone, here's something fun and cool, have at it.

And I am deliberately building something that I would like to be part of a bigger structure and a bigger ecosystem. And I think. That's sort of the constant trade off. You don't want to, you don't want to stifle people just sharing code that is useful and fun, but there's got to be some declaration of intent

somewhere.

I think the sort of the coordination point, or choke point depending how you look at it, is probably going to be the software repositories, because they can set the terms under which they agree to distribute the software. And so if you, you don't like those terms, you are within your rights to take the software, which is open source into running yourself and within your rights to just distribute source from a website that you own, like there's, there's alternatives, like they're not as convenient, right?

They aren't, but, you know, that's, that's the trade off. Yeah, I think

that's, and it's similar to, all of this reminds me of, I was a Debian maintainer back in the day when you know, you were sort of in one of two big Linux camps. And I was quite shocked when I sort of moved to that point level of suddenly having all of these security processes enforced on me, but it was the right thing to do because that was the distribution center, you know, to all of these volunteers.

Oh, and like, so a

lot of these things were, Debian community already had a lot of these

Yeah, I mean, I think, you know, as, as much as, you know, I hate to, you know, particularly towards the end of the webinar, proclaim the death of the operating system distribution. A lot of these problems, I think, have been solved in smaller communities before.

We're just now dealing with them happening faster and a bit bigger scale. And, you know, in, in tech, I feel like we love nothing more than to ignore the. Discoveries of the past.

Yeah. And so I was about to say, so what do you think there are the biggest challenges in software security? Or is it like we've been talking about how there's just so many challenges and it's just all of them together.

But if you were going to give yourself a top one or two, what would be your. Favorites.

Well, that's tough. This, this goes back to that earlier discussion about culture versus practices. There's, there's this vast amount of latent risk out there, and we've just got to sort of chip away at, at everything that gives, right?

We're pushing in every direction at once, and anything that gives, we push harder because we're getting some progress out of it. We're retiring some risk from it.

Building up those layers of security.

Building it up and, and, you know, reducing the net risk for everybody, which is, which is the sort of the goal, you know, that there's that problem that open source is basically a comment, right?

Like it's, it's a kind of a resource that you can't exclude people from using, but where if lots of people use it, then that puts pressure on the maintainers, it's rivalrous as economists call it. And that's Commons, and they're difficult to govern, they're difficult to manage because, you know, everybody's an individual, they've got different incentives to, to be selfish, and the difficulty is finding those well positioned parties to be involved.

So, to their credit, I know we bashed up big companies, but to their credit, a lot of them are coming to the table. Or trying through the open source security foundation, which I participate in open SSF. So you've got your Googles and your Microsofts and your Amazons and, and a whole bunch of companies participating contributing money, contributing folks time, trying to sort of attack this on all fronts.

The trick is, is going to be like, to your point here, like, will it just seem like a loud crescendo? To open source maintainers, like here's a massive list of things that we can offer you. Where do I start?

Like I, is that the 10 point mobility plan is, is a part of the open SSF. Way to secure open source.

And do you feel like open source is One of the most important things to secure when we're talking about software in general.

Oh, yeah, yeah, it, it, depending who you ask, it's, it's present. That was a leading question. Yeah, it wasn't, it's the sky blue um, only on sunny days. Yeah, it's, it's everywhere now.

It's in pacemakers. It's in nuclear power plants. Like there, there isn't, there isn't a single critical or high, you know, high consequence piece of infrastructure. Whether social or technical that doesn't rely on it. We, we have to like, it's, it's the soft underbelly of, of the whole of the social economic system at the moment.

And what do you think about regulation? Cause I know the U S federal government is bringing in so many rules about S bombs and even vulnerabilities. And do you see that as a way to improve security of a product?

Strictly yes this, this is a good example of that argument from Tragedy of the Digital Commons article that the costs should be pushed onto the large companies that currently free ride and have the resources to not free ride and the U.

S. government is in a great position because it's the single largest purchaser of software in the world. To push, push those standards down and to make them common. And once they become common, then other consumers from those companies will say, well, you already have that capability. I demand it also. And that creates a sort of a flywheel effect.

But in terms of regulation of open source software itself, outside of those big companies, like your regular maintainer at home on a weekend. Dear God, no. That, that would, that would kill the golden goose, but not before the goose, you know? Defecated all over the bed.

Hey Nigel, what do you think?

And , I think what is a rather hairy thread to mix metaphors. We don't value maintenance enough in society, and I think this is sort of part of the problem that, you know, and this is why I think right to repair movements and all of these things are so important that. You know, you work in lots of, we have a culture in software development that I think reflects society in general at the moment, which is, it is considered better to launch new things than to iterate on existing things.

And the job of maintainers everywhere is to iterate on the existing things. And I think the healthiest software engineering environments I've ever worked in have been the ones where. Really senior folks are sort of proclaimed, you know, lauded for their ability to look at systems, make small incremental changes to them over time, keep them going in the right direction, and that that's recognized as valuable.

And I think this, this is sort of the whole problem. We recognize value maintainers anywhere near enough. And so they feel at the end of the supply chain when we should be going, you know, no, you're a critical part of this whole process. You know, if I could wait, it would be around us valuing the act of maintenance more so that big companies did want to participate in it so that they, you know, reached out to maintain projects with respect, you know, I think Google does a reasonably good job of this.

Like we've had Google reach out. It is a security vulnerability and something you, you ship, you know, we've seen some of our users have it, you know, they basically wield a big stick and go, if you don't do something about this in 30 days or 60 days, whatever, we'll just, we'll shout it from the rooftops and they can, cause they're Google.

I think there are ways to do that sort of encourage people to do the right thing, but fundamentally we've got to value the act and process of maintenance more everywhere.

I wonder if like Government funding could help. I know like obviously the 10 point mobility plan should improve security and that's using money.

I know maybe to there was talk about resetting, putting funding towards resetting to FAA. Shocking public repositories but do you see, do you think that maintainers could get paid for improving security of obviously selective products like that are used in critical systems? Like, do you think that would, that's a solution maintainable.

In the

long run, I'm concerned that it goes back to that problem of metrics. thAt, that it will, you know, the incentive is just to do what, what the funder says. And that will attract people, you know, like the, the story of the, the British trying to get rid of Cobras in India. And they pay people to bring in Cobra heads and people just started breeding Cobras.

Right. Or something similar where there's gun buybacks and people are just 3d 3d printing guns on mass and bringing in boxes of 3d printed guns and making money that way. I'm, I'm concerned about that. I think where government has a role in terms of funding, at least, would be on what you might think of as sustainment activities.

So, things like subsidizing or fully funding training, right? Making it freely available to as many people as possible. Encouraging colleges and universities to pick it up as part of their curricula. Things like you know, shared resources for software repositories. Shared resources for open source projects that need, you know, a security review, a lot of things that the open SSF is already doing can definitely be scaled up with government funding.

What do you think about punitive approaches to like, and this is something I'm always curious about because it feels like. Most of the huge companies that suffered data breaches that were honestly pretty derelict in terms of, not everyone, they just haven't been punished like either by the markets or by governments.

And so why would you invest in security when it doesn't actually

matter? Yeah, I, I, I'm, I'm a bit of a. You know, like I consider myself a centrist. I used to be a libertarian, but I'm about to sound like a raving, loony lefty, but because I think there is far too many things in corporate malfeasance in which the punishment is a fine, whereas it should be criminal time for the executives who authorized or who failed to authorize, you know, some activity because that's the only thing that actually gets their attention.

If you get fined, it doesn't fall on the people who made the decision. It falls on the shareholders. Exactly. people in the data bridge.

Right. Like Optus is a good one. Like you have a company that literally litigated, you know, pressured, lobbied the government to make sure companies weren't accountable in these sort of situations.

And then now all of these millions of people have had their data spread

all over

the internet. Including me, my passport number got stolen. Yeah, I actually, I was listening to the Security Weekly podcast and at the end of it they talked about insurance as a way to To drive companies to, to do more, to be better at security and it can be a more effective way than compliance or that, like when you, when you have a data breach and you realize you're not insured and you have to pay a lot of money to maybe for on.

peOple so suing you or even to get back to where you were, if you, if you've lost data, that, that is a, quite an effective way. Why

not? Why not? But no, we, we, we have punishments for people who, you know, like if you don't do fire safety in your factory, right, not only do you mess up your insurance and not only can you face fines, but the people who are responsible are criminally malfeasance.

They can go to jail for neglecting fire safety. You know, the consequences of, of data breaches are dire. The consequences of lackadaisical security are only going to grow worse as time goes on. And as all matters somehow becomes programmable, basically, then this stuff really matters. And I think this argument that like, Oh, but the corporate veil is sacrosanct.

It's just like the corporate veil is there to deal with you know, questions of who owes debt to whom, like who, who can be. Who, who, who is liable for how much? It didn't give you like a magical get out of jail card. That was never the idea. So as I said, I sound like a raving loony on this point because I'm so frustrated by companies that walk away with a fine and the executives are still there, right?

They don't get sacked, they just go like, Oh, well, that's the cost of doing business. And that to me is psychotic.

I mean, you know, yeah, there's not enough accountability at the corporate level. Absolutely. We need to run that one. Rise up and smash the system, Jeff. We're going to do it.

Right, nodes of all countries.

On that, I think we're going to announce our prize. Hillary, do you want to? Now that we've done the rally. We've gotten to where we were meant to go. So the prize that are announced there in the chat, we have Hiroku Jiyoto Duta, who gets a free lunch. He's sharing on, on the. Streaming platform, we have Arthur Courage, he has a free lunch, Jin Su Pricepack, Arjun Joshi Pricepack, I'm, I'm so sorry, I'm butchering these poor people's names, Caitlyn, Seo, Oh god, Caitlin, I'm so sorry.

You get a prize pack! Hunter Kuhn, prize pack. And Hilary is going to be reaching out to everybody over email with, with your details. To send it on your, send it on to you. But I hope everybody enjoyed our talk today. I loved it. I'm so sorry about my My speaker issues, you guys were such pros.

You continued on the conversation.

Another way to put it is that we talk to you much.

And thank you for being such a wonderful guest shock. And Nigel, it was like really nice to talk to you. So it's bye from our guests. You guys can say bye. Bye. Thanks for having us. bye from me. So thanks everybody for joining.

We'll see you at the next monthly Cloudsmith webinar. Bye. Bye.