We chatted with ReversingLabs Field CISO Matthew Rose about the ESF’s new guidance on software supply chain security.

EPISODE TRANSCRIPT

PAUL ROBERTS
Hey everybody, welcome back to ConversingLabs. This is ReversingLabs podcast, where we talk about all manner of topics related to threat hunting, threat intelligence, cybersecurity, software assurance, and more. I'm your host, Paul Roberts. I'm the cyber content lead here at ReversingLabs and very happy to have in the studio our own Matt Rose. Matt, welcome.

MATTHEW ROSE
Thank you, Paul. Thank you for having me. It's an honor to be on the podcast today.

PAUL ROBERTS
We're really thrilled to have you. Your name, you've got a reputation that precedes you. But for those listeners or viewers who aren't familiar with you and your background, Matt, tell us just a little bit about yourself and the work you do here at ReversingLabs.

MATTHEW ROSE
Sure. As Paul mentioned, my name is Matt Rose. I've been in the application security space and I'm really dating myself about close to 20 years now. There was a lot less white in the beard when I started down my journey. I've really been in the space from the beginning in kind of origin phase of SAST or static application security testing. It was with Fortify from their kind of beginning stages through acquisition by HP. I've also been with Checkmarx from their build up in the U.S. through acquisition by private equity. But really what I pride myself in is really working with organizations to define effective programs to find risk, and not only find the risk, but mitigate the risk. Because that's kind of one of the biggest problems I see in the industry, even after doing this for so long, that people are really good at finding stuff, but fixing it not so much. That tends to be more of a challenge and really like to walk people down that path and add some pop culture and some interesting references or analogies into the conversation to make it more conversational than presentational.

PAUL ROBERTS
Not so dry, right?

MATTHEW ROSE
Dry is bad.

PAUL ROBERTS
Dry is bad. So like you said, you go way back in the application security testing area. Just talk a little bit about kind of how you've seen that space evolve in the last couple of decades. I think it's really moved from kind of very specialized thing and maybe it was large companies or financial services companies did this to really just part and parcel of really what every company is doing to some degree or another. But just talk about that a little bit, how you've seen that space change.

MATTHEW ROSE
Absolutely. I mean, there's been an explosion of different types of technologies to help with different problems. The application security industry as a whole has been constantly trying to keep up with the changes in the way software is actually being developed. New languages, new platforms. When I started in the industry, mobile wasn't even a thing. You were only dealing with waterfall technology development cycles that were very plotting and slow. So you had plenty of time to do that. All of a sudden, we've added nitroglycerin to the gas tank and now we're going at 1000 miles an hour.

PAUL ROBERTS
So the other interesting development or release that we've seen just in the last month or so is this thing called the Enduring Security Framework, practice guidelines for secure, software supply chain security. This came out of a working group called the Enduring. It's got some crazy long name, but Enduring Security Framework, blah, blah, blah working group. NSA is involved in it. CISA, the Cybersecurity and Infrastructure Security Agency, and the ODNI, the Office of Director of National Intelligence, all put representatives in this group came up with these practice guidelines. I know you've had a chance to look at that and ponder it. What can you tell us? What do organizations out there, particularly those getting selling into the federal government, need to know about this thing?

MATTHEW ROSE
Yeah, I think the biggest thing to think about this, and this goes with the whole executive order type of aspect is software developed today, there's not one person in an organization that truly understands the whole application. You're talking about cloud native development, you're talking about silos of development, component development. And what that does is it really creates unknown. Like, I know my little swim lane, my branch of the main trunk of the release cycle, there's 50 other ones that are all together and just having complete transparency of what's in there and what everybody's done is very difficult to find. And I think what they're really trying to say is, hey, you have to prove to us that you're not doing anything wrong. And this may be an accident, this may not be on purpose, you may be using frameworks that aren't approved or things like that, but just a stepped approach to verify all the things within the application are copasetic in terms of risk. And I think that the document itself does a great job. It's a very lengthy document. I mean, there's a lot of information in there and I think it could be a little bit...

PAUL ROBERTS
65 pages or something like that.

MATTHEW ROSE
But yeah, I think it could be squished down a little bit to reference the old Seinfeld reference with J. Peterman, "those are a lot of words." Definitely a lot of words. So you had to get through it. But I think that what they're really just saying is just make sure that you're doing the right things at every step of the equation and don't just look at something as we said, you know, shifting left or shifting right. Shift everywhere within your ecosystem to find all the risk, the malware, the open source vulnerabilities, the inheritance of vulnerabilities within open source just by the nature of the way they're built and have the people trained appropriately. And that's something being in the industry 20 years. We've said this forever. Train the developers, train people on security, and it's still a problem 20 years later. So why is this document calling this out again? This is something that should just be part of every organization's process is to not only look for issues, remediate issues, but educate so you're future proofing yourself in the best way you can.

PAUL ROBERTS
Yeah, and I think that was one of the things that struck me as interesting and maybe a little bit different about that Enduring Security Framework guidance that came out was the focus on what people often kind of jokingly refer to as layer eight, the developers themselves. A lot of the guidance or guidelines were around using static and dynamic application security testing, doing binary scanning, two factor authentication and things like that for development accounts. But there was a lot of focus on or discussion of, first of all, insider threat. There was a lot of guidance around protecting and hardening development environments themselves. So maybe air gapping or isolating development environments. Hardening development workstations so that they're less susceptible to being compromised. And I thought that was really interesting. I don't know about you, but it's also guidance that has come in for some criticism, particularly from the sort of tech community sort of saying, look, that works if you're the NSA. It doesn't really work if you're a run of the mill software publisher or software development organization who has people working from home, people needing to do different types of things on their workstations, not just development. What do you think about that?

MATTHEW ROSE
I think it's a recipe for disaster personally. So again, you're thinking about what no, I don't pull punches ever. Hardening workstations or making it more difficult to somebody to do their job is just a recipe for disaster. I mean, talking about any security technology in the space, whether it's on the source code level, open source level, whatever that is, if it's a hindrance, if it's a barrier to success, it's going to be a problem. And if you're starting to harden developer workstations, environments, IDEs and make it much more difficult for them to get work done. Because I don't know how you work when you're coding and things like that. It's like you code for a little bit. You do some things and you think and you ponder and you maybe look at your other machine to reference something. You're like, oh, this machine is locked again. Or now I got to go relog into the VPN or I got to put in a pin or an oath token or something like that. Oh my gosh, it's going to drive people crazy and I get the security aspect of it. But again, where is the barrier? I mean, just don't let people leave and put them in a cement bunker so they can just code for twelve days straight and then you release them or something. It just seems a little bit over the top to me.

PAUL ROBERTS
I think that was the impression it created was this sort of harkening back to these days when the developer would kind of badge into the secure room and sit at the mainframe and do their work there. Obviously, that was quite secure. You needed physical access and so on. But like you said, these days you're developing your IDE might just be another tab in your browser or another application on your desktop. And you're using that for a lot of different things. That does create the possibility for security incidents. And we've seen, obviously, nation state actors, cybercriminal gangs are interested in developers and are interested in compromising development environments. So I guess what's the happy medium there for development organizations to not send their developers fleeing, but also address some of those risks?

MATTHEW ROSE
Yeah, I think that being... Understanding the process and doing a little research on how developers work and finding a way that's copacetic and acceptable to both parties. Because I don't think the heavy hand of coming down on developers and locking up their workstation and hardening environments, that's just going to create contention, but maybe have some sort of working session to identify what that would actually look like and what would be the most security to get in terms of locking down the system and the most that development would be willing to allow their day to day to be hindered. Because I think that's the only way to do it because you're always making an assumption when you don't know the response of the other party. So I think there has to be a much deeper psychological not psychological, but educational session to understand, okay, what would be the best case scenario and really mentioning that training thing. I'm going to use the example on my board here. The training example sounds great and the document goes through a lot of you have to train developers to code securely and do everything but okay, even if you give them the perfect training, so they're all capable of understanding risk. But typically developers work on different modules of code. So if they code this secure, they code this secure, they code this secure. These are three modules that are all coded securely based on that training and everything like that. But when the package is actually compiled in a real world example, these are all brought together and all of a sudden you have a data flow that basically doesn't propagate an issue or a security risk until it's all brought together. So even if you train these people very well about what they're working on, you still have risk because issues don't really arise until everything's brought together from a dataflow entry point to execution or malware being introduced. That only kind of comes to light when it's actually communicating to another piece of code that it needs to leverage itself.

PAUL ROBERTS
Right.

MATTHEW ROSE
So looking at it at a holistic picture and that's where I really feel that, yeah, great developer training is fantastic, but the automation and integration of any security technology application, security technology is important to the success. And that's something that needs to be thought of. Is it's not just about knowing. It's about things that you're not even going to know to look for, because you're never privy to all the information in one spot.

PAUL ROBERTS
Totally agree. The other issue that comes up, of course, is third party dependencies as well, which is you can have developers who create impeccable, pristine secure code, but if you're also relying on open source modules and libraries third party proprietary libraries that you don't have visibility into or you haven't bothered to assess the security of. As we've seen, that can be a big problem as well.

MATTHEW ROSE
No, absolutely. And thinking about it this way too, I think that the push for open source usage and API usage is based on a direct result of the speed of DevOps processes. Instead of releasing once a month or once every three months, ten times a day, 100 times a day. So developers just don't have enough time to reinvent the wheel for mundane tasks. So they're going to leverage the existing capabilities that are out there in open source or APIs and functionality and other applications to get their job, and then they tie it all together. So thinking about that is, you know, this is a new kind of frontier in terms of risks, because speed is king right now. Speed is king. I like to say that software development is now with aggressive CI/CD pipeline, like the social media aspect. I want it now. I want to know what you had for lunch. I want to know what you're having for dinner, where you went for your walk, where you're on vacation. The same thing is with the developers. They want to basically release that new feature because somebody wants a new feature immediately, or a bug fix immediately, and writing a new component or writing a new piece of functionality that is already out there just is a waste of time.

PAUL ROBERTS
Yeah, totally agree. So in addition to the ESF framework, about two or three weeks ago, the Office of Management and Budget (OMB) came out with a new administrative memo. This is following on the Executive Order that basically laid out for federal agencies what they need to do to comply with the Executive Order. And it really concentrated on the need for software publishers get these federal agencies to get their software suppliers to attest to the security of the software that they were licensing or services or licensing to the government. They're self attesting to this, which we could talk about that there's self attesting to this, but there were some really interesting kind of guidelines there. There was talk about potentially getting them to use SBOMs, but not a requirement. What do we think about this new guidance from Office of Management and Budget? And what impact is that going to have on the many companies that are selling software and services right now into the federal government?

MATTHEW ROSE
Yeah, I mean, I think it was a great document and described a lot of things but to me, it's kind of like the fox guarding the hen house where you're asking to self attest. Are you going to tell them that you have well, we don't really do the right things in terms of security testing of our code, and we don't know if there's malware in this package. So that's kind of interesting. And they left it like the wording was very vague in that you have to self attest and you have to self attest in these ways, or these ways. We really like you to use an SBOMs, but we're not mandating an SBOMs at this point. But there's going to be future dates that kind of talk about this. Or that if you can't do it, you can get an extension for a set period of time. So okay, it's there. And I think people are really thinking about this, and I think it's vitally important because it just kind of is setting the initial phases of this. But I think if they came across heavy handing saying, you need an SBOM today, it's going to be bad news for a lot of different people. But I think that they were really just setting the stage to push everybody to a SBOM framework that is accepted by everyone. And I think this is their way to be a little passive aggressive in it. Not heavy handed, but saying self attest however you want, but SBOMs is what we really want. And if you want to make it easy on yourself, give us an SBOM format that we like.

PAUL ROBERTS
Well, we know that the Department of Defense sort of flirted with the idea of third party or independent attestation with their CMMC guidelines. And that kind of blew up a little bit because there weren't enough organizations to do the assessments. And there was kind of a panic amongst the software vendors of like, how are we going to get this done? And particularly for the smaller vendors, how are we going to be able to do this in time to be able to offer our stop? So my guess is they kind of learned a lesson from that and said, let's not start there. Let's start with this self attestation, which is obviously easier to get done. Yeah, you're right. The SBOM thing and SBOM, obviously, Software Bill of Materials, are those sort of a silver bullet in this software attestation thing? Are they going to have the impact that people think that they're going to have in terms of, again, laying out that list of ingredients or list of components that organizations can then use to kind of validate and then assess?

MATTHEW ROSE
Yeah, I think I've always been a proponent of SBOMs as being very important. I mean, the manufacturing world has used the same concept for many, many years. Everything that's going into the vehicle or the microwave or whatever that is. But the thing that I think we've kind of missed is SBOMs, again, don't keep up with the speed of DevOps. So it's like you're constantly changing the code. So the files are changing, the open source packages are changing. So SBOMs have to be an ongoing thing. That's part of the CI/CD process. I have an SBOM for release 1.0, and all of a sudden you're on 1.123456789. I'm pretty much guaranteeing you that that SBOM is different from 1.1 to 1.2. There's something different, new files, new things in there. And the other thing that people immediately jump on, the software composition analysis, like they provide me with SBOMs. Well, that's just for the open source. I mean, you need a complete SBOM of everything. Every file, is their malware, every open source package, all the inheritance of the open source packages, so on and so forth. So a lot of times we do have to reassess what an SBOM truly is, and the format really has to be all encompassing of everything in that application, not just a subset like the open source packages. That's just my opinion, because I think that it's a great concept, but it has to keep up with the speed of DevOps because the code is constantly changing, so the bill of materials is constantly going to change.

PAUL ROBERTS
Right. So, as I said, there's been a lot of this guidance that's come out. The Enduring Security Framework, the Executive Order, the various NIST standards and guidelines that have come out in the past couple of years. There's some legislation pending in Congress that would address specifically open source security. So there's a lot floating out there, I think, for organizations that are selling into the government. This is sort of confusing to try and line all these things up. What would your recommendations that be to software organizations, development organizations out there, what to pay attention to at this stage, and what to prioritize as they're looking to get into conformance and get into compliance with these guidelines?

MATTHEW ROSE
Are you a Game of Thrones fan? Do you watch Game of Thrones?

PAUL ROBERTS
Of course, yes.

MATTHEW ROSE
So my comment to the software companies is winter is coming, okay? They're saying these things, but it's already very complicated and lengthy to do business with the government in terms of getting FedRAMP, getting all the things lined up to actually sell to the government, working with the integrators, all the I's dotted, T's cross, so on and so forth. If they're starting down the path of talking about SBOMs, winter is coming, or SBOMs are coming, I would say start today, if not yesterday, to start putting in the processes or the kind of capabilities to create accurate SBOMs in an ongoing kind of way for your software development process. And that could be anybody. I mean, this doesn't have to be ISVs and software development houses. I think anybody that wants to sell a service or anything is going to have to provide this information and get ahead of the curve, because it's already very cumbersome to do business with the government. This is coming and you better address it now than later.

PAUL ROBERTS
And I think the idea of the federal government or the Biden Administration anyway, is that this will become a standard that will extend beyond the government, right. That the private sector will in essence start to gravitate towards these types of guarantees and attestations as well in private contracts. Do you see that happening?

MATTHEW ROSE
Yeah, I absolutely do. I mean, it doesn't if it's not at the federal government, it could be at the state government level too. I mean, if there's an issue and somebody gives you another issue, comes up with a supply chain issue where malware is inserted accidentally, on purpose, nation state, whatever that is, you may be having federal issues and state level issues and all sorts of regulatory issues against that. So I think it's not that difficult to process just to be able to use automation, integration of technologies, whether that's a combination of multiple technologies or a complete software supply chain technology, to find everything, report everything and do that in a continuous way. Because that's the biggest thing is I think a lot of people are freaking out because the speed of DevOps, I keep harping on this, is creating kind of everyone trying to leave through one exit after a football game or a sporting event. You're all trying to get through that and you're all trying to figure out how to get out. And there's a big backlog. You need to actually kind of open that up and think about, okay, how do I address this easily with automation rather than manual processes or spreadsheets or questionnaires to basically put something together which again has the big elephant in the room, which is human error. I mean, what if you're putting these things together and SBOMs together in a manual way and you miss something not going to be good. So you need technology to help you.

PAUL ROBERTS
There in the end of the day. We've had this conversation before with PCI and other regulations. There's always this sort of managing for compliance and managing for security. Do you see these guidelines and mandates in the long term actually leading to better security outcomes or just more hoops and stuff for just more compliance work and talk, but no real big change in terms of security outcomes?

MATTHEW ROSE
As long as it's handled correctly. So an example I'll give here has been doing this a long time, especially around the SAST space. And the two biggest things when I was in the industry in 2005 was I need to find sequel injection and cross site scripting. And this is on the SAS side of the house. 2022. You talk to people. What are you worried about in your application? SQL injection and cross site scripting. What has happened? Why is this still? This is ongoing for years and years and years. It's like we're just constantly playing a game of chicken. So I think maybe a little heavy handed. But having something that really gives you all the information lets you make assumptions on risk in a consistent and repeatable fashion is important. And I think instead of people trying to create their own homegrown security standards, policies, posture, whatever that is, but having that mandated at a governmental level because again, the world is not going to exist without software. Every company is a technology company to do today. It doesn't matter if it's a bank or a retail or even a bakery, they're all using technology to get their job done. If we don't standardize on this like we standardize on other things in the world, there's going to be a lot of opportunity, let's just say, for nefarious individuals.

PAUL ROBERTS
So, final question, Matt, like you said, winter is coming for organizations, whether they're selling to federal government agencies, executive branch agencies or not. What would you tell them at this point to be focusing on and prioritizing in terms of software and software supply chain security?

MATTHEW ROSE
I think it goes down to you don't know what you don't know and finding ways to answer the questions you don't even know how to ask. And I think that's with looking at the complete package prior to release because again, you're talking about very complex development processes that are siloed branches of a trunk of a major process. And the final check has to be a final exam. Even if it's a banking app that a commercial bank is releasing or it's an ISP releasing a piece of commercial software, there has to be a check that verifies this is good to go, or there are issues in it because we don't want this to happen again. In terms of supply chain breaches, insertions of code, SolarWinds type of issues, I think that there has to be that final check and that final check has to be industry wide standard because a lot of times people talk about their application security program. Who are the champions within the program, who's doing the best things? They're all different. I've seen it like some of the biggest companies in the world don't do a great job at it and some smaller companies do a great job at it. Why can't there just be a standard that basically the one check you have to do is to make sure that there is a viable package you're releasing, whether it's internally or externally, for malware, for vulnerabilities that are addressable?

PAUL ROBERTS
Right. This is what we do with vehicle airbags and things like that, right? It's not like Toyota and Ford all get to have different standards for how, you know, their airbags, you know, protect passengers, right? There are standards and they all have to have to meet those standards, right. Hey, Matt Rose, it's been a great conversation. Before we go, is there anything I didn't ask you that you wanted to say?

MATTHEW ROSE
I think the biggest thing is just supply chain security needs to be more mainstream and not thought of something that is more for the SOC or analysts that it's just as important as all the AppSec tooling that you're spending tons of money on resources on from, as I mentioned, the pillars of application security SAST, Dash, Is, Rasp, SCA all these ones, these acronyms star AST Technologies... Supply chain is just as important. It's, I think, maybe even more important than just finding vulnerabilities at the code level or the running application. Because again, it's not just the potential, it is malware potentially and it can be propagated very quickly in a way that you can't control.

PAUL ROBERTS
We're seeing that all the time these days, right? I mean those attacks on open source repositories and CI/CD pipelines seem to be becoming more and more common.

MATTHEW ROSE
Absolutely.

PAUL ROBERTS
Matt, thanks so much for coming in and talking to us on ConversingLabs podcast. I'm sure we're going to have you back soon.

MATTHEW ROSE
Thank you, Paul, I appreciate being here.

PAUL ROBERTS
My pleasure. Great to have you.