Can Frameworks Stop Supply Chain Attacks?

In this episode of ConversingLabs Podcast, host Carolynn van Arsdale welcomes North Carolina State University Professor Laurie Williams and Ph.D. student Sivana Hamer to discuss their team’s research on the effectiveness of software supply chain security (SSCS) frameworks. Their study, “Closing the Chain,” (PDF) found that software products would still be vulnerable to attacks like SolarWinds, Log4j and XZ Utils – even if they fully enforced 10 well-known SSCS frameworks published by government, industry, academia and open source.

EPISODE TRANSCRIPT

Carolynn van Arsdale: [00:00:00] Hello everyone. Welcome back to another episode of ConversingLabs. This is a podcast from the team here at ReversingLabs where we dig into the latest developments in areas like malware analysis, threat hunting, threat intelligence, and also software supply chain security. I'm Carolynn van Arsdale. I'm your host for today's episode.

I manage all kinds of content here at ReversingLabs, and today I'm thrilled to welcome [00:01:00] on two guests. First, we have Professor Laurie Williams and also Ph.D. student, Sivana Hamer. Both from North Carolina State University both from the Computer Science Department. Professor Williams is a Goodnight Distinguished university professor in security sciences who also co-directs the NC State Secure Computing Institute. With more than 260 publications, Professor Williams has published research on various software topics such as software security, agile software development, and software reliability. Her advisee, Sivana, is a third year computer science Ph.D. student, currently researching the state of software supply chain security. Recent research efforts of hers include how software dependency changes have been tracked, as well as vulnerabilities compared between ChatGPT and Stack Overflow.

Very interesting stuff. So Professor Williams and Sivana, welcome to ConversingLabs. It's wonderful to have you [00:02:00] both on today.

Professor Laurie Williams: Yeah, it's great to be here. Thank you for the opportunity. 

Sivana Hamer: Yeah, thank you. 

Carolynn van Arsdale: Yes, of course. So we always do this with new guests on the show. We ask you to give your elevator pitch of who you are, how you're involved in cybersecurity. And today we're talking about software supply chain security. So I'd love to hear more about that.

So Professor Williams, let's go ahead and start with you please. Would love to hear more about your academic career, how you settled on security sciences, and what you find so interesting about software supply chain security.

Professor Laurie Williams: Sure. Thanks. So as you've mentioned, I'm a professor in the computer science department at NC State where I've been for 25 years, I can't believe I can say that. But really I didn't take a traditional path to academia. I started out at IBM and worked as an industrial engineer at IBM and got an MBA during that time.

So it just shows people's paths can be very sorted. Eventually I got a Ph.D. in computer science after [00:03:00] I had three babies, so not the normal path. And because I really wanted to be an academic, so I felt it at that time. And so I took that step. And as an academic I really did quickly zone in on software security.

And the reason is because I truly have a passion for helping save the world from cyber criminals. It's absolutely a passion that I have. And teaching students to do the same and mentoring Ph.D. students so they can then go out into the world and do the same brings a lot of passion to me.

So that's how I got where I am. 

Carolynn van Arsdale: Awesome. Thank you so much for sharing. I really admire, and I think I hear this story a lot when speaking with folks in academia and professors, hearing how they moved from industry into academia, bringing those real world lessons into your work and building that passion for securing the world. Very cool. Thank you for sharing.

So Sivana, let's turn to you. So you've been focused on the state of software supply chain security. I'd love to hear more about your academic [00:04:00] journey and your research interests and where it's taken you. 

Sivana Hamer: Yeah, so I guess my journey's a bit shorter at this point still, but basically I guess I started doing research during my undergrad and master's in Costa Rica actually, which is pretty cool. I was actually very interested in a field that we have in research called empirical software engineering. So it's basically using real data to inform what's happening in these different software processes because software's like everywhere and it's super important.

And I really also like measurement, which is a very specific topic within empirical software engineering. So it's like, how can you measure stuff? And I ended up working with Laurie because she's really well known in the field of empirical software engineering. And I've now become involved in software security and software supply chain because again, it's something that people really care about, and I've really grown to love it. It's really cool to like work on problems that people care about and every day, they're like facing these challenges and it's really cool, I guess as a student to be able to like, try to help them, to make a world a better place [00:05:00] overall. 

Carolynn van Arsdale: Yeah. Very well said. I think too, and I'll definitely ask you both about this regarding software supply chain security, because of course cybersecurity has been around for 30 plus years. I feel like software supply chain security, on the other hand, obviously it's been around, software's been around, but talking about it in general discourse, seeing it in the news, it's newer, right?

And especially as you pointed out, academic research in this area is extremely important, in addition to other cybersecurity topics. Can you both just share for me why academic research is so essential for software supply chain security, especially where we are now with the current threat landscape?

Professor Laurie Williams: Sure. I guess I can go first. I won't always go first, Sivana. I became interested in software supply chain security, really because a student of mine was interested in software supply chain security, and as you say, it's a relatively new topic. And so he, and I'll give a shout out, Iam Tiaz, who works at Google now and in [00:06:00] 2019, he really became interested in this. 80% of all software is open source. And so he started to work on looking at the security release of open source packages, a comparison of SCA tools before SCA tools were cool. Code review coverage, contributor reputation metrics.

He had been involved with all of these things before SolarWinds, before everyone's focused on software supply chain security. And he really did bring it to focus with me and we were really getting a lot of momentum when SolarWinds happened in late 2020. The executive order happened in 2021.

And we were well on our way. And we did get a large National Science Foundation grant, which I can talk more about later. But as part of that, we run three software supply chain summits with industry and government. And they do say that, so the practitioners are saying that, they're just trying to survive.

They're trying to put out products. They're trying [00:07:00] to do whatever they can do to not be the victim of an attack and that there is a need for fundamental research to find solutions to software supply chain security. And I agree and that's really what motivates me to be involved with academic research and software supply chain security.

Sivana Hamer: I think adding a bit more to that, right? I think that like us as academics have a very different like skillset than like industry and they're both valuable, like definitely in government, right? I think everybody in this field like benefits from like different perspectives and different ways that we solve problems, right?

Because again, like as academics, we probably, we don't have as much power as like to influence, what's happening or like to adopt stuff. But we do have some skill sets of like, how can we research stuff using a lot of methods and making it like very scientific and academically rigorous that can help policy makers or software organizations have data that backs it up. So I think it's like a field that we also all benefit from helping each other a lot, which has been like super exciting that everyone, at least that I've usually interacted with is like [00:08:00] super nice and like super inviting.

Carolynn van Arsdale: Yeah, I will say it's definitely a team sport, right? And I'm sure Professor Williams, as you said you hear from folks in industry, folks in government and academics play a really big part in solving the puzzle, the very big puzzle of software supply chain security.

There's so much work to be done. So let's get into why you all were invited here today. It was because of an amazing report that you all put out with a team of other researchers at NC State. It's called "Closing the Chain: How to reduce your risk of being SolarWinds, Log4j or XZ Utils." So this was a really cool report.

I really enjoyed reading it. Very essential. A great example of why academia is essential to software supply chain security right now in terms of fitting those puzzle pieces together as to how we can solve the problem. So I won't speak for you. Please take it away and tell us more about this report, the premise of it and what led your team to ask these questions in the first place? 

Professor Laurie Williams: I [00:09:00] mean, I was probably the initial motivation behind it all. But then Sivana took it over and excelled. And so the initial motivation was I did a sabbatical in 2022 and 2023 with Synopsis, it's now Black Duck, in that sabbatical created a framework called the Proactive Software Supply Chain Risk Management Framework. That was intentionally the union of 10 other frameworks that industry knows about. And it brings together all the things in theory people are supposed to do to reduce supply chain risk.

And that came up with 73 tasks. That's a lot of things to do and organizations can't do them all. And so there have been other frameworks, maturity models that have different ways of prioritizing. The intention is really to prioritize based on risk reduction, how can we adopt the tasks that are most likely to mitigate what the attackers are doing.

And so with that global [00:10:00] mind, Sivana really took it to heart and did a tremendous study and that's where I'll have her took over. 

Sivana Hamer: So I guess on my end, when I started my Ph.D., I was doing several works. So then we were talking about more what I would say my main dissertation works are gonna be, which again is related to that part of what's the state of the supply chain and how can we measure it.

And from my point of view, I was actually like asking myself how can I start measuring stuff if I don't even understand what attacks are happening? Or like how we actually work in detail. So I was like I should actually understand these like attacks that are happening generally, in a more detailed way. I guess in our case we did like attack techniques specifically that the attackers use with CTI reports, but specifically just generally understanding that because it's really hard to like secure something. You have no idea what's happening. So I talked with this with Laurie, right? I think I called it like dissecting the attacks at one point because we're like opening them up and seeing them. And then I looked more into it and it's like when airplanes crash, you're supposed to actually retrospectively look up why did this happen? Like why is this [00:11:00] catastrophe happening? Looking back into it, and Laurie told me when she did her sabbatical we had some collaborators from industry. Shout out to Chris and Rob, that we were also talking a little bit about that, like maybe you could like actually look into like big attacks, specifically in our case, we looked at in the end SolarWinds, Log4j, XZ Utils, which are like a trifecta of very popular attacks. So they had talked a little bit about that and how we could maybe use it with PSSCRM, right, being the encapsulations of all these different frameworks to understand, hey are these frameworks actually mitigating these attacks from a practitioner point of view? So we, I guess in this case, we benefited a lot from having this like very close industry collaboration. Again, they're great collaborators, amazing people too, but it was very beneficial to have also their point of view and their insights and experiences that helped the study. And that's where it started overall. 

Professor Laurie Williams: Yeah. Chris Madden and Robert Hines are from Yahoo, for the shout out to them. 

Carolynn van Arsdale: We love the shout outs here. Thank you. Yeah. And Sivana, I love how you explained the importance of [00:12:00] dissecting attacks. I think at ReversingLabs, we've talked a lot about the anatomy of a software supply chain attack. I definitely wanna get into these three major incidents, which our audience, they are into cybersecurity, but just to set everybody on the same stage here, these are the three most popular, infamous, well known, I think consequential or could have been more consequential, software supply chain attacks.

And each is pretty distinct from one another. As you said, looking into the anatomy of each is important. I guess in your own words, we don't have to go too deep in the weeds, but can you talk about each,

Sivana Hamer: There's a lot weeds.

Carolynn van Arsdale: -Each of these. There are a lot of weeds. Can you talk about, each of these incidents, SolarWinds, Log4j, XZ Utils and how they differ from one another?

Sivana Hamer: I'll try to give a detailed, broad overview, but again, this could be like a topic of like hours definitely. And we read so many reports. SolarWinds is at a more high level, I would say it's an attack that mostly focused on the build infrastructure. [00:13:00] How people got into their environment is still something that at least is still not publicly disclosed.

And I don't know the details. I don't have clearance. people have not told me specifically, but generally what attackers actually did is they introduced a malware called Sunspot within the SolarWinds environment. And what Sunspot would actually do is they would detect if actually they were building software with Visual Studio Code or generally.

So when it noticed it was actually building the software, they introduced another malware that it's a bit more well known, Sunburst. In Sunburst, what they would actually check that, hey can we actually introduce Sunburst or not? Is it safe or not? They had a lot of things they were checking.

They also tested it multiple times. It didn't just put Sunburst in the first time. They were like, Hey, if we change this and we tamper with the build, would someone notices? So this process carried out through several months. And when Sunburst would be put into the software right away, then be built by SolarWinds, it would be actually be signed as if everything was normal.

And then dependent systems, which included US government, big companies, among others, then installed Sunburst. Again, it's a [00:14:00] Trojan in their systems. And Sunburst specifically, you could send commands to it. So the attackers could be like, Hey could you download this? That'd be really nice. Could you send me your data? That would also be pretty nice. So they would also install additional payloads in these systems. Some of them were also specifically tailored to specific systems. They would do recon among other stuff. So I guess that's SolarWinds at a very more general view. Log4j is a bit different in the sense that this is again, Log4j was an open source package by Apache. Specifically the feature that was vulnerable was actually developed in 2013, so it was like a long time ago, and it was a feature that was added. Again, it was just there. Particularly a JNDI lookup feature, and again, this is a contributor, added the feature, the team checked it, everything seemed pretty normal.

And then in 2021, actually a developer and I probably botching the name of the company, Alibaba. But they actually noticed that, hey there's actually vulnerability in this feature. So they told Apache, they actually disclosed pretty nicely and they're [00:15:00] like, Hey just a heads up, look into this, right?

And then they're like, oh, we have to fix this. So they were actually trying to fix it. And what actually ended up happening is they were actually discussing in WeChat, I believe it's a forum, in some online forum, some POCs about Log4j, which again, Log4j is like pretty insidious because everybody uses Log4j generally.

So what ended up happening is they had to publicly disclose the vulnerability before they had a full patch available or full fix available. So what ended up happening is they released a fix. Attackers actually started exploiting it when the CVE or the vulnerability was publicly disclosed.

At the same time, they noticed that the fix was not very complete and there were other issues and some fixes added other things that attackers could leverage and so on so forth. So it became like a sort of a scramble. People didn't know if they were actually patching or not fully. It's still an issue even nowadays, like at Sonotype, which is a very big company, they also have these reports and there's still people who are using the vulnerable version of Log4j, like still this point, I don't know, so many years later.

Carolynn van Arsdale: But there's also been evidence [00:16:00] that nation-state backed actors- 

Sivana Hamer: Oh yeah.

Carolynn van Arsdale: Have been exploiting this vulnerability still, years later.

Sivana Hamer: And all of them, I guess all of these are sorta nation-states, I guess in XZ Utils it's not confirmed, but it's pretty... suspected. 

Carolynn van Arsdale: Yeah, they were quite sophisticated, which I won't steal your thunder. 

Sivana Hamer: And I guess the last one XZ Utils, a bit different than the other ones in some other senses, right?

So again, this one is an open source one, but the difference, Log4j was accidentally vulnerable really, wasn't like the developer's fault that it was vulnerable, that they introduced a feature. Particularly in XZ Utils, there was actually these sock puppet accounts that were created throughout several years, and they actually became maintainers of XZ Utils, which is a package that if you go to the dependency tree is super important for a lot of like packages. Particularly they leverage that it was used for SSHD, which is for some Linux distributions, it actually used that. Anyways, so they leveraged that, Hey, we have this dependency that other people depend upon. And they noticed that actually XZ Utils only had [00:17:00] one main developer. So they were like, Hey, maybe we could leverage this. So they ended up in pressuring campaigns, actually pressuring the main developer at one point to give access to what ended up being a sock puppet account.

But then after, throughout several years, contributed to the software. And there's this well-known document that the original maintainer did, like after the attack, where they actually went through all the commits or a lot of them, and most of them were actually legit.

Carolynn van Arsdale: Yes.

Sivana Hamer: They actually contributed to the software. It wasn't like, oh, I only did a the malware and that was it, and bye - they actually contributed throughout multiple years, like a lot. But at one point they started to implement, in that case again, the malware in XZ Utils actually had some bugs, so it would crash sometimes.

And after that, as developer at Microsoft Andres Freund, I probably mispronouncing his name, anyways, he actually noticed it publicly disclosed that, hey, this vulnerability is happening. Or that someone is actually leveraging this, this seems to be malicious too at one point, because they were sort of like, Hey, this is weird, the binary doesn't match the one that's [00:18:00] there. They also leverage that XZ Utils again, used binaries in their tests, so it would like decompress these binaries, do a lot of crazy stuff like you would add and delete and move characters. It was a lot of stuff.

But generally these attacks are like pretty complex, all of them, and pretty well known because they affected, and I guess in the case of XZ Utils might have affected, right, because it was disclosed before it was leveraged, a lot of organizations worldwide. 

Carolynn van Arsdale: Yeah. And I think that, in choosing these three incidents, I think they were well chosen, because they all focus on different areas of the supply chain. Mm-hmm. Mm-hmm. All connected to each other, of course. But, SolarWinds being more closed source, Log4j as you said it wasn't necessarily an intentional malicious effort at first, but was exploited as a software vulnerability. And then the last one, XZ Utils, I think really speaks to the current threat landscape now with open source software being heavily targeted. In addition to, which I definitely wanna get into later, is how a lot of the folks that uphold open source are [00:19:00] volunteers, these maintainers, and in that case, with XZ Utils, that was totally taken advantage of. Great job with that very succinct overview, but very well delivered.

Sivana Hamer: Thank you.

Carolynn van Arsdale: Professor Williams, anything you wanna add on that?

Professor Laurie Williams: She did a great job. Great job.

Carolynn van Arsdale: Yeah. So Professor Williams, I know that you spoke a little bit about the frameworks that were involved in this study. There were 10 frameworks, including one that you authored. Can you just tell us a bit more about that breakdown in terms of the frameworks? Some come from government, some come from open source, others from industry, academia of course. Just give us an overview, please of those 10 and how they differ, how they compare, what they mean for companies today trying to secure their supply chains.

Professor Laurie Williams: Yeah, sure. As I mentioned before, PSSCRM is the union of 10 frameworks. Not intending to be the 11th framework, but to be the union by definition. And as you mentioned, those frameworks come from the US government, so some NIST [00:20:00] frameworks. Cloud Native has a framework on supply chain security and SLSA and OpenSSF have frameworks.

The Executive Order 14028 that came out in April, 2021, had a self-attestation framework. And so in coming up with the union there was an explicit choice to bring together frameworks that dealt with software supply chain security, not general security. So things like NIST-853 or ISO 2700-1 were not involved because they're very general, so it was really focused on software supply chain security.

And with that, you would think why 10, how many different looks are there at software supply chain security? And they were chosen not only because of their relation to software supply chain security, but each actually has a different focus. So for example, NIST 800-161, it's about supply chain security, hardware and software [00:21:00] supply chain security.

But it especially has tasks related to governance, so vendor management and policy that the others didn't have. NIST SSDF, that gets lots of publicity-

Carolynn van Arsdale: Secure Software Development Framework, to break down the acronym.

Professor Laurie Williams: Thank you. Yes. 

Carolynn van Arsdale: Yeah. 

Professor Laurie Williams: And the D in there is development, and so the key there is that in our different categories or different practices in PSSCRM, they especially populated the product grouping, the product practice. That's the development, that's the software engineering practice. And so SSDF especially was influential there. And then the other two practices in PSSCRM are environment, which are IT-related and deployment.

Which, is the deployment, but PCERT, things like that. And the Cloud Native framework and the SALSA framework specialize in those areas. And so we do have a grid that talks about, where did all [00:22:00] of the 73 come from, which are overlap, which of the frameworks- like a task only appeared in one framework and which are the most influential?

So I would say they really were synergistically brought together into PSSCRM. And then from that framework, Sivana really was able to structure her work. 

Carolynn van Arsdale: Yeah. Thank you for that overview. And I think that you also point out a really important aspect here, which is software supply chain security isn't meant to be solved by one team within an organization.

Professor Laurie Williams: Yes.

Carolynn van Arsdale: You point out that it spans from the governance, risk, compliance piece, all the way going to the left, to developers which, that's a whole other podcast topic in and itself of how developers can get more onto it, which we have talked about before on the podcast- to everything in between, right?

So I think rightfully so, important to have a selection of frameworks that focus on those different areas of the supply chain.

Professor Laurie Williams: And bring 'em together.

Carolynn van Arsdale: Yeah, exactly. So let's get into the [00:23:00] fun part, which is the results of your study, which were really fascinating. I definitely don't wanna steal the thunder again here, but overall, after, looking at over a hundred cyber threat intelligence, CTI reports, of the three attacks that we talked about, SolarWinds, Log4j, XZ Utils, comparing those with Professor Williams, as you said, the 73 tasks throughout the frameworks, what did you find? How were these frameworks matching up with the different mitigation techniques organizations need to be taking? 

Sivana Hamer: So I guess adding to what we found, I guess general findings, because there's a lot of things we found. But on the task side mostly, first of all, we had, like, when we did these mappings, we actually generated something we called like a starter kit. So we rank different tasks in these different frameworks, right?

And we're like, okay, based on these attacks, you would mitigate these amount of attack techniques. Oh yeah, adding to that, I think we forgot to mention, we mapped all these CTI reports to MITRE attack,[00:24:00] which is a pretty well known framework for TTPs. Particularly in our case, the attack techniques describe how attackers did stuff, so we mapped those attack techniques to PSSCRM, which included these like software supply chain attacks. And from the reports we had, okay, what are all these attack techniques? What are all these tests? And we created a mapping between them. And based on that, then we had okay, let's rank the tasks within PSSCRM, right?

And find which are the most prominent ones that you're supposed to do if you wanted to defend your supply chain. And for the top three, we found it was actually access control, system monitoring and boundary protection. Actually, if you look at the top 10, because again, we did a top 10, like an OWASP 10 type of thing, but we actually found they're very more like software security related rather than software supply chain related, which initially we were sort of like, huh, that's interesting. Like I expected I was gonna get- 

Carolynn van Arsdale: But I am curious, could you just define quickly for our audience- maybe this is a can of worms that is a lot to open up, but software security in comparison to software supply chain security?

Sivana Hamer: I guess yeah, so I guess [00:25:00] software supply chain security, right, obviously is part of software security, right? I guess software security, I would see it as more broad.

I think Laurie would probably have a better definition than me, and I feel like software supply chain security is a specific area that we focus on, like specific types of attacks that prey upon like you depending upon the software. That's what I would say specifically. But having a good software security posture is we found, I guess related to this is having a good software supply chain security posture.

Professor Laurie Williams: Yeah. And I guess I could add that in that software security, so that's what I've been teaching for years is the intersection of software engineering and security. So how do you build a secure product? And there are a lot of things we've been doing for a long time, scanning for vulnerabilities or doing security design or things like that, that have been being done for a long time. And then along comes these new types of attacks and new things that have to be done specifically for supply chain.

Carolynn van Arsdale: Understood. [00:26:00] Yeah. I think too that a lot of what you just said, Professor Williams is very reminiscent of Secure by Design, which the US Cybersecurity and Infrastructure Security Agency, I believe they put out two years ago, maybe three years ago, my concept of time is not great. But really applies to those principles. So thanks for breaking that down. I don't know if you have any thoughts, Professor Williams on maybe when that came out, how it impacted the industry?

Professor Laurie Williams: Yeah, I'll isolate some specific tasks. Things that are new, if you will can be updating your components when they're vulnerable or managing vulnerable components, and choosing your components. So not just taking anything off of anywhere, but actually making explicit choices. So those are just, three examples. There are others that were not on the scene before that are on the scene now. 

Carolynn van Arsdale: Thank you. Sivana, I'll let you continue.

Sivana Hamer: So I guess we had these top 10 again, mostly, or like general good software security things. It did [00:27:00] include some extra stuff that was more software supply chain security. Hey, you should update your dependencies if there's actually a vulnerable version, among others. But again, they're mostly focused on like more broad software security. Again, initially I was a bit taken aback when we ran our results and I was like, huh, that's a bit weird to me.

But then it was like it's hard to secure your sup- well, we hypothesize again why we got this. It might be hard to secure your supply chain if you're not doing like access control or if you have a good access control posture, right? So we were like, maybe it's done, maybe all these newer tasks that are more on our radar now, they might be very forward-looking.

So again, how we like created our mapping, they had to be agreed upon tasks. So it might be that, again, it was somewhat interesting. I would say that's main finding one, the starter kit of what people should do. Also another interesting thing is like in our starter kit too, we had tasks from different parts of the, like as Laurie explained, there's different groups in PSSCRM, right?

So we had tasks from governance, we had tasks from product, we had tasks from the environment. Then we had tasks from like deployment that were in the top 10 really of what you should [00:28:00] do. So again, I think it's a sort of known thing in security that you have to have a defense in depth, right?

You can't like focus only on your product or only on the environment or only on deployment or only on governance. You have, it's a thing that goes across everybody, so that was also it. Then we also have like different metrics we created for frameworks. So we have like coverage and completeness and all that.

And that was basically if they mapped to stuff that was within the framework that was referenced in PSSCRM. General findings, again, this is somewhat expected that no framework was gonna cover all these attacks. As Laurie mentioned before, every framework has a different view of the slice of the supply chain, right?

So again, I guess we don't really expect any framework to fully encapsulate everything. But I guess that was somewhat interesting that we had that too. I guess adding to that, I don't know, you can interrupt me whenever.

Carolynn van Arsdale: No.

Sivana Hamer: Because there's a lot of cool findings. The frameworks that did better off based on our metrics again was actually the US government frameworks [00:29:00] that we had generally. So NIST SSDF, NIST 800-161, even the self-attestation form, which I was very surprised the form because again, it was, what are the basic software security tasks that people should do?

And I guess the final interesting, more main finding that we had generally, we also, when we were analyzing these different CTI reports, we were mapping manually. Me, I guess shout out to Jacob Bowen. We were like manually mapping these different reports and when we were mapping them too, we were trying to map them if they had a task that they mentioned and we mapped those tasks to PSSCRM, if that makes sense.

Basically based on that, we had tasks that we found in these different CTI reports that were not in PSSCRM. So we found the tasks that we call, like gap tasks that we were not able to map at all. We discussed these tasks with our group multiple times.

Some tasks just became a little bit bigger. So for example, on dependency update something that was a task that people talked a lot about coming from XZ Utils was flighting, right, of waiting a bit when you're updating your dependency. So there [00:30:00] was a dependency update task, but we increased that.

But there's some other stuff that was not mentioned in any frameworks, like supporting open source software was not mentioned really explicitly in any of these frameworks. Or also having like environment-

Carolynn van Arsdale: That's a big one.

Sivana Hamer: I know. Environmental scanning tools, so generally having tools that are scanning like your environment, like similar to how you have like your SAST and DAST for your product, but having more specific tools for your environment and then building these response partnerships that are super valuable before an attack happens. Because when an attack happens, you wanna have a group or network that you can collaborate and build upon rather than just being like alone and sad. 

Carolynn van Arsdale: Alone and sad is a good way to describe it. After the fact.

Sivana Hamer: After an attack.

Carolynn van Arsdale: Yeah. Thank you for that breakdown. And really the big takeaway here is, just at a high level, at the end of the day, if you look at these 10 frameworks, and as you said, each framework wasn't meant to solve every single problem.

Sivana Hamer: Yeah.

Carolynn van Arsdale: Every single risk that the software supply chain [00:31:00] presents. But overall if organizations apply all 73 tasks in a perfect world, all 73 tasks of these 10 frameworks, at the end of the day, they would still be vulnerable to the kinds of software supply chain attacks: SolarWinds, Log4j, XU Utils.

So that's a pretty big finding to say. I am curious. And Professor Williams would love for you to jump in here too, since you're pretty in depth with these frameworks. Why do you think these big areas of open source software scanning and environmental scanning tools, why were they left out? 

Professor Laurie Williams: Yeah, all I could say is they tried their best, and we fully intend to continue the work that Sivana was doing. And we're into the PSSCRM too. And when we get that webpage out, we'll list the orphans. And we'll list those three and we will encourage the framework authors to include those orphans.

I'll call 'em, I don't know if that's the right word, in their frameworks, because we don't wanna be the 11th framework or the 16th [00:32:00] framework, we want to be the union. But we wanna highlight when a framework didn't do something. And we have the traceability from what Sivana did.

This is why we think this is a problem. Look at this specific report on XZ (Utils) or on SolarWinds and what they did and how this could have helped. And so it is just a continuing evolution and we see our role in being able to help organizations prioritize their task adoption 'cause they can't do them all. But also to help the framework authors to see maybe with some blind spots they have. 

Carolynn van Arsdale: Exactly. Yeah. A way for them to move forward, 100%. I'm curious, have you had feedback from industry partners that you all are connected with so far regarding the research? 

Professor Laurie Williams: Yes. I don't know why Sivana isn't jumping in- 

Sivana Hamer: I'm shy. I'm shy, yeah.

Professor Laurie Williams: So we did some popular press is what I'll call it, reaction. And I've taken Sivana out on the road show for [00:33:00] work and whenever we possibly can to industry and government organizations. And so have gotten some great feedback and, without saying the agency, there was one US government agency that said this is great, we were trying to figure out how to prioritize task adoption and mitigating the things attackers doing now is a great way to do it. We're gonna do these 10. And so that's exactly the type of reaction that we wanted to have.

Sivana Hamer: I guess overall, very positive. Like nerve wracking when you do studies like these and talking to people who actually are pretty well known and know about these attacks and overall I think we'd had a pretty positive reception of the study and hopefully people can contribute more. And again we're planning to continuing to do this and gather more data, like how can we actually stop these attacks or other security outcomes generally and how could we automate what we did? Because again, a lot of what we did was very manually intensive. It was very hard process and hopefully what we would like to do is over time continue to do this and like ingest information, what's happening, hopefully as close as [00:34:00] real time as possible. So those are exciting, research directions that we are looking at. 

Professor Laurie Williams: Yeah, we have, as Sivana mentioned, particularly the digesting CTI and mapping it to the MITRE ATT&CK was very labor intensive and there's a number of people, a number of papers and studies with automating that process, CTI ingestion to MITRE ATT&CK mapping, and none of them perform well. And so that's one of the things that we're really working on right now and we have some promising approach that hopefully we'll be publishing in February of next year that can automate that process.

And then we should be able to take some of the newest attacks and the CTI from those and to find out what are the attacker techniques and map them to mitigations and perform this prioritization. 

Carolynn van Arsdale: Nice. Yeah, and I know too that I believe MITRE ATT&CK just put out a new update regarding its framework. So very important too to jump on that in terms of [00:35:00] automation. Automation really is going to help cybersecurity in a lot of ways. So well said. 

Professor Laurie Williams: Imagine that we're using an LLM. 

Carolynn van Arsdale: Yep. So should we have a whole other podcast now on what your thoughts are on AI and cybersecurity? Do you guys have another hour? Maybe three? So I think also too you all brought up a good point and it ties in well to my next question. So there have been, of course, other supply chain attacks that are quite notable. One I'll bring up is the attack on the voiceover internet protocol provider 3CX, which correct me if I'm wrong probably similar nature to the SolarWinds incident of 2020.

And then looking to this year specifically, open source software has been hit tremendously. For instance, the Shai-hulud worm that took over npm not too long ago, that was a really big incident. I'm curious how, looking at these attacks and other supply chain attacks that I have not brought up, do the [00:36:00] attack patterns that you have looked at in your research going to SolarWinds, XZ Utils, Log4j, how do they compare to these other incidents? And are there other messages and lessons from these attacks that maybe SolarWinds and XZ Utils and Log4j didn't speak to? 

Professor Laurie Williams: Yeah, I think without actually doing the detailed work, we can't really comment and I'll say that, and I'll put Sivana on the spot, of the three that you did, there were some techniques that were in all three. 

Sivana Hamer: Yeah, in all three. I remember one was something subvert chest controls, because again, that's very elemental of all these attacks. I think there's a lot of things that was- there's a supply chain attack technique that's like supply chain, so that one definitely was part of all the attacks. There's some other ones. I remember there was obfuscation, I believe that was part of all the attacks too. Because attackers, when they get into these systems basically they try to hide to stay longer and then do more stuff.

I think in the paper we do have the [00:37:00] 12 that are there. There might be overlap with the attack techniques, again in these new attacks? Maybe there's not, I think it's interesting to know how do they compare, what's this broad set of stuff. I guess without going into the whole detail of these attacks and doing like this whole manual work would be hard to say.

I've read about these attacks in a broad level and I guess I have some thoughts about them, but again, it's hard without doing the whole like approach. 

Professor Laurie Williams: So there would, I'm sure it may overlap, but then there may be some things that- because we all have generally a high level overview of what those attacks were, but when you get into the details, there may be some unique things.

Carolynn van Arsdale: Yeah and it's exciting to hear that you wanna continue this research, and looking at automation, 'cause as attacks continue to happen, because they will, you can do your magic and looking forward to seeing those results. So this is really the million dollar question here for you both that I have in regards to, what now?

So we had this great research that you all put out regarding what these frameworks missed, [00:38:00] what are the most important mitigation tactics that organizations need to take. As Professor Williams, I think you pointed out in the beginning, it's a lot for organizations to apply software supply chain security practices from these frameworks.

And I think oftentimes with the speed of software development, the speed of business, it can be a disincentive for software producers to care about these things to implement them. So if you were a cyber czar and you had the power to incentivize software firms to invest in the necessary resources to secure their pipelines and their supply chains, how would you go about it?

Does it matter if there are legal mandates, would that help the issue or are there other ways to compel security investments for software producers? 

Professor Laurie Williams: Yeah, unfortunately, I'll say I, I do think it's government regulations.

Carolynn van Arsdale: Gotcha.

Professor Laurie Williams: We saw the biggest jump- I was working with organizations like I did some [00:39:00] PSSCRM assessments during my sabbatical, and it was the executive order and the need to self-attest that caused organizations to all of a sudden have a supply chain security department. Without that, they wouldn't have had that. And yes, there's exponential growth in supply chain security attacks, so it's not that people would've done nothing, but because of the need, if you wanna sell to the US government, you need to attest, and everyone jumped on. And then the European CRA is again generating a reaction. I just think that those types of things have to happen for people to really take it seriously. 

Carolynn van Arsdale: Sivana, anything you wanna add?

Sivana Hamer: I guess we hope to help a little bit with the data side of having data that hopefully motivates a bit more. But I feel like regulation it seems to be also, I guess from my less fewer experience overall, but it does seem like it's good.

I think it's, at least in our space, the good thing is like there's a lot of, I feel like there's a lot of feedback that you can give to these frameworks and standards, like how to make them better too, if [00:40:00] you're interested in that. But overall, it does feel like mandates are helping drive progress.

Carolynn van Arsdale: Yeah. As you said, Executive Order 14028. I hope I got the numbers right.

Sivana Hamer: Yeah.

Carolynn van Arsdale: I've seen it so many times. I better remember it now. That really, as you said, was a catalyst for folks caring about software supply chain security, and before that, SolarWinds of course. So just to be mindful of your time, I know you know, we've talked about a lot of great things. Is there anything else before we close out today that you two wanted to talk about that I didn't get the chance to ask you that you wanna share with our audience? Any key takeaways?

Professor Laurie Williams: Yeah, one thing I'll say is just an invitation to collaborate with us at NC State. We do have a large National Science Foundation research center. And as part of that and really we always want to ground our work, software supply chain security is a practical problem that's affecting the world.

And so we can't be in the ivory tower doing our research without collaborating with companies. And so if it's interesting for [00:41:00] you to work with us, we would love to work with you. Some of the ways: we have three software supply chain summits per year where we sit down with organizations and organizations share with each other what their challenges are, what their approaches are and we're the flies on the wall listening for research ideas.

We have a community day, which is next week, annually. And we do PSSCRM assessments, so we'll work with an organization and assess where you are and give you a gap analysis and compare you to other companies. If people are familiar with BSIMM, it's really that type of an assessment. But even in general, if there's some research topic you think we should be looking at, we would love to hear from you.

Sivana Hamer: And I guess on my end again, thank you for the invite overall. I think it's really cool, I guess for us as academics and I think industry, government, and open source to collaborate overall, I think we all benefit from this, right? I guess also adding to apart from what we do, I know like OpenSSF [00:42:00] is a pretty big group and I don't think we have that much time to talk about like open source software and the big role it plays.

But I think that's important to bring up, they have a ton of great initiatives, and we work, I guess, with a lot of people in OpenSSF and we try, overall, just helping generally. I think the supply chain is at the end of the day, it's a problem that's created because we all depend upon each other by definition, in a way, by theory.

So it's only a problem that we can solve also if we all collaborate with each other. And again, and hopefully if you have any interesting problems, you can also contact us, because yeah, we're super excited to always work with industry, government, or open source. 

Carolynn van Arsdale: Yes. I know. I think that one of my favorite parts about working in cybersecurity and software supply chain security in particular, is the people, right?

The community is really awesome. Connecting with others in the community, sharing those ideas, especially folks at OpenSSF, Linux Foundation, OWASP. There's so many great organizations out there, folks like you in academia, and then also looking at industry and government. It's really important that, as I said earlier, it is a team [00:43:00] sport, of course. And for those interested in reading the report, learning more about Professor Williams and Sivana, we'll go ahead and link everything you need to know in our show notes once this podcast is produced and put out. And we'll also share all of that on our socials as well.

So Professor Williams and Sivana Hamer, thank you so much for joining us today on the Conversing Labs Podcast. It was a pleasure to have you on and thank you for your time. 

Professor Laurie Williams: Thank you for giving us the opportunity to share our work.

Sivana Hamer: Yeah, no, thank you. It was pretty fun. 

Carolynn van Arsdale: Of course. We'll see you all next time. Thank you so much for joining us today. Bye-bye.