In this episode of ConversingLabs, host Paul Roberts chats with ReversingLabs threat researcher Karlo Zanki about key findings in the new report, The State of Software Supply Chain Security 2024.

[ Get the report: The State of Software Supply Chain Security 2024 ]

EPISODE TRANSCRIPT

PAUL ROBERTS
Welcome everybody to another episode of ConversingLabs. I'm your host, Paul Roberts. I'm the cyber content lead at ReversingLabs, and I'm very pleased to welcome back into the ConversingLabs studio Karlo Zanki, who is a software threat researcher here at ReversingLabs. It's been a while since we've had you in. Welcome back, Karlo.

KARLO ZANKI
Thank you. Hi, Paul. Glad to be here.

PAUL ROBERTS
Happy Valentine's Day.

KARLO ZANKI
To all of them who are listening.

PAUL ROBERTS
I painted my office red just for the occasion. Not really. We're here to talk today about the report that we came out with a few weeks back here at ReversingLabs, The State of Software Supply Chain Security. The research that you and your team did was really incremental to that report.

I wanted to bring you in and talk over some of the findings, but before we do that, tell the audience, the folks here, just a little bit about the kind of work that you do here at ReversingLabs. Like what your job is and what type of research you do.

KARLO ZANKI
Thanks, Paul, for the invitation. I'm a software threat researcher at ReversingLabs, and I together with two more persons, are apart of the threat research team and our task is to monitor open source package repositories and try to find packages in those and improve our threat detection mechanism during that time. We do it on a daily basis and we try to keep up-to-date and find some interesting topics in that bunch of malicious packages that get published.

PAUL ROBERTS
And you've been doing this for a while, right? This software supply chain and researching open source threats. This is something that you've been doing for quite a while.

KARLO ZANKI
Yeah, for more than two years, we are actually monitoring package repositories. So it's relatively awhile.

PAUL ROBERTS
Talk a little bit about some of the big trends that you're seeing in the work that you do, just in terms of the threat landscape and for software supply chain risks, some of what, has come across your radar.

KARLO ZANKI
In the last three years, the focus of malicious actors is generally moving towards developers. They are recognizing developers as valuable targets, as some kind of power users which can be a good step into organizations. And software supply chain attacks are a way to compromise not only the direct targets, but also to cascadingly infect more valuable targets.

Everything started with the first known, most famous, software supply chain attack on SolarWinds company. And in the past year things continued in that way, and the number of malicious packages targeting developers increased both on the old target npm users, and it has also moved towards new targets, which include PyPI, NuGet, and likely other repositories.

The threat actors are scaling things up and wider, so they're pushing more power on all repositories and they're trying to catch repositories with their net also.

PAUL ROBERTS
Talk about monitoring software supply chain risks. I think a lot of people hear that and their head explodes. How do you even do that? Because even just looking at the open source piece of this, that is just a huge landscape of open source projects, platforms, developers, hundreds of thousands, millions of developers. What do we mean by that? What types of information are we here at ReversingLabs collecting and from where? And how are we using it?

KARLO ZANKI
Basically we are processing the daily published packages for package repositories, which are thousands, tens of thousands of packages or various package repositories that get published each day. We process those packages with our detained platform and that is a tool for static analysis.

So we basically do static analysis during which we extract some behavior indicators, which can be a dozen, one hundred, fifty behavior indicators per package, and files inside the package. And we look for anomalies for some suspicious combinations of behaviors and suspicious files. And we try to catch something interesting with that combination of behaviors.

So we did some research. We concluded, okay, packages that do this, and this can be suspicious, and we try to detect such various behaviors. There are hundreds of behavior indicators that get extracted using our platform. And that's basically the way we scale that big number of public packages.

So we try to minimize the landscape we need to look at.

PAUL ROBERTS
This isn't that different from the modern endpoint security, you don't look at malware based on a signature, but rather you're often looking for a set of behaviors or, so on that makes it easier to find stuff. But in this case, we're talking about the components, the kind of raw components of applications of stuff that might be hiding in code, really.

KARLO ZANKI
Yeah, there are various types, in case of npm and PyPI, we are mostly analyzing source files because that's the way they get published in PyPI packages. But also, the advantage of our platform is that we are capable of analyzing binary files. For example, compiled binary and DLLs that mostly get published to NuGet repository.

And the difference is that most of the traditional security solutions do some kind of dynamic analysis. Besides signature based detections. And we've mainly, our main tool is static analysis. So we extract behaviors during static analysis.

PAUL ROBERTS
So 2023, which The State of Software Supply Chain Security reports, mostly in data that we compiled during 2023, a busy year in the software supply chain security space. So ReversingLabs, on its own, detected a number of malicious campaigns on open source platforms and there were additional incidents as well that we didn't detect, but that we were able to help analyze. Talk about some of the malicious campaigns that caught your attention this year that you and your team discovered and why you find them interesting.

KARLO ZANKI
Probably the most significant one in the past year was in March, the attack on 3CX company. Why it's more interesting is because it's commercial software, it has a large base of users. And it was.

PAUL ROBERTS
Voice over IP vendor, yep.

KARLO ZANKI
Yeah, and it was conducted by a nation-state attacker as attributed. So it's advanced- APT, Advanced Persistent Threats, so the attacker was quite skillful. The whole attack was pretty complex. It was cascading events. It all started with a compromised application, which led to the compromise of 3CX applications. It's an example of how serious things can get because the 3CX clients, like with many other commercial software, receive automatic updates and they trust that software because it's digitally signed. And okay, it's some implicit trust and you allow that software to be installed in your computer automatically.

That's what made that incident quite impactful to end users. So it's distributed during automatic updates of the software.

PAUL ROBERTS
This is like everybody's worst nightmare. If you're an enterprise, right? You're getting software updates all the time. Over the last couple of decades, we've really been pushing organizations to do automatic updates, right?

To close that, that patching window. So not only you're getting them, but you're in theory, just applying them immediately and they arrive and then boom. Okay, this update had a backdoor in it.

KARLO ZANKI
That's basically what software supply chain attack is, why it's so dangerous. Because somebody is infiltrating something you have trust in.

PAUL ROBERTS
In this case we didn't discover the 3CX compromise. I think SentinelOne was an endpoint, EDR company that detected the malicious behavior and flagged it. And even then it took a while for people to realize that it actually was a compromise.

KARLO ZANKI
To be sure that the reporting was true positive, not false positive. It was a bit go here and there.

PAUL ROBERTS
But you in particular took a look at that compromised update binary file, analyzed it and were able to find some real, as we put it, red flags that were in that before, that might have tipped the company off to the fact that something was wrong with this update.

Can you just talk about some of the things that you discovered?

KARLO ZANKI
Yeah, the main point in that incident was the presence of the signed DLL library, which had content dependent behind the digital signature. So that's really uncommon practice. If you see that, it's likely suspicious because when somebody signs his software, he takes the software content, makes some hash out of the content, and creates digital signature which gets appended after it.

And if you push something behind that isn't covered with digital signature. So why would you do something like that for legitimate purposes? That's one of the biggest flags that flew over there and it's not that hard to detect. So that was something that could be analyzed, but of course, for that, you need to be capable to deconstruct the entire installer package, which is not a simple task.

So unpacking compiled packages from the various installer package types. It's a complex task, and that's something in which our product is good. It spores a wild number of file formats that can be automatically unpacked during static analysis. So I don't mean running the sample in some sandbox and catching the dropped files... you push a package into our solution and we take apart statically without executing that package, it's components and their the threat could be detected based on the shared indicators. 

PAUL ROBERTS
And again, this is stuff that any software producer should be looking at as a way to identify unusual behavior that may not necessarily be malicious, but like you said, the sort of the coincidence of all these behaviors or features raises a likelihood that something is going on.

KARLO ZANKI
The first thing you need to do is to be aware of the possibility for something like that to happen, and to have procedures in your organization that force you to make security assessment of the software you are going to install on your system. I believe that most of the companies are still failing at this step. So you first have to be aware that you need to do that, then you need to make procedures to make it work, and then you need to employ tools which can help you do that.

But I believe the gap is still too wide at the place where people are still not conscious enough about the possibility of such attacks happening to them.

PAUL ROBERTS
So one of the things that we called out in The State of Software Supply Chain Security was a big shift, first of all, a continued increase just in the number of malicious packages we found on open source repositories. So like you said, Python Package Index, npm, NuGet... 28 percent increase year over year between 2022 and 2023. But a big shift in where we found those packages from, historically npm, which has the most open source projects and developers working on it and therefore had the most threats. This year in 2020- or last year rather 2023, we saw a big shift here at ReversingLabs to the Python Package Index, PyPI, as where we were finding suspicious or malicious packages. Talk about that, Karlo. What explains that?

KARLO ZANKI
Basically, I wouldn't even call it a shift. I would call it shift, meaning that they lost the focus on that and turn it here. They are just expanding the net, which they are trying to, and instantly, there were several bigger campaigns, which included more packages published in an automated way, which are being apart of the same campaigns.

So the numbers for that reason are a bit bigger, but I mentioned only 3CX from the last year. There are also several original research, which we detected, which are mentioning as you said, npm, PyPI, NuGet, and the attackers are constantly improving the techniques they are using these attacks.
Regarding your question, I wouldn't call it shift, I would just call it as expansion. They are still present in npm. They are periodically trying some new, but they are also expanding to new repositories using their weaknesses to exploit them while those repositories haven't adapted security measures to counter those threats.

And regarding those situations, we shared several blogs. One was talking about compiled PyPI files, so they were using compiled Python files, which are harder to analyze than standard source code Python files, to hide malicious behavior in them. So they are just exploiting some weak spot in the PyPI repository.

We said Operation Brainleeches started using npm packages. So it's again, big campaign, several packages which were combining both phishing campaign and software supply chain campaign in one big campaign, let's say. We said also VMConnect was big because it was conducted by a state-sponsored actor. We were able to attribute that actions to North Korean threat actors, which are one of the biggest contributors in the second half of 2023, because they are trying to infiltrate development organizations to steal crypto tokens, logging credentials, similar stuff, wallets.

So because they are hunting for digital assets to aid them in financing their activities. That was a big story because in the second half of 2023, North Korean actors, state sponsors, started targeting both npm and PyPI repositories. So that's another example where we can see that they didn't turn focus, but they are targeting everything.
We also had in the later part of the year, a campaign, IAmReboot campaign, targeting NuGet repositories-

PAUL ROBERTS
IAmReboot.

KARLO ZANKI
We didn't see much malware being talked about found in NuGet repositories, but in the second half of the year we managed to see that since August there was a threat actor group that was continuously trying to push malicious packages to NuGet, and they are constantly using bracing edge techniques to acquire execution.

They were employing three different techniques, which were quite novel, not so novel, they were seen before, but weren't talked a lot about and weren't used previously by malware. So we see activity across all package repositories and the attackers are trying to push where they feel they have the most capabilities.

PAUL ROBERTS
I know, obviously in the, just in the traditional malware threat landscape there are highly targeted kind of custom malware designed for specific environments or specific targets. But obviously, most of the activity is low skill automated, spammy type stuff.

Do we see the same dynamic playing out in the open source space? I know our report, like we, we would filter out what we considered sort of spam campaigns that were just automated and low quality, low effort, just to have a higher fidelity kind of data stream. But do we see that same dynamic playing out in the software supply chain context where, again, you might have an automated campaign that's throwing, thousands of packages into a package manager like npm.

KARLO ZANKI
Always like the pyramid, you have the biggest portion of attacks are low complexity attacks and the smallest number of samples are high complexity, but in the later half, last two or three months, we have seen a big increase in usage of open source platforms and open source malware used during this campaign.

There are a bunch of open source malware hosted on GitHub, I don't know, Empire Framework, Havoc malware, TurkoRat, different kinds of malware which get published to GitHub with that always note don't use this for running educational purposes. So malware is using open source malware, that's like software is using open source software. They are adapting the same logic. Such a big number of open source malware makes it easier for malware authors to create those simple malware with low complexity that gets published most of the time. But here and then we see custom malware which get used in directing campaigns like 3CX.

And that is high value malware with the biggest impact for the end user. So it's always like a pyramid.

PAUL ROBERTS
And with that 3CX attack, I think we know that the ultimate target of that attack were 3CX customers who were in the cryptocurrency industry, it was a small number of the customers who actually got targeted and there seemed to be a common thread of links to crypto.

So we've talked about two, two different kind of scenarios here. One 3CX commercial software closed source that's used by a lot of companies, large and small. And the other, just these wide ranging open source threats as a, let's just take like the software producer side.

You're making software, you're almost certainly using open source modules and components, libraries in whatever application you're making. You're also probably using proprietary closed source components. What do you do as a software producer to assess the risk, both from the open source stuff you're using and from the proprietary closed source stuff?

KARLO ZANKI
There, there are several steps we should do, but you should always be aware that the potential risk can be present anywhere. One of suggestion is to make some trust checks. Is it a verified publisher? How often does he publish? Is this the first version of the package? That is okay, there are good heuristics. But you can take example of Ledger Connect Kit case. In that case, you had a trusted publisher who lost control over his npm account and malware got published, malicious versions of that package got published. Up there and in matter of hours, four to six hours, a lot of customers downloaded malicious versions of that.

So trust factor itself isn't enough to make sure that there is no malware in your package. What you should do is always check what you use in your product, make security assessment of it, not just trust based, but use tools that can help you find malware inside. Most of the developers don't have their knowledge, nor technology, nor capabilities, nor time to make security assessment of all that open source packages they are going to use.
But there are tools out there which can help you with that. And the second thing you need to do is also make sure that you verify what you are distributing to your customers. When you package something, you don't just only use them. Make a validation that you don't have malware inside that, that it's exactly what you wanted it to have, that there are no subframe behaviors in it.

You always need to verify what you're putting into your product and verify what you are distributing to your software. Like 3CX case, they could have detected the malware before it got to their customers if they analyzed what was coming out. Even if they didn't. Catch it on the input.

PAUL ROBERTS
Okay, final question is the flip side of that, which is, as the end user, as the software consumer, the end user organization again, you're downloading software and updates from all different vendors.
And what lessons should you be taking from a report like State of Software Supply Chain Security? What should you be doing differently to address this risk that, hey, that software update, that sign software update, may be malware.

KARLO ZANKI
Again, you likely channel even more knowledge, capabilities, money, and resources to make security assessments of the software, which you are going to use.

But... The same thing, you should be in the same position as the developer. You should try to write to some security tool that can help you with that and try to analyze if it contains security risks. You likely won't be able to do any security assessment on your own. You will have to use some tools.
But you need to be aware that there is threat for you, especially in today's world, when there are lots of users of cryptocurrency. I don't want to offence anything, but people from plumber, all the way to financial broker, are doing crypto today. And they are all in danger of losing their digital assets by installing untrusted software onto their computers. So everybody should be aware of that.

PAUL ROBERTS
Final, final question. Looking ahead in 2024, any interesting trends or patterns that you're seeing that organizations, software producers, software consumers should be mindful of, should be thinking about or aware of, just in the threat landscape?

KARLO ZANKI
I believe that the package repositories will still get crumbled with malware samples published to them. And things which communities should be looking at is malware is going to use open source infrastructure for malicious purposes more and more. It's like using GitHub for hosting malware, for issuing commands to malware and similar stuff.
The malware actors are adapting all the time and they are always using breaking edge technologies. I don't know, non-mainstream languages like Rust or Go, they are compiling things using Nuitka compilers, for example, for Python. So they're always using non-mainstream technologies beside using the regular stuff, you need to be aware of that too.

But the main I believe trend in next year is going to be usage of open source infrastructure for malicious purposes and it's going to scale up. We have seen it on PyPI, NuGet, we have seen it on npm, so it's something that is going to be present all the way. Regardless, I believe one of the main targets will remain crypto assets.

PAUL ROBERTS
If that's your industry, you need to be particularly-

KARLO ZANKI
It's the easy money, and there are targets of typical criminals, financial motivators, and also state-sponsored attackers like North Korean, which are seeing that as one of main funding in foreign currency, let's say. So you should be aware that cryptocurrency are one of the main targets.

PAUL ROBERTS
Karlo Zanki, Software Threat Researcher at ReversingLabs. Thanks so much for coming on and talking to us about the State of Software Supply Chain Security 2024, and we look forward to having you on again soon.

KARLO ZANKI
Thank you, Paul. I had a great time. Hope to see you soon.

PAUL ROBERTS
As did we. Absolutely.

[ Get the report: The State of Software Supply Chain Security 2024 ]