In this episode, host Paul Roberts chats with Daniel Adamitis, a Principal Information Security Engineer at Lumen Technologies' Black Lotus Labs about Volt Typhoon’s exploitation of SOHO routers.

EPISODE TRANSCRIPT

Paul Roberts: Well, welcome back to another episode of ConversingLabs. I'm Paul Roberts, the host of ConversingLabs and the Cyber Content Lead here at ReversingLabs. I'm very pleased to have with us Danny Adamitis of Lumen Technologies, Black Lotus Labs. Danny, welcome. I think this is the first time we've had you on the show, thrilled to have you.

Danny Adamitis: I hope it's not the last time.

Paul Roberts: It definitely won't be. So for folks who aren't familiar with you, Danny, talk about the role that you have at Black Lotus. Black Lotus is like a lab that operates within Lumen, so just talk about that relationship too. 

Danny Adamitis: So we'll start off with the high level overview. I was hired almost three and a half years ago into what was called CenturyLink. And then during the pandemic, CenturyLink started to rebrand as what we're calling Lumen Technologies, and we basically started to transform your traditional ISP into more of a technology firm. And when we did that, we obviously started offering a number of security-based solutions. We started to add things like our Bare Metal Edge, we started to offer more SASE, we started to offer some of these security products. The next thought was, how do we start mining these security products to better protect our customers, better protect our enterprise, and better protect everyone? So our solution was what we call Black Lotus Labs. This is for those of you who are familiar with the security space, it's like an equivalent to a Unit 42 or something like that, where the idea is we take all the telemetry from all of our various products that we have as Lumen, we mine that, we correlate all the different data points, and that helps us to uncover some of these very low fidelity signals. When we talk about this, our big thing is we have our core ISP backbone, so this was created from our former level 3 SX, we run a number of cables, and we just are able to get some sampled net flow. I believe it's something like 200 billion sessions a day, and that allows us to create some of these really fun internet maps that we're going to talk about in a minute for Volt Typhoon. We also have things like DNS data from our resolvers, we have access to some of our various managed modems. Again, these are things that we operate on behalf of Lumen. We also take our customers privacy and security very seriously, so we try to help them. And when we look for some of these threats, we try to understand them, reverse them, provide some context, and then we issue back these kind of reports to the community in the form of a blog, which are available to anyone free of service, there's no signup. This is our way of trying to keep everyone in.

Paul Roberts: That's really interesting because one of the limitations of any security company is often just, you've got a narrow view of the world that's really limited to your customer base, whatever that customer base is. So company like Lumen, obviously that's a huge base. What types of things do you monitor for? And what types of unique insights do you find that just having that huge base of monitoring gives you? 

Danny Adamitis: I love to tell fun stories because I feel like it just makes things a little bit easier. So again, when you think about people like Microsoft, I feel like some of their best reports come when things start going after the Microsoft ecosystem. When you talk about Wiz things, it's going after cloud. My thing was, I'm trying to harness this and we're an ISP, so I want to focus on things that are most prominent to an actual ISP. And when you take a look around what we actually have, it tends to be a bunch of old servers. There are things like Linux, Solaris, it's just a ton of routers. How do we mine the assets we have at our disposal? And then, how do we make these really cool, interesting stories? This is one of those creations where we started looking at things like the SOHO router space, and we were able to start seeing some weird activity, and we were able to actually pivot off this. So again, this kind of then links us to our next story. This actually all started back in the week of Christmas 2021, when we were previously working on another campaign called ZuoRAT. For those of you who aren't familiar, we have a blog post out there. The super high level is that this was a very advanced piece of malware that was targeting SOHO routers, and there's two things that really just enamored my attention. The first is that they have this ability to do DNS hijacking, and then there is a second capability to do HTTP hijacking. When I saw this, my thing was, you don't just put these sorts of capabilities into a malware to not use them, there has to be some sort of DNS or HTTP hijack server that exists in the ether somewhere. So, we started monitoring all of the known compromised routers to see if we can identify something that looked like this abnormal HTTP hijack server. And that's actually where this investigation started, we found our first kind of server that we're now calling the JDY cluster, and that it was communicating with around 100 Cisco RB320s. And when we started looking at this, it's really anomalous to have 100 U.S. based Cisco RB320s and 325s exclusively communicating with one server based in Volkshire with a self signed node every single day with a regular beacon interval. 

Paul Roberts: That's Cisco router. That's a SOHO, that's a small office, home office router. Yeah. 

Danny Adamitis: So this is like a smaller one, something you might have in your house. We always say kudos to Cisco, like this is something I would put in my grandmother's house. So my grandmother loves to have internet. She loves to, you know, watch her telenovelas. It's all great and dandy, but I want something with a lot of uptime, where again, I don't want to get phone calls every other day asking me to fix the router, or I can't connect to the internet, or what's going on. You want something that has high uptime, very low maintenance, and just constantly runs all the time. So again, these things were available. They weren't super expensive, I think they were like $100, $200 dollars, but because they've just been out there, and they have this really good uptime, it makes for a really good access vector because it just runs all the time and it was designed to push out a lot of bandwidth so you could use it for a lot of fun and interesting things, rather than just watching your standard Hulu and Netflix like everyone else. 

Paul Roberts: And when we're talking about SOHO devices, you're talking about broadband routers generally, but that's not all that these malicious actors are interested in. Just reading through the Black Lotus research reports, there are other IOT devices, cameras, and stuff like that, that are also on the target list. Can you just talk about what types of hardware they're interested in? 

Danny Adamitis: Yes, so we saw it was predominantly three different types of operating systems. So we saw the Cisco RV320s, which were all MIPs. We saw a number of the NETGEAR pro safe devices, which actually run arm. And then we also found a handful of targeted access IP cameras towards the end of December 2023. And again, the easiest way to think about this is that when we actually talk about things like Volt Typhoon, a lot of people seem to think of this as one kind of huge homogenous thing. And in my mind, the way I think about it is it's almost two separate activity clusters, or two separate intrusions, that are working concurrently with each other. So what we were tracking was what we call this enablement piece of the KV Botnet. The KV Botnet itself was actually designed to create this covert infrastructure. So again, if you have anyone who works in security, they'll know, okay, the first thing you do is you configure your firewall to block all Chinese ASNs. We're going to block things like maybe some of these ASNs that tend to be a little bit shadier and accept Bitcoin, but we're going to allow all of our remote employees to connect into the firewall because that's what remote work really is. So what this piece was actually doing is it was creating all of these kind of covert hot points that allow the Chinese to pivot through various different points within the United States to actually look more like a remote employee, and then they allow them to evade that kind of standard detection. Once they did that's when it starts with that second intrusion. And that was what our counterparts at Microsoft were focusing on. That was more of the living off the land, you're running your NTDLS, you're grabbing your credentials, and doing some of that stuff, where that's more of your like traditional campaign.

Paul Roberts: So these networks of hundreds or thousands of these routers, this isn't about DDoS, it's not huge populations of these. This is really just about spreading that attack footprint out and also launching attacks, whether that's brute force password attacks or what have you, from what looked to be domestic U.S. IP addresses that aren't going to set off alarms. That's the idea, how do we look as normal as possible in the early stages of this attack? 

Danny Adamitis: Yes, we were tracking this one right there. We're actually deploying malware and we're tracking a couple other ones where they're just deploying a modified version of Beardrop. And again, the idea is you're just creating SSH tunnels to look like anyone else who's coming from middle America. 

Paul Roberts: Yeah, your reports have a lot of info on how this malware works. Often what's invisible is the earliest stages, the initial compromise, but I'm guessing you probably have some sense of how the earliest stages of these attacks play out. What is your sense of they're hacking my SOHO, my broadband router in my basement with the cobwebs on it. How are they doing that? Or is it even an attack on that router? Is it an attack on me personally and then they're moving laterally to my router? 

Danny Adamitis: We assess that this is primarily an attack against the router itself, so one of the other kind of big things we want to note, especially with this report, is it tends to be a lot of what we call end of life equipment. So what this means is that this is no longer being supported by the manufacturer. This is not being supported by Cisco, NETGEAR, and these other firms. This is very routine across the industry, this is not specific to them. They basically will release the software and they go, we'll support this for anywhere from, 7 to 15 years. But the thing is, a lot of the underlying actual code base itself as it's all common, a lot of the stuff is built on things like the Linux kernel. And again, it's the Linux kernel at the time it was compiled 15 years ago. So a lot of the times, if you're able to see a new CVE that comes out tomorrow, a lot of that code is actually still embedded in these older systems. So our running suspicion is that they're just taking the latest and greatest CVEs, kind of modifying some of the NOP slugs to work against these older kernels or these older software variants. And again, without these things being end of life, there is no software update, there is no patching mechanism. Once it's out there, it's going to be exposed and vulnerable to all these attacks. So really the only thing you can do is, unfortunately, rip and replace or take that piece of equipment off your stack and just remove it from the internet. But no one ever wants to do that because I already paid for it, it's running, I'm still getting my 99 percent uptime, and the golden rule of IT is, if it's still working, don't touch it. 

Paul Roberts: That's right. 

Danny Adamitis: It goes against that natural instinct a lot of people have when they talk about the circuit.

Paul Roberts: And we saw this not just with SOHO devices, but actually the Ivanti Pulse VPN attacks, as we know from Eclipsium, those were also devices running an end of life CentOS operating system that hadn't been updated in four years, because it was end of life. And these are devices running in CISA, high security organizations. But there was not visibility into that on the part of a customer, right? Because the firmware was encrypted and, and a lot of companies are not asking, what operating system is this thing running? 

Danny Adamitis: And this is the thing is they're trying to pivot into these areas where they don't have good visibility. So again, this is one of the nice things is I will say over the last, 10, 15 years, I remember before you used to be able to throw a malicious document, run PowerShell on the Windows system, and you could probably do that for a couple of days, if not a few weeks before anyone really noticed. And again, things have gotten significantly better, which is great, and I'm very appreciative of that. But there's still really no good solution for things like edge routers, there's no good solution for things like firewalls, there's no good solution for, again, a lot of these big enterprises are still running a mainframe or Solaris system. Point me to the EDR product that supports mainframes. 

Paul Roberts: Lumen Technology, but not EDR. But there's this real accountability question, right? And you really see that with SOHO, because from the homeowner's standpoint, does it matter if the Chinese own my SOHO router? If I still get Netflix not so much. Does it matter for our economy, for our society, if the Chinese are owning millions of SOHO routers? Yes, it does, right? But there's this kind of individual versus corporate versus government accountability gap, right? Where the security of those little devices does matter, but who's on that? Whose responsibility is that? 

Danny Adamitis: Yes, and this is the problem is it turns into a tragedy of the commons for all of my other poli-sci friends. Where again, everyone gets to benefit from it, but who's actually paying the cost for it. Again we're doing our best, I can say from the Lumen perspective, to try to monitor for these things and alert on them. But unfortunately, if it's coming from a different ASN, there's not really a whole lot we could do other than send our counterparts a lovely email and hope they get to it when it gets to it. 

Paul Roberts: The other interesting thing, and I think you pointed this out in the ZuoRAT, is that you have seen evidence that these aren't always attacks merely aimed at some other target, right? Oh, we just want to do brute password attacks that in some cases, you've seen SOHO router compromises where the attacker is actually interested in who owns that router and what's going on within that local environment. Can you just talk about that? 

Danny Adamitis: Again, foresight, we're working on another blog, where we break down some of the stuff, where we talk about the different types of SOHO base attacks. So one of the ones we talk about is what we call these trains that are passed through networks. So a great example of this is something like the KV Botnet, where the main goal is to get this and if we boil it down, they're stealing bandwidth, and they're looking like a U.S. person, or again, a person from whatever country they're targeting. But that one doesn't really pose any threat to the actual underlying person, organization, or anything like that. There's a second type of attack where, again, we highlighted this last year, where we did a report called HiatusRAT, where they were actually deploying things like a packet capture tool, so that one does start to pose a threat to enterprises because okay now they're starting to passively collect everything that traverses through my router that hits on these certain issues, whether it be a port, a protocol, an IP address. And then the third one that we're talking about here is what we're calling a connection hijacking malware. So this is an example of things like ZuoRAT, and we have some new stuff coming up soon, please keep an eye on our blog, where essentially they're looking for connections that are stemming from the adjacent land, and then they're actually trying to hijack that connection to enable their access to the adjacent network. So again, this was something that we find to be extremely concerning, especially during things like pandemic, when there's still a lot of companies working from home, that they could actually target these kind of end of life devices, they can use as a foothold, they can move into things like your corporate laptop, corporate cell phone, or your personal cell phone, and they can start mining all of that data there and then use that to walk into your actual corporate network. So it's a very big problem that doesn't really have a great solution at this point. 

Paul Roberts: What would your fix be for this EOL problem, where OEMs can basically arbitrarily decide when they're just not going to support software anymore, stop issuing security updates. You're right, generally it's seven or eight years, but it doesn't have to be, there are no laws around it, it could be three years. And we see this in the consumer space a lot, like just this kind of, we're done, we're into lifing this product, sorry. What would your fix be for that? Because as you've pointed out clearly, there's a kind of public health impact on that, on us.

Danny Adamitis: I think it's a multifaceted problem. So the first thing is, there should be at least some regulation against, is this going to be supported for again, I'm not saying it has to be 20 years, but maybe it should be more than two. And there should be some middle ground that, again, everyone's going to lobby for. The other thing that we're trying to push for and help with is that kind of secure by design stuff. Where the idea is, I know this is a big CISA initiative, I'm going to be honest, it's not going to save us today, but if we can start planning those SIGs today to potentially start having that impact in five years from now, then, someone else won't have to deal with the same problems that we're dealing with now. So again, we see that as an investment and that's something we could be doing now, we could be moving to more memory safety that could help eliminate some of these bugs. It's not going to make software impenetrable, but if we can raise that bar a little bit, it's going to make it a little bit more difficult. And again, if I'm being honest, at the end of the day, the only thing to do is try to set more trip alarms and make everything harder for the attacker and trying to create more canaries that way when something does happen, it's not, oh my god, this happened three years ago, it's okay, we saw this was yesterday, let's start getting on top of it. 

Paul Roberts: And I know you've got a hard stop at the top of the hour, but a couple more questions. CISA just came out with a fact sheet on Volt Typhoon, I think yesterday, the day before, and I threw the link into the chat for the people who are in the audience here. This Volt Typhoon is this Chinese APT that you've seen behind a lot of the campaigns that you were talking about. Interestingly, the recommendations also talk preventatively for organizations pay a lot more attention to your software supply chain and to your vendors and the software and applications that you're using. That's easier said than done these days, but for organizations out there that are reading that and being like, how do I screen software providers around the types of things that a group like Volt Typhoon is targeting, where do they start? What's on that list of things to look for?

Danny Adamitis: So there's a quick and easy problem. To get to the heart of the matter, the issue is it's going to be incredibly difficult because again, if we're going to take this at face value, the answer is you're going to have to reverse all of the software updates to bring send into your environment, which takes a lot of time, a lot of energy, a lot of manpower, woman powers, it's going to be difficult. The other solution that again, I feel has actually had some promise in the past is implementing some of these best practices that everyone seems to forget about. So when I talk about this and you brought up things like the Ivanti attack, the reason at least they said publicly that they found this is because they noticed things like lateral movement. So again, by setting up things like virtual LANs, and by setting up some of these east west detections, we can start going, hey, I'm not quite sure what exactly is happening with my software vendor, but why is this server that no one ever logs into running PS? Like there's things like that, that I think could help trigger all this. Now, again, I'm just going to be more of a security realist. You're never going to be able to stop all 0-days. China is going to find an 0-day, whether it be a software vendor, whether it be a perimeter firewall, whether it be an API, we can't say that we're going to make an impenetrable fortress. But what we can do is actually put additional MOCs behind that initial wall, so that way, if they do try to move laterally from our firewall to something like our domain controller, we can actually put better detections in place for things like domain controllers to go: who is running NTDLS? Why are they running this? Is this a person? If this is a real IT staff, call the person on their cell phone and say, are you actually doing this or is this someone else? Again, if you're an enterprise, you can call your employees during work hours and ask these questions. And there's just some things like that where I think it's more of that focus has to be on the east west movement, locking down accounts, implementing things like hardware based tokens. Again, that's something that everyone has known. These YubiKeys and stuff have been around for 10 years, they're still not widely adopted. If you're operating a high security environment, any domain admin should be running hardware based tokens. And again, the idea is that going to stop anyone? Could they still perform operations when the add thing's on box and use everything? Yes. But again, we're trying to raise the barriers to entry to make it harder for people to do this sort of stuff and kind of trip off the alarm. Where again, they can't just walk in as we talked about in the blog, they can't walk in over Christmas break when everyone's at home with their families and run ragged for two weeks and get away with everything before everyone comes back.

Paul Roberts: Which we've seen every holiday season, pretty much. Really good points, and I think that is an incredibly important point, which is talking about software supply chain risk isn't about building an impenetrable wall, it's really just about responding to where threats and attacks are moving, and at the end of the day, no defense is perfect. You still need to have those layered approaches to monitoring and detection and so on. We hear a lot from CISA and the government about software bills of materials. Would that help on the end user organization side?

Danny Adamitis: I think it helps to know what exactly is in the software. So we hit on this briefly before we were talking about SOHOs and the firewalls. Are they running what version of CentOS? And I think the Software Bill of Rights will start addressing things like this. Oh, this is running CentOS version 2? That kind of helps people prioritize patching. Because again, at the end of the day, this is also going to be a resource constrained environment, there's going to be how many patches that come out on Tuesday, which one gets done first? And I think something like software bill rights can help stakeholders prioritize some of that tasking. We could argue that things like perimeter devices that have a remote code execution should be first. And again, everyone will go, yeah, that makes sense. Okay, but then what do we do after lunch? What do we do for those other four hours? And I think that's what helps them to prioritize what needs to be done first and what can potentially wait until Wednesday. It's just because, unfortunately, a lot of these organizations just don't have the resources to get everything done in the first four hours. Or, again, they're afraid, they're hesitant to push out patches because it might cause, downstream software issues, and they don't want to break their own product because downtime is costly.

Paul Roberts: So before we go, last question, Danny. Obviously, folks are listening in really interested in what is in your pipeline? What is Danny and the Black Lotus crew interested in working in investigating? What do we have to look forward to? Because your reports are always eyebrow raising.

Danny Adamitis: So we have a couple more things coming out from the SOHO router space. I think we have one potentially next week. So we have some things coming up from the crimeware flavor of some other stuff that's talking more about nation states. And again, I think we're going to see what else we can do. We're not only doing SOHOs we actually did in my opinion, a cool one last year, I think it was November, where we found someone using a EBPF kernel exploit for things like a perimeter device. So again, this is one of those things where, we're also acknowledge that this is two sides of the coin, like we talked about with Volt Typhoon, there's a lot of things to go after SOHOs, but then they still have to do something on the corporate network. So we're going to see what we can find in the next space, and if we find something that really grabs our attention, I will be sure to let you know. 

Paul Roberts: So stay tuned. I'll definitely be watching your RSS feed. Hey, Danny Adamitis, thank you so much for coming on and talking to us about the amazing research that you do and and thank you for doing it. And I really look forward to having you back on ConversingLabs. 

Danny Adamitis: Thanks. I look forward to being back. 

Paul Roberts: Absolutely. Take care.