September Threat Thursday
September 05, 2019
Chris Goettl | Director, Product Management, Security | Ivanti
Phil Richards | Chief Security Officer | Ivanti
We're analyzing the biggest security threats currently impacting global IT teams. Here's what we're tracking for the end of August into September:
- An ongoing ransomware attack that's hit 23 Texas cities
- An update on a Cybersecurity State of Emergency in Louisiana
- Updates in the Capitol One data breach
Join Ivanti's Phil Richards and Chris Goettl as they deliver the insights and news you need to stay vigilant.
Jared: All right, welcome everyone to our first ever Threat Thursday webinar. We've got a lot to cover this morning, but first I wanted to give you guys a few pieces of helpful information. We're recording this webinar, and later today, you will receive an email with the presentation and the slides. Feel free to share it with your colleagues, your parents, your dogs, your kids. Also, we love questions. Utilize the Q&A in chat, in the WebX. I will see them. I'll make sure the presenters get those questions, and we'll be answering those throughout the entire presentation.
Jared: Also, if you're having audio issues, it might be best to dial in, and I'll share that information in the chat right now so that you guys can make sure that you have a great connection now.
Jared: Without further ado, I'd like to introduce Phil Richards and Chris Goettl, joining us from the Salt Lake office. Guys, welcome to our first ever Threat Thursday.
Phil Richards: Hey everybody. Thanks, Jared, and welcome, everybody, to the first Threat Thursday vlog or webinar that we're going to have. This is the first [inaudible 00:01:13] Chris of webinars that we're going to do. Probably do them about once a month. We'll kind of do them opposite of the Patch Tuesday webinars. So, you get Patch Tuesday on, what is it? The second Tuesday?
Chris Goettl: Second, second Tuesday of the month.
Phil Richards: And then we're going to try to schedule the Threat Thursday ones on the fourth Thursday.
Chris Goettl: Right.
Phil Richards: We're a week late this month. It's because of me, though. Frankly, I'm a bit lazy.
Chris Goettl: Oh, you know. You know, there was a lot of things to get done for this.
Chris Goettl: We've got some really cool ways of representing this content. It's something that we talk about a lot at threats and things on the Patch Tuesday webinar as well, but it became very apparent that when you look at attacks that happen, and you break those down and you look at the tactics used, you look at the motives behind it, we can start to really understand how best to combat threat actors, if we look at these types of examples.
Chris Goettl: So that's what Threat Thursday is really about. It's a way for us to be able to try to break these things down and give you... I don't know if it's sage advice. I don't know if we're sage. [inaudible 00:02:16]
Chris Goettl: Give you guys some- [crosstalk 00:02:17]
Phil Richards: Some advice.
Chris Goettl: Some expert advice. Some insight into what made these attacks successful, what drove the motivation behind them, and how you might combat them.
Phil Richards: We're also going to try to look at some common threads between these different attacks so that you can kind of see underlying what maybe is moving the markets, and that provides some insight into what groups or entities or companies might be vulnerable.
Chris Goettl: Absolutely.
Phil Richards: So, let's talk a little bit about our agenda. We're going to actually, this month, we're going to talk about initially, at least, ransomware in the public sector. So, our agenda items are kind of fun. We'll kind of go through that a little bit, and then after a couple of different use cases, we'll kind of try to pull together some information around what's common between those attacks.
Phil Richards: Then we'll finish up with an additional attack that kind of doesn't talk [inaudible 00:03:15] about ransomware in the public sector, but it's a few things. Then we'll wrap it up with some Q&A after that.
Chris Goettl: Absolutely. So, starting off, don't mess with Texas.
Phil Richards: Don't mess with Texas!
Phil Richards: So, what happened here is, during the month of August, 23 municipal government entities were hit with a ransomware attack. Now, we indicate here that it's a coordinated ransomware attack. Chris, what's a coordinated ransomware attack?
Chris Goettl: So, this is an attack where a single threat actor is launching a simultaneous attack across either a large environment, or multiple environments of a similar type. In this case, a bunch of government agencies throughout part of Texas.
Phil Richards: Yeah. So, that attack, the interesting thing. We'll talk about this in a little bit more detail when we get to the point of the discussion, but obviously the intent of the attack is to provide the most concern, or wreak the most havoc, as possible. And one of the ways they do that is by making sure that everything kind of blows up all at the same time.
Chris Goettl: Absolutely. So, you know, there's a couple of different types of ransomware attacks. There's the day-to-day, "Okay, we're just going to create ransomware as a service. We're going to spam things out there. We're going to launch all sorts of phishing campaigns, and we're gonna hope to hit a machine here, a machine there. Eventually- [crosstalk 00:04:42]
Phil Richards: [crosstalk 00:04:42] Maybe a third of a Bitcoin or a tenth of a Bitcoin, or something like that.
Chris Goettl: Exactly.
Phil Richards: Chump change.
Chris Goettl: Chump change. That's somebody who's setting things in motion and then just letting money come in, and it either does or it doesn't.
Chris Goettl: The attacks we're talking about today, these are coordinated, very focused attacks. They're going to be a much bigger payout. In fact, the payout that was demanded for this Texas incident was $2.5 million.
Phil Richards: Wow. So, that's a lot of Bitcoin.
Chris Goettl: It is a huge amount of Bitcoin. To get that type of payout, though, the attacker needs to cause enough pain and create a sense of urgency that a- [inaudible 00:05:23] [crosstalk 00:05:23]
Phil Richards: I want to actually divert for a half a second. Why Bitcoin? Why do the attackers want to receive payment in Bitcoin? I mean, gold is nice. Dollars are fine. Why Bitcoin?
Chris Goettl: So, Bitcoin makes it harder to track them down. It does make it harder to. It makes it an easier payout, as well. There's a lot of facilities that can make it so that somebody can transact a payment and get Bitcoin and be able to pay somebody in Bitcoin electronically. It makes it so much easier to collect.
Phil Richards: Anonymous.
Chris Goettl: For an attack. Anonymously, yes.
Phil Richards: Okay, good. So, obviously the exposure type is a system lockout. Ransomware basically says, "Hey, we're going to lock all your files and we're going to trash all your computers, and until you pay us a ransom, you're not going to be able to get access to those files."
Chris Goettl: Right. So now in this case, probably the hardest part about this was the threat actor had to spend some time and effort figuring out, "How am I going to hit a large enough set of environments to warrant a large payout?" So they found a method to get into these 23 government agencies.
Phil Richards: Yes, they did, and that's what we have listed down here under attack vectors. MSP stands for...
Chris Goettl: Managed Service Provider. [crosstalk 00:06:44] Somebody you trust.
Phil Richards: [crosstalk 00:06:44] Managed Service Provider. Yeah. So, Managed Service Provider, those are the guys that for a lot of these entities, they might manage their routers, their firewalls. They look at the log entries on their servers and sometimes on their work stations. They review incidents that are coming in for those entities, and the interesting thing is that the attackers actually used their access into these 23 local government agencies as the way in to plant this ransomware.
Chris Goettl: Yeah, and one thing to be very clear on is, we're definitely not attacking MSPs on this. MSPs are of huge value to companies. Companies today, we can't hire enough talent. Right? We can't keep all of the skill-sets that we need. So it's natural that we turn to Managed Service Providers to help build [crosstalk 00:07:38] those [inaudible 00:07:38].
Phil Richards: Exactly. We're not even saying in this case that the MSP actually even did anything wrong.
Chris Goettl: Correct. Other third parties that you work with. It could be your HVAC vendor, it could be your physical security vendor. It could be any other source that comes into your environment and is granted access for any reason, and we're actually going to talk about one other scenario later today that's not ransomware-related that covers this as well. Those third parties that play in your environment do warrant a level of scrutiny and assessment in your security plan.
Phil Richards: And more and more, we're seeing attack vectors that target those third parties, which is why one of the things that we have listed down here under recommendations is vendor risk management. It is really important that not only you make sure that your vendors are doing the right kind of thing from your perspective. They need to be hiring people the way that you would expect them to hire and vet and validate employees. They need to have the same kind of controls that you have in place, and probably most important, you need to make sure that your authentication processes that you have with them are really, really locked down.
Chris Goettl: Right. So there was another, not too distant past here, very successful attack that used a similar method to get into an environment, if you remember NotPetya.
Phil Richards: I do.
Chris Goettl: The attack on the country of the Ukraine that had a pretty widespread collateral effect as well. With that attack, the threat actors there had a very interesting way to get in, too.
Phil Richards: They sure did.
Chris Goettl: They infected the tax software. Basically, you had two choices in the country of the Ukraine. You had to pick one or the other, so these guys picked one of those and they infected a software update that actually went into the environment months before to pre-stage the attack. So, looking at this, it's important to understand. And the reason why we're emphasizing this one is, again. Understanding how a threat actor can get into your environment. Now we need to worry about not just our environment, but the other things that have access to our environment.
Phil Richards: The interesting thing about those other environments is, you don't get to control somebody else's environment. What you can do, and what you need to do, is make sure that number one, they have policies and procedures in place so that they're controlling their own environment. And number two, that you can receive an audit from that company that indicates that they're actually doing what their policies and procedures say they're supposed to do.
Chris Goettl: Right. So, let's talk a little bit, you know. So we've got this large-scale ransomware attack. 23 agencies in a simultaneous attack. So, the attacker, they found their way into this environment. They found a vector to get in. They spread themselves out through the environment. The course of spreading yourself out there is going to be a combination thing.
Chris Goettl: As you look at the right side of our screen right now, you're going to see a variety of recommendations. Let's talk a little bit about, why are those recommendations there.
Phil Richards: Yeah. So, a lot of these recommendations speak directly to the types of things that took place in this attack, assuming that the right kind of things had been put in place, would have either mediated or mitigated the extent of the attack, or would have stopped it dead in its tracks.
Phil Richards: Number one we have up here is backup and recovery. Now, we'll talk about this a little bit later, but backup and recovery is not the be-all, end-all for ransomware attacks. It's an important capability. It can give you a little bit of insulation around the impact of a ransomware attack. But it certainly is not the holy grail.
Chris Goettl: Right. So backup and recovery won't stop a ransomware attack from happening.
Phil Richards: Nope.
Chris Goettl: But if I'm going to have any hope of recovering without paying the ransom, I need a good backup and recovery system, right?
Phil Richards: Absolutely.
Chris Goettl: So, that's something that we do want to stress. That's not going to prevent the attack. But obviously, if you want to be able to recover from incidents like this, you need to have a good backup and recovery system. And you need to also test that process on a regular basis, and make sure it's functioning.
Phil Richards: Can't overemphasize that enough, I think. Testing your backup and recovery is critical. A lot of organizations have a backup plan that goes untested, and then when they need it, the logistics and the mechanics of recovering thousands of files across hundreds or thousands of servers and workstations ends up proving to be very, very challenging.
Chris Goettl: Right.
Phil Richards: Next on our list is multi-factor authentication. This is particularly important when you have outside organizations like an MSP or like some other third-party entity that needs to be granted access into your environment, or if you have workers that will be performing work, and especially administrative work, off-site.
Phil Richards: Multi-factor authentication just means that weak passwords are, even though they [inaudible 00:12:49] support.
Chris Goettl: Right, so it's not a secret that passwords are a human weakness. We can only remember so much. I don't know how many accounts you have, Phil, but I think I've got probably at least 30 different accounts that I really need to use on a- [crosstalk 00:13:10]
Phil Richards: And you're using [inaudible 00:13:11] password for all of them, I bet, though.
Chris Goettl: Oh, absolutely. And it involves- [crosstalk 00:13:15]
Phil Richards: Probably. [crosstalk 00:13:16]
Chris Goettl: No, not at all. But it's one of those things where how many of those accounts can you remember?
Phil Richards: Exactly, yeah.
Chris Goettl: So we need to help the human element out here. We can't trust the user to be more diligent like more of us security-minded people, where we do try to keep a broader variety, stronger passwords, better variety of them. Things that are hard to remember, even. So, two-factor authentication helps to solve part of the human weakness in privilege management and access management.
Phil Richards: As a matter of fact, password weakness and human weaknesses is one of the things that's really addressed in the National Institute of Science and Technology's latest recommendations around passwords. One of the problems is, we all have heard the advice that we should have passwords that have lots of complexity to them, where they have asterisks and hash signs and numbers and capital letters, and all that kind of stuff.
Phil Richards: Well, those become really, really hard to remember. So we, as humans, Chris. What do we do when we have something that's really hard to remember?
Chris Goettl: We write it down.
Phil Richards: We write it down and put it on a sticky note and put it on our computer.
Chris Goettl: Yeah.
Phil Richards: So the latest guidance from this is to try to move away from that. And what they're saying is, just have a longer password that isn't already cracked, but that doesn't have all that complexity that makes it hard to remember. They're stressing length over the complexity factors for that very reason, because as humans, we're fallible, and it becomes more difficult to remember those short passwords that have all kinds of gobbledegook and stuff.
Chris Goettl: Well, Phil, I came across some great advice about passwords just a few weeks back. And this one's definitely not mine. I did find it out there somewhere, but I thought it was extremely useful information. And that is, you know, passwords are like underwear. Make them exotic, keep them private, change them frequently.
Phil Richards: There's a lot that goes into that comment, Chris, and I'm not going to ask you about your underpants. I don't think that that's part of the job.
Phil Richards: The other thing that I'd heard, actually, is that always use your dog's name in a password, and if your password has to expire, then you need to rename your dog.
Chris Goettl: Oh. Love that.
Jared: Guys, what do you guys think about password managers? Tools that help you manage multiple passwords?
Phil Richards: Jared, that is fantastic. Password managers, depending on the ones you choose, can end up being a great way to be able to provide quite a bit of variety to your passwords, generating very complex passwords, and you as a human being don't have to remember, as you said, 40 passwords that are really complex.
Chris Goettl: Absolutely. And that's something that, especially for your IT organization and administrative passwords and service credentials and things like that, it's critically important to use a service like that. You know, we can't obviously make a technology like that available to all end users, so that's where two-factor comes into play. [crosstalk 00:16:21] Technology that's more easily usable for all of us day-to-day users who can't remember our own birthdays at times.
Chris Goettl: But yeah, absolutely. You should definitely look into having a secret vault of some sort to store those critical credentials that give you access to things that can actually even expire and give you a new credential or session token just for the window that you want to use it for, and then it becomes invalid again. So different things like that, or especially critical gateways into your environment or resources that would provide [inaudible 00:17:01] access to your environment, it's good to get to more and more strict levels- [crosstalk 00:17:07]
Phil Richards: That's a control that we're using internally here at Ivanti for our super-critical functions, where basically administrators have a one-time password, and they have to check it out of a vault, and when they check it back in, the password rotates on those accounts. And so it's absolutely an important component of good password management.
Phil Richards: Multi-factor authentication and password vaults. They're very strong, and they absolutely would have stopped this attack dead in its tracks.
Chris Goettl: Yeah, the window that they used to gain access to these 23 agencies would have been thwarted by having those types of preventative measures in place.
Phil Richards: Let's talk about patching.
Chris Goettl: Yeah. Once they got into the environment, once I get a foothold, the next couple of things here that I need to do is, I need to spread out. So I've got a single machine in 23 environments. Well, that's not going to be enough to make you pay $2.5 million, is it?
Phil Richards: Sure isn't.
Chris Goettl: So, I need to spread out.
Phil Richards: And I need to hit thousands of machines and be able to lock hundreds of thousands of files, because that's how I get to be worth $2.5 million.
Chris Goettl: Exactly, so- [crosstalk 00:18:10]
Phil Richards: And the way these guys did it is through CVE exploits. Critical vulnerabilities and exploits. The interesting thing about this, Chris, and make sure I'm not going off the rails here, is they used pretty current vulnerabilities in these attacks. Is that right?
Chris Goettl: Absolutely. So, whether a vulnerability is three years old or three days old, there are certain vulnerabilities that are going to be a higher risk to us than others, and threat actors can move pretty quickly on exposing these. So these next two, patch vulnerabilities and restrict admin privileges, are kind of used in concert to do this next stage.
Chris Goettl: I need to figure out the system that I landed on. I've got access to one system. Am I an admin yet? [crosstalk 00:18:53]
Phil Richards: Not even close.
Chris Goettl: Do I have the level of privilege I need? Okay. So I'm going to use a vulnerability exploit to elevate my privilege level, yeah? Now, once I've got that privilege level, I've got a whole slew of capabilities available to me on this local system, including the ability to use interesting tools like [inaudible 00:19:12] or gain access to additional credentials that have been used locally and could still be valid.
Chris Goettl: If I can get my hands on those, now I can start to spread out through that level of privileged access using the identity that you expect, and the tools that you expect.
Phil Richards: In the cyber security world, we call that pivoting, and that pivoting is critical to being able to... That's the expand part of land and expand, in an environment. Once you get into an environment, that's not the only thing that the criminals are trying to do. They're trying to make sure that they can pivot into more and more systems, elevate privileges, take advantage of unpatched services that are running on those systems so that they can get access to even more and more systems, more and more authentication of power.
Phil Richards: Unfortunately for a lot of our organizations, we tend to spend an awful lot of time on perimeter defenses, which is really important, but once those perimeter defenses are breached, we kind of have this soft underbelly of relatively weaker privileges that allow the criminals to pivot or expand into the environment.
Chris Goettl: Right. In this case, the perimeter was defeated as soon as we found that door in through a trusted vendor. Once we got in there, though... So, one thing that is an interesting question to ask is, what is your kind of hack? What is your ability to close a vulnerability across your attack surface?
Phil Richards: And that is a critical question. What we're seeing now is that we're seeing attacks that are exploiting vulnerabilities in the 90 to 120 days old range. We occasionally see attacks that are exploiting vulnerabilities that are in the 30 day range, which is really, really frightening. That's just about as frightening as a zero-day attack at that point.
Chris Goettl: So, if you look at statistics from the Verizon data breach investigation force, over the last few years here, they've had different ways of looking at this. But one of them that was very interesting was around the average age of an update by the time it gets exploited.
Chris Goettl: So if there were 100 vulnerabilities that we're looking at, and 10 of those were going to be exploited, within the first 14 days, you might see a couple of exploits happening there. When you get to 14 to 28 days, 50% of those are going to be exploited.
Phil Richards: Wow.
Chris Goettl: And then, by the time you get up to that 90 day range, 90% of those. So nine out of 10 of those that were going to be exploited would have been exploited- [crosstalk 00:21:59]
Phil Richards: Would have been exploited by then.
Chris Goettl: The reason why they can do that is, there's some threat actors who are going out and they're breaking down and trying to hack code that exists today. Those are the zero-day vulnerabilities that we see exploited from time to time.
Chris Goettl: There's a second tier of assessment that they do. So when [inaudible 00:22:19] system update, they've got the ability to do a differential. "I know what the code was. I know what the code is now." They have tools that can de-compile and then compare those two, and they can go line-by-line through the code and see what was corrected. And the smart guys know how to then go and deconstruct that and create an exploit, take advantage of that.
Phil Richards: Wow, okay.
Chris Goettl: And what they're relying on is our our inability to close those gaps quickly. So, if they can do that and in two weeks, they can have an exploit from the time that an update is released from Microsoft or Google or whatever. They can have an exploit created and be able to hand that off. That gives them an ability to move pretty quickly.
Phil Richards: You know, Chris, what you're talking about, the ability to accurately assess an organization's vulnerabilities with those types of attacks is really at the heart of what Mitre has put together with their attack assessment framework. The focus of that science is really around digging into what the criminals' behavior is, what vulnerabilities they're looking to exploit, and rolling that out on a per-industry basis, or vertical basis and that kind of thing.
Phil Richards: So, absolutely true. There are tools that can help our customers get some insight into that in terms of their own vulnerability and risk, but essentially, at the end of the day, the story is always going to be, patch your systems as quickly as possible.
Chris Goettl: Yes.
Chris Goettl: So, let's move on to vendor risk management, and then I think we want to jump to the next topic here.
Phil Richards: Yeah. So, vendor risk management is all about making sure, as we stated a little bit earlier, that your vendors are kind of working around the same sort of activities that your company handles in terms of vetting employees, employee management, termination of employees. Making sure that they don't have access rights after they're terminated. There's a whole litany of security issues that you want to make sure that your vendors are using.
Phil Richards: There is a common framework for gathering that information. It's from an organization called Shared Assessments called the Standard Information Gathering or SIG template. So, if you've heard of the SIG Lite, or the SIG template, that's really a set of questions that kind of asks vendors, are they holding serve with their own security protocol and requirements within their organization?
Chris Goettl: And so in my role as heading up product management for our security products here, our customers, and many of you on the phone here, are demanding more often those type of SIG Lite assessments and certifications of our software, whether it's on premises or in the cloud. It's good to ask those questions.
Chris Goettl: For vendors like us? Yeah, we've got to do more diligence to make sure that we're doing those things, and you need to keep us accountable for it.
Phil Richards: Absolutely.
Chris Goettl: We've got to do it. [crosstalk 00:25:20]
Phil Richards: And we're happy to do it. What we don't want to do, is we don't want to do a one-off for every single one of our customers. We've got thousands of customers. [inaudible 00:25:29] Which is really nice about the Shared Assessments template, is that it's a standardized set of those questions. I can provide the same answers to essentially everybody who asks those questions of us.
Chris Goettl: Right, right.
Chris Goettl: All right, let's transition.
Phil Richards: Yeah, let's move on. So, school's out forever. This is a nod to the old Alice Cooper tune from a few years back. Chris and I are old, so for those of you who don't get that, that just means that you're too young to understand.
Phil Richards: But this was an attack on four school districts in the state of Louisiana. There's a couple things that were really interesting about this attack. The governor of Louisiana declared a state of emergency, which gave him access to funds and capabilities, such as being able to call out the National Guard.
Phil Richards: You know, when we first looked at this now, we thought, "National Guard? Isn't that a military group?" We were kind of interested. It ended up being a kind of an interesting thing.
Phil Richards: This isn't the first time that that's happened, by the way. But as it turns out, the National Guard is made up of weekend warriors. Most of them are part-time. Most folks in the National Guard are part-time, and they have day jobs, and there is a whole group of IT security and IT specialists that work for the National Guard on the weekend.
Chris Goettl: Right. And over the years, I can even count several individuals that I knew who were in the National Guard who were also in our IT organization. So it was a very smart move to be able to pull from a pool of resources that would be well-equipped to deal with this type of crisis.
Phil Richards: Quick access to a lot of intelligentn people. The reason why that's important is because as we said earlier, a ransomware attack of this nature, demanding these kind of ransoms.
Phil Richards: What that means is, you've got hundreds of thousands of servers and workstations that have already been compromised, and you have thousands to hundreds of thousands of files that have been encrypted. There's a lot of busywork in terms of being able to recover those systems. It's not just a matter of, ask nicely to the bad guys and they'll give you the key to unlock your files. Oftentimes, systems have been compromised that are no longer workable.
Chris Goettl: Yeah, one other thing that was interesting about this one is the timing. So, there was a reason that it was happening right before the school year was about to start.
Phil Richards: There is? Why would you do that?
Chris Goettl: Well, you know, the teachers are all coming back in and they're preparing all of their syllabuses and their lesson plans, and IT is making sure all their devices are working, the labs are set up, and every teacher's got what they need. [crosstalk 00:28:15]
Phil Richards: And then all of a sudden, everything on the IT side goes dark.
Chris Goettl: Right. Timing- [crosstalk 00:28:21]
Phil Richards: Every file is blocked.
Chris Goettl: Timing.
Chris Goettl: It's interesting, you know, as a marketer. Product management, product marketing. It's kind of the place where my brain spends most of the time.
Phil Richards: And I tend to think of you guys and cyber criminals as kind of the same thing. [crosstalk 00:28:35]
Chris Goettl: Thank you, Phil. No.
Chris Goettl: But when you're thinking about monitoring [inaudible 00:28:41], you want to maximize your effectiveness. There's a few things you're looking for. You want a broad target. You want to be able to message to as many people as possible. In the threat actor world, you want to be able to attack as many people as possible, or as broad of an area as possible. You have to create a sense of urgency.
Phil Richards: [crosstalk 00:28:59] Yes, you do.
Chris Goettl: In this case, the timing around this was the sense of urgency. If we can bring-
Phil Richards: That's right. [crosstalk 00:29:04] The school year starting in the beginning of September, and this thing happened at the beginning of August. That's a month. You've got a month to figure it out.
Chris Goettl: It didn't make it to the point where there was no way they could even pay out in time, because the attacker in this case was thinking through this. If I'm reading this correctly, they thought about the fact that to get approval for a payout, they're going to need a little bit of time. It's going to escalate through a series of people before that happens. So we've got to give them a window, but we've got to make it close enough where they see a sense of urgency. They know the school year's about to start, and for them to get back up and running before students coming walking in the door on the first day of school-
Phil Richards: That's a challenge, but it's doable. They can see that as an objective.
Chris Goettl: Exactly. So I've got the perfect opportunity. Now I've just got to figure out a way in.
Phil Richards: Well, if you're a bad guy, that is absolutely brilliant. Because what you're doing is you're saying, look, let's take it away from, "Should I pay the ransom or not?" and get to "I've only got 28 days to get this thing fixed. If I pay the ransom, then that shortens my timeframe by a couple of weeks." Interesting. That's amazing.
Chris Goettl: So, now to get into the environment. These guys, they didn't rely on a third party to get them in. They used a few different tactics. They had some phishing tech being used here. That phishing scheme put a remote access trojan into these environments.
Phil Richards: Well, let's talk a little bit about exploit kits first. The phishing scheme oftentimes, as you know, what we do here is why try to provide phishing training to our employees to not click on links that they receive in email. That if you have to go to an email address, go to it manually. Don't click on it from a link in a email. That's the basic advice.
Phil Richards: Unfortunately, we've seen this. We do this all the time. We click on [inaudible 00:30:48] that are in our email because it's a shortcut. It takes us there. And most of the time, it takes us to the right place.
Phil Richards: Sometimes it takes us to the wrong place, and that's where the exploit kit comes in. Basically, there's a website that has some software running on it. What it does is basically opens up a mini Nessus scan of your machine, looks and finds out what things aren't patched, and basically builds a compromiser or an exploit specific to what's not patched on your workstation. That's what an exploit kit is, that's what it does. And then it opens up the machine so that the bad guy can put a RAT on the machine.
Chris Goettl: Right. And that RAT is that back door that now, I've got a foothold. I can keep coming back in without having to re-phish a user.
Phil Richards: Yes. That is really important for this particular attack, because as Chris mentioned earlier, the timing was critical. We want to orchestrate this so that a thousand or ten thousand machines go down at the exact same time. And I do that by having a remote access trojan running those machines [inaudible 00:31:51] orchestrated attack and deploy it basically simultaneously to all the machines.
Chris Goettl: But now once I've gotten into the environment, they're using a lot of the same tactics you see in our recommendations there. We've got patch vulnerabilities and privilege management again. This is where I'm going to figure out on the system I'm on, do I have the level of access I need to spread out more? Do I have other vulnerabilities to exploit? Can I get a higher privilege level? And from there, they're going to start to pivot and spread out through the environment again.
Phil Richards: Absolutely, yeah. And really what we're seeing is a lot of the same kind of mechanics as you mentioned that we saw in Texas after the original attack.
Phil Richards: So, we've talked a little bit about Texas, we talked a little bit about Louisiana. We've got a slide here that talks about the fact that we're seeing this at a lot of different local and education and state government entities throughout the United States, and in fact, throughout the world. You can see some of these entities have paid the ransoms. Some of these entities did not pay the ransom, and different things like that. And it- [crosstalk 00:32:56]
Chris Goettl: Yeah, now we can talk about some of the financials of this. We've got a payout there for Tallahassee, $498,000. We've got the Georgia one that the cyber criminals got paid $400,000 there. The ones in Florida, within basically, what was it? A week? Two week span, maybe? Within a week, over a million dollars was paid out across two different Florida cities. There was $600,000 from one city, and another $400,000 from another one, all within quick succession.
Phil Richards: So, this is amazing, and it's a little bit different. I mean, we've always talked about the fact that you shouldn't pay the ransom. Let's talk a little bit about ransom mechanics, because this is important. A lot of these entities feel like they don't have any choice but to pay the ransom. And one of- [crosstalk 00:33:48]
Chris Goettl: And if you don't have a good backup and recovery, you will definitely do that. [crosstalk 00:33:52]
Phil Richards: Well, certainly. Certainly. But that's not the only problem, right? What's the problem [inaudible 00:33:57]? Let's talk a little bit about the problem with the [inaudible 00:33:58]. Let's assume that all these organizations had a backup and a recovery strategy. As soon as I get on a system and orchestrate the locking of hundreds of thousands of files. Tens or hundreds of thousands of files. Plus, I've got ransomware running on thousands of those. What do I have to do to recover that?
Chris Goettl: There's a lot of things that [crosstalk 00:34:19]-
Phil Richards: Lot of busy work.
Chris Goettl: If you're a city, so let's go to, there's a couple of them. So, we may find out the overall economic cost of some of these attacks down the road here, but a few of them that happened in the past year, the city of Baltimore. If you look at the attack that happened to those guys, the total expected cost over that recovery from that ransomware attack reached $18 million.
Phil Richards: $18 million dollars.
Chris Goettl: The city of Atlanta last year, the SamSam ransomware attack that hit them? It hit six departments throughout the 13 or so departments across the city of Atlanta. The hard cost for that reached $2.7 million. This was to get services like 911 and other things back up and running in short order with temporary services. Also, to hire in specialized security services through third-party service providers who would come in and help do the recovery.
Chris Goettl: That was just 2.7 in hard costs. Then there was the total amount expected to reach about $17 million in additional soft costs. And that was the fact that the judicial system went back to pen and paper for over a month, because all the systems they would've been- [crosstalk 00:35:35]
Phil Richards: Were compromised and hadn't been recovered yet.
Chris Goettl: Exactly. They had city officials crammed around laptops they pulled out of closets that were retired systems and just got working. There was an image in one of the news articles that had three city officials working on the same laptop, having a meeting, because that's all they had to work with.
Jared: So, question real quick, then, coming in is, is ransomware insurance... I mean, do you get it? Or do you just shore up your defenses?
Phil Richards: So, ransomware insurance. And by the way, cyber insurance has been used in a lot of these cases to pay ransoms. So you can kind of call it ransomware insurance.
Phil Richards: What ransomware insurance does, or paying the ransom, what that does, is it kind of gives you a shortcut to some of those things, but you're dependent on the good graces of the criminals that are ransoming you in the first place to kind of hold up their end of the bargain. To unlock your files.
Chris Goettl: Right. And whether it's because they really just didn't follow through, or for other technical reasons, because in this case, it's technology. It could fail in the process as well. Not also ransoms paid actually result in your systems being freed back up. What is the stat on that?
Phil Richards: Lately, the stat has been about 26% of the ransoms paid actually result in getting your files back restored, and short-cutting some of those recovery processes.
Chris Goettl: I don't know about you, Phil, but I'm kind of frugal about my gambling.
Phil Richards: 26% is not a very good payout.
Chris Goettl: Yeah, 26% chance on half a million dollars? [inaudible 00:37:06]
Phil Richards: It's a risk. But let's talk through the mechanics of recovering systems, which is what really leads to the reason for people paying these ransoms. If the bad guys have thousands of systems, or hundreds of thousands of files, the first thing I need to do is I've got to re-image all those systems, right?
Chris Goettl: Right.
Phil Richards: Why do I have to do that? The systems haven't been turned off. In fact, the bad guys are in control and that's the problem.
Chris Goettl: Yeah.
Phil Richards: Bad guys are in control.
Chris Goettl: So we've got to contain the environment first, and then we've got to go and re-provision all systems, so- [crosstalk 00:37:39]
Phil Richards: So it doesn't make sense to restore the files first.
Chris Goettl: Right.
Phil Richards: Because?
Chris Goettl: The system's still compromised.
Phil Richards: The bad guys'll just re-encrypt them.
Chris Goettl: So we've got to re-provision the system first. So, do we have the ability to do that? That should be part of your recovery plan. Do we have a quick and easy means to recover? Whether it's Microsoft Autopilot or provisioning. For those of you on the call who are end-point manager customers, you've got a great provisioning platform within there. Make sure that you've got the ability to quickly re-provision the system.
Phil Richards: And that you test it.
Chris Goettl: And that it's tested, yes, absolutely. Now, if you're using a provisioning service, typically that's every new system that goes out the door, so that part of it is probably more hard. The bigger part is, once it becomes time to restore the user profiles, the files that need to be on there, everything else? That gets used a lot less frequently, and we've got to test those more often [crosstalk 00:38:34] to make sure they're working.
Phil Richards: You have to check. Because a system that you just recovered to bare metal doesn't really do you a lot of good. You still have to put users on it. You have to put user configurations, user files on there. And all that has to happen before you start to restore files.
Chris Goettl: And there was a ransomware attack that happened not so long ago here to Arizona Beverages that was a good example of that. They had backup and recovery systems in place. What they found, though, is they were not functional. They were not working. So they couldn't recover, even though they had those processes in place.
Phil Richards: Oftentimes, these organizations that are hit with these very, very large attacks are finding that even if they have backup and recovery systems in place, they can't be brought up quick enough to mitigate the damage.
Phil Richards: And so, paying a ransom, if it's a half million dollars versus an $18 million cost to re-image everything. If I can get these systems up and running for a couple of weeks for half a million dollars, that's actually not... Even if it's a 26% chance, it might be worth it. It was worth it to some of these municipal powers.
Chris Goettl: Yeah. So, the financial ramifications of this definitely were attractive enough to several of these.
Chris Goettl: Now, let's talk about the other side. Let's talk about those who are saying, "No, we're taking a stance. We're not going to pay out."
Phil Richards: So, the US Conference of Mayors adopted a resolution just a couple of months ago that said as a group of mayors for cities of populations of 30,000 or more, we are not going to pay ransoms.
Phil Richards: We talk a lot about why companies and organizations might pay ransoms. The reason not to pay the ransom, of course, is that [inaudible 00:40:16] they're criminals. The only reason they're in this game is to get money, and if there's no pot of gold at the end of the rainbow, then they're likely not going to spend the time and effort and money to make the attack.
Chris Goettl: And in that conference, that special session there, there were 227 mayors who attended that one, and before that was wrapped up, 225 of them had signed that resolution to not pay the ransom. So that kind of unified message is important.
Chris Goettl: It's one of those things. It's all about enablement. If we're encouraging bad behaviors, it will continue.
Phil Richards: Exactly.
Chris Goettl: If we show a propensity to pay out, that's going to make it so threat actors will continue to get money there, and they won't just move on to something else. So the more payouts that happen, it propagates.
Chris Goettl: So there's a risk in doing a payout. There might be a short-term gain of getting back up and running faster, but the long-term effect is, we make it worse. Ransomware for a while there was on a decline, but the threat actors took a step back out and they thought through better ways to do it. One particular- [crosstalk 00:41:32]
Phil Richards: To now make it more efficient.
Chris Goettl: Right. Well, one particular group that made a really big shift in how ransomware attacks were being done was the SamSam threat actor group. They were getting about $330,000 a month over the course of two years in operation before a couple of their ringleaders got taken down at the end of last year. They'd netted over $6 million in ransoms.
Phil Richards: That's amazing.
Chris Goettl: A lot of their focus was on healthcare, but they were the ones behind Atlanta, and they were targeting public sector as well.
Phil Richards: When it comes to paying ransoms, one of the things that kind of makes it a little easier for organizations to pay ransoms is the fact that they've got cyber insurance, and cyber insurance funds can be used to pay a ransom. So we do have that listed here. The question is, is that a good thing that cyber insurance can be used to pay ransoms? Or is it a bad thing?
Phil Richards: Certainly [inaudible 00:42:23] the bad guys.
Chris Goettl: And this is one where, on this topic, we are giving you our opinions, as people who are thinking about this space more often and the motives behind it, and what it enables threat actors to do. But obviously, there's bigger variables when you're the one that holds the decision that has to be made.
Phil Richards: We recognize that this is a very complex issue. For some organizations, an attack might hit at exactly the wrong time and exactly the wrong group inside your organization, and you might not have a choice but to roll the dice on that 26% chance.
Phil Richards: By the way, when you pay the ransom, you also increase by a factor of 10 your likelihood of being hit again for another another round.
Chris Goettl: Oh, yeah.
Phil Richards: So, you've got that to look forward to.
Chris Goettl: If they know you're willing to pay out, you might be repeat target.
Phil Richards: Yeah. So, we talked a little bit about insurance. We talked about backup and recovery issues, and we talked about [inaudible 00:43:27] individuals who have the technical expertize to re-image machines, to do recovery and restore activities to make sure the machines are in good working order. And when you're talking about thousands of machines, you need thousands of guys, in some cases.
Chris Goettl: Absolutely.
Phil Richards: All right, let's move on. We're moving a little bit past ransomware now to a breach, and we're calling this the Capitol One breach, but that's probably not fair. Of course, our tagline is who's in your wallet?
Chris Goettl: This is more of a, this was the one that caught headlines the most, but there were actually other companies that got hit by this as well.
Phil Richards: Yeah. There was a couple dozen at least potential company names that were listed on a person's page of compromised files.
Phil Richards: So, 106 million not dollars, but actual credit cards and social security numbers, were impacted. The interesting thing about this is, as far as we can tell, this person hadn't had a chance to monetize-
Chris Goettl: Correct.
Phil Richards: This particular breach. [crosstalk 00:44:42]
Chris Goettl: They were able to get their hands on it, but they hadn't found a way to actually sell it yet and release it out.
Chris Goettl: So, the interesting thing about this. This is not a ransomware attack. There's an argument of, was this an insider threat or not?
Phil Richards: Interesting.
Chris Goettl: So, the perpetrator in this case was not a Capital One employee, but a prevous employee.
Phil Richards: Former employee, yeah.
Chris Goettl: Former employee of a platform that Capital One- [crosstalk 00:45:14]
Phil Richards: A cloud service platform that Capital One was using.
Chris Goettl: But because of this specialized nature of information that this person had access to, they knew how to get into this environment.
Phil Richards: So, did this person have access to secret words and passwords and accounts and things like that, that they should not have had after they were discharged from the company's...?
Chris Goettl: Absolutely. But it was one of those things where, so think about when you off-board employees. This is one of those things that on-boarding and off-boarding is a very important part of getting a user into your environment, getting them out of there. It has a lot of security implications as well. As you release a user from your employment, you want to make sure that you've taken all access away from them.
Chris Goettl: Well, the piece that was missing from this, they may have taken away- [crosstalk 00:46:05]
Phil Richards: Access.
Chris Goettl: Personalized access that this person had. But there were a few things that this person knew about that weren't accounted for.
Phil Richards: I see. So, we're talking about knowledge about default account names and passwords. We're talking about privileges that maybe aren't... This person might not have had an account in to get onto these systems directly, but because of the knowledge that they had, they know where some of the weak points were.
Chris Goettl: Correct.
Phil Richards: Okay.
Chris Goettl: And because [inaudible 00:46:38] information, there was a default configuration in these devices across, actually, more than Capital One. Like you said, it was dozens of companies that had the same device within their hosted environments.
Phil Richards: Now, let's get down a little bit into the weeds with this. Not too much.
Phil Richards: A lot of companies used Apache as a web server. Apache's a great web server, but one of the things that you need to know about if you're going to deploy Apache is that by default, it comes configured with all of the lights turned on. With all of the mods configured to run. You need to be shutting down mods, as an organization, that you don't plan to use. The exploit that took place actually was through one of these modifications, or mods, to Apache.
Chris Goettl: Right. And the common piece between all of these environments in this cloud provider's environment was that they all used the same device. That was a virtual device that was pushed out, running Apache. A mod that was not being used by any of these. So, basically, they didn't go in and change the default configurations on it, because they weren't- [crosstalk 00:47:49]
Phil Richards: Because they weren't using it.
Chris Goettl: Right. It also wasn't turned off. So it was there, accessible, and had a default configuration which had a credential that allowed this person to gain access through that.
Phil Richards: Frightening stuff.
Chris Goettl: Exactly.
Phil Richards: Really frightening stuff. And that's why at the top of our recommendations list you'll see configuration management. If you're deploying Apache, and Apache particularly, you need to go through the hardening guidelines that are available on the web all over the place, from Apache and from other organizations as well, to be able to kind of lock down Apache before you start to use it.
Chris Goettl: Right. And this particular device, we actually went and looked at Apache's site as we were investigating this one, and they did have some hardening guidance on that. They had even a list of certain modules that they highly recommend turning off right away. This one in particular wasn't on there. The mod security module that was exploited here. But I would expect that it probably will be added to that hardening guide over time.
Chris Goettl: The key there, though, is that they had four or five mods that they highly recommend taking out, and then they recommended assessing any other mods to determine- [crosstalk 00:48:59]
Phil Richards: Basically if you're not using, turn it off.
Chris Goettl: If you're using it, you need to look at its configuration and change any defaults that would allow somebody to gain access to it without having to really expend any effort.
Chris Goettl: If you're not going to use it, you need to disable it.
Phil Richards: Yeah. If you're looking for some empirical guidance around hardening for some of those things, look to CIS Benchmarks. CIS Benchmarks actually provide a horrific amount of detail around a lot of these different software components about how to harden those things to your best advantage.
Phil Richards: So, configuration management, critical. That's an important part.
Phil Richards: Privilege management, again. This goes back to what we talked about earlier about pivoting. This person was able to get into an environment. That's nice, but it's not enough. They have to then be able to use exploits to be able to elevate their privileges. And managing those privileges is critical.
Phil Richards: Again, vendor risk management. In this particular case, the vendor is a known, good cloud service provider.
Chris Goettl: Very highly noted.
Phil Richards: Yes.
Chris Goettl: Very popular.
Phil Richards: And at the same time, you still have to vet your vendors. Your vendors have to be good to go.
Phil Richards: Need-to-know access. That's a concept that we in security use. Access appropriate to role. You should provide enough access for individuals to do their job, but no more than that. [crosstalk 00:50:22]
Chris Goettl: Right, and this is one of those things. I'm the kind of employee that's probably a security guy's nightmare, Phil. I've been with Ivanki for nearly 15 years. Actually, not even just Ivanki, but I started with a little company that got acquired by a much bigger company, and then I got acquired by another company after that. So- [crosstalk 00:50:45]
Phil Richards: And you have all that access still, don't you?
Chris Goettl: Well, I won't say.
Chris Goettl: But throughout that time, I worked in our support team. I worked in our pre-sales team. I worked in our product teams, and now I'm a director of product management. Think about all the different permissions that I've had over the years. All the access to different systems that we've had throughout there, even for companies that I no longer work for, because the products that I was with spun out into a different entity, so do I still have access to the things at that other one?
Chris Goettl: So, former employers that I worked for there have to worry about me as well, and then the new employer has to worry about- [crosstalk 00:51:23]
Phil Richards: Chris, you're bringing up a really good point. When somebody changes jobs, the IT team's responsibility is to make sure they do not have access associated with their former job, and only have access associated with their new job.
Chris Goettl: Absolutely.
Phil Richards: Oftentimes what we do here is, we remove all access from that employee, and then re-provision all that access in order to make sure that that stays the way it is.
Chris Goettl: That's a good, clean way to do it, and having an identity access management solution like- [inaudible 00:51:52]. We do have [inaudible 00:51:54], absolutely. And having that makes it so that a lot of that can be automated, and human errors won't happen.
Phil Richards: Okay, that's it. We went a little bit longer than we thought we were going to. We really want to thank everybody for your attention to this. Jared, do we have any questions from anybody that we want to walk through?
Jared: Sorry, guys. Just finishing up this email to the FBI about all that access that Chris apparently still has, so...
Chris Goettl: Thank you, Jared.
Phil Richards: That'd be perfect, thank you. Send it to me so I can sign onto it.
Jared: Yeah. [Ken Kemp 00:52:28] and I were chatting in chat here, and he talked about the total cost of SamSam attack in Atlanta, which records show around $2.6 million. But there's an article in March showing that the city's new CIO, they brought this guy in after the attack, said that the total cost to the taxpayers could be as high as $17 million. I mean, what are some of the long-term residual effects of an attack like that?
Chris Goettl: Yeah, so, that was one that after that one happened, I analyzed it [inaudible 00:52:56] used it in a lot of conversations. And by the way, hi Ken. Good to see you again.
Chris Goettl: But that attack? Yeah, the 2.6 was the hard cost. So, that was the security expertise they brought in.
Phil Richards: Re-imaging machines.
Chris Goettl: Emergency services, things like that. The soft cost is where it reached $17 million. So that was things like, you know. Your court appearance having to be rescheduled, and delays to transit, or paperwork for a permit, or the whole... Six departments. [crosstalk 00:53:27] These are departments in a leading US city.
Phil Richards: Yeah, so this is a major city with running operations, and there are costs with delays to those services not being running.
Chris Goettl: Yeah. So that's where those soft costs came in, and the city of Baltimore, very similar there. The soft cost for that reached $18 million, estimated. That was the cost of doing business as a city agency. All the things that they basically... Having six departments across your city be basically sitting there twiddling their thumbs, telling people who are there needing their services [inaudible 00:54:00] back. We don't know when, but come back some other day, because we can't help you today.
Phil Richards: Just not today.
Chris Goettl: Right.
Phil Richards: It'd be great if you can come back tomorrow, but we might not be open then either.
Chris Goettl: Oh, yeah, absolutely. So that's where those costs ramp up significantly.
Jared: Awesome, excellent. Thanks, guys. Just a little praise from [Ken Olson 00:54:20] right now, saying, "A lot of good information. Looking forward to the next webinar. Thank you Chris and Phil." So I have to echo that. Thanks Chris and Phil.
Jared: Guys, I think that's all we're going to be able to handle today. Any closing comments before we wrap this up?
Phil Richards: Well, just one last thing from me. The reason why we're doing this is so that you can put yourself in the position of some of these organizations that are seeing these kind of attacks. Think about your own organization, and what the impact to your company or business entity or corporate or government entity would be, if you were to have some sort of an attack like we're seeing here. A coordinated attack where all of your machines get locked up, and you're having to pay a big ransom.
Phil Richards: It's something that's happening now more and more frequently, so it's something you need to be thinking about. How well-prepared are you? What kind of processes and procedures do you need to put in place now so that you can weather that kind of a storm.
Chris Goettl: Yeah, and I would echo that. I think that again, like I said at the beginning, looking at attacks like this, one of the things that we want to make out of this Threat Thursday series is, we want to create a repository for you. So, like the three different slides you saw there, that broke down these attacks? We want to get to a point where somebody can come and basically just search for, "Show me all the profiled public sector attacks." Or all the healthcare ones, or-
Phil Richards: Or everything that had a RAT associated with it.
Chris Goettl: Right, yeah. So, a way to be able to go and get to a repository of knowledge that lets you find the commonalities. A lot of what we're doing, a lot of security, is all about looking at patterns, looking at behaviors-
Phil Richards: Finding out what's coming.
Chris Goettl: And finding out how to break those patterns or behaviors before they cripple you. So, we're trying to do the same thing there.
Chris Goettl: So, definitely keep coming back. We're going to look at not just public sector. We're going to be looking at all verticals. We're going to be looking at a variety of different attacks over time here, and trying to again, bring the right information so you can better understand how to combat these.
Phil Richards: Yeah. Looking forward to having some additional... We see a lot of new things coming out, so the one in September, in a couple few weeks in September here, is going to be fantastic, and we're really excited about it. Thank you all for attending.
Chris Goettl: Thanks a lot.
Jared: That's it. All right. I love that. We need a Jerry Springer final thoughts built-in segment at the end of this, guys.
Phil Richards: Jerry Springer?
Jared: You know, I don't know if we want to go full-on Jerry Springer. That might get- [crosstalk 00:56:59]
Phil Richards: That might get crazy. [inaudible 00:57:00]
Jared: In the studio, yeah. Yeah, we need to be careful, or at least consult with our insurance agency first.
Jared: All right. Hey, thanks everybody for joining. Look for that recording and the slides being sent to you later today, and stay tuned for the next Threat Thursday. Thanks, and have a great rest of your Thursday.
Phil Richards: Thanks. [inaudible 00:57:25]