Hacked!?! How can I Fix This Fast?
October 04, 2017
Eran Livne | Principal Product Manager | Ivanti
Amber Boehm | Manager, Product Marketing | Ivanti
Phil Richards | Chief Security Officer | Ivanti
While WannaCry and NotPetya remain open wounds for many organizations, the Shadow Brokers are back with a new stolen exploit, and Equifax just put an exclamation point on the risk to personal data. What do you do if your business gets breached?
Sit in as our CISO and Principal Product Manager dish on the state of security, and how the newly automated remediation features in Ivanti Endpoint Security for Endpoint Manager 2017.3 can help you effectively respond to attacks that do get through.
Phil: That's where we want to start. Incident management is focused on restoring operations. It's focused on immediate action, the things we have to do immediately, not to fix the problem but to restore operations. Sometimes those two things can be different. As an example, most of the time, you don’t want code fixes to be part of your incident resolution. You don't want code fixes to be necessary to restore operations. You want to do things like recycling servers, making configuration changes, and putting in the kinds of things you can do quickly to restore service, restore operations. Sometimes that means taking certain components of your operation away from your users as a method of restoring operations, at least partially. The intent behind incident management is really about rapid response and restoring operation.
We save the activities around fixing the problem―identifying the root cause, addressing the root cause, modifying our process―we save all of that for the problem management side. I'm not suggesting those things aren't important, they are important, but it's important to put them in the right order because your first and most important goal is to make sure your services are restored, so we're going to start by talking about service restoration.
Deploying to All Endpoints
One of the things you're going to want to do when it comes to restoring service is make sure that whatever process or whatever component you need to deploy, you're able to deploy across all of your endpoints. If, for example, you need to apply a specific patch to make sure intruders can't get in, you want to make sure that patch is deployed to all of your endpoints. Deploying a patch to only some of your endpoints means the bad guys will gravitate to the endpoints that don't have the patch applied. At Ivanti, we have some important cutting-edge technologies that not only allow you to take control of those endpoints, but also quickly establish a presence on those endpoints so you can orchestrate some of this work from an endpoints perspective. Let me turn it over to Eran so he can talk a little about managing endpoints.
Eran: Thanks, Phil. Managing endpoints is very important, as Phil said, because you need to make sure you can control all of those endpoints, deploy your policies, enforce those policies, and take actions on all those endpoints. In most cases, it means you need an agent running on those endpoints, not in all cases, but in most cases. Technology allows you to push to agents, or you can install them. You have different ways of deploying those agents. In some cases, you can work agentless. For example, if you want to patch your machine, specifically if it's a server, you may not want an agent running on that machine, so you can do it remotely without an agent present, or if you have VMs, you may want to patch the VMs without actually needing to run the VM itself.
All this technology is valuable for you. We believe it's important to use all these types of technology, but the end result should be that you are able to manage and control every endpoint you have in your environment.
Phil: Thanks, Eran. That's right, and one of the ways you can ensure you're taking control of all of the endpoints is through what we call discovery services. Discovery is the process of, once you have some of the endpoints deployed, making sure the software is looking around your network and identifying endpoints, whether they be workstations or servers, that maybe you didn't realize you had in your environment. Eran is going to talk a little about some of the technology around discovery services.
Eran: You cannot protect what you don't know. Our discovery technology will do a few things for you. It will go out and discover all the unknown machines, and then when you discover them, you can right-click, deploy the agent, and make them managed. We try to make it easy for you to deploy those agents, find what you don't have, and manage those agents. I will give you an example. We know you have a large environment with different subnets, and it can be deployed all over the world. Our technology is not only active going through the subnet and NMap to find all of those machines, but we can also elect a specific machine in each of your subnets and have this machine do its own discovery. We call it passive discovery because we can listen to all of the machines in your subnet and make sure we report, we discover those and allow you to deploy the agent. Again, you don't have to have an agent on all of those machines. By the way, any machines including routers and network components we can discover passively, report back to our console, and allow you to push the agents so you'll have those managed.
Phil: Thanks, Eran. To summarize, it's important as you're going through this incident management process that you're able to orchestrate whatever activities you have across all of your endpoints. That involves managing those endpoints by making sure you have either an agent or an agentless capability to orchestrate, and the discovery services to make sure you're identifying assets you might not have known about previously.
Remediation Using Ivanti Software
After you have the components installed on those workstations, there are a number of things you can and should be able to do to perform some of the remediation. If you know about vulnerabilities on specific machines, there are several things you can do through this software we talked about previously. Ivanti software allows you to do things like isolate a machine from the network, so it stops the carnage and stops the bleeding; install or update antivirus and antimalware software; and make sure your endpoints have proper configuration components deployed to them. Eran is going to talk a little more in detail about how those work with our Ivanti software.
Eran: When I talk to customers, sometimes a customer finds a breach through contact from the FBI or something. When the FBI reports, "Hey, you’ve probably been hacked," they usually give you the specific machine that may be the target, or specific IP ranges, or something you can start to work with. What customers usually do is take those machines out of the network. They may physically go to the machine, take out the cable, go to the router, disable the machine from the router, or even shut down the machine. It means you cannot work. This machine cannot work, and you cannot do more forensic because if the machine is out of the network, you cannot connect to it. You have to physically go to this machine.
“Isolate from the network” is the ability to isolate the machine programmatically from the network without the need to physically disconnect a cable or make any change on your network environment. The most important part is, while the machine is isolated and cannot connect to other machines, you can still fully control and manage this machine. You can deploy software to this machine, you can remote control, you can do your forensic analysis, whatever you need to do, but the machine is isolated. It cannot infect other machines. You can stop the breach as soon as you can.
Talking further with customers, they tell me, "We want to make sure our AV is up-to-date." Most of them have an antivirus solution. They want to make sure every machine is running the latest, the finishing set, and they're running a scan against those machines. Our product can help you, whatever AV you're using, whatever vendor you're using, we can help you make sure your software, your virus definition is up-to-date, and we can report on this, we can give you a report. It doesn't matter which AV vendor you're using, we support most AVs on the planet right now. We can help you make it very simple and speed up the process of making sure your virus definition is up-to-date.
In configuration management, you want to make sure the ports are not open, you want to make sure you don't have a demeans, all those best practices. Using our software, you can deploy those scripts, you can deploy the configuration, you can deploy SCUP, which is the standard format for configuration management. You can scan the machine and very quickly get the reports back to your SISO saying, "This is where we are. What do we need to do right now?"
Phil: By way of summary, that step is all about isolating your environment and making sure you're taking care of, with pinpoint accuracy, those elements that need to be fixed to stop the hemorrhaging, stop the bleeding in your environment. We're talking about isolating the machine from a network, making sure you're installing the right kind of software on it, and then making sure you have an appropriate configuration.
Patching, Application Control, and Privilege Management
After you've done that part of the hygiene, that part of your assessment and initial work, there's likely some additional work that needs to take place. As you're going through the discovery process of finding out how widespread an event is or how deeply threat actors have penetrated your environment, you likely will find you need to perform specific patching and some immediate privilege management. Often, threat actors get into an environment through an account that has administrative privileges that, for whatever reason, shouldn't have administrative privileges, so you might need to lock that down.
Then you have application control. Often, threat actors use what are known as backdoors, which are applications that run on a system to guarantee the attacker still has access even after you fix everything you think you're supposed to fix. Identifying those backdoors and locking them up is an important process.
Eran is going to talk a little more, from a product perspective, about how Ivanti software manages and handles these three activities.
Eran: From a product perspective, and think in the context of you've just been hacked, now what? We're not talking about best practices for patching or best practices for privilege management, we'll talk about those later. You're being hacked, now what? The first thing your bosses will want you to do is at least build reports that give an understanding of where you are. How good are you at patching your Windows machine? How good are you at patching your Java, your Adobe? Those are the main candidates for attacks to start. Those are what the bad guys are leveraging to access your environment. First of all, run those reports, get that information out there, know where you are. Use our patch.
Privilege management can give you a report that has admin rights, where you are. Application control can give you reports of what applications you are using, what is the reputation of those applications, are they unknown applications. Start first by building those reports, know where you are so you can start reacting. In most cases, you have to enforce patches. Remember that for WannaCry, there was a patch. If you were not deployed, WannaCry could spread in your system. Run those reports and apply those patches. Use our tools to apply those patches quickly and patch your machine.
Make sure you don't have admin rights. Use our tools to prevent admin rights. Build a list of known applications, or at least build a list of applications you want to block. Use our tools to learn, and then take the action of blocking those problematic or suspicious applications. All of our tools will give you an easy, straightforward way to achieve those goals. We believe simplicity and ease-of-use to get you running is very important.
Eran: Thanks, Eran. The focus is on making sure you're staying focused on the incident and not trying to patch the whole world and do everything for privilege management you could do in your environment. Assess where you are, as Eran said, and then surgically apply patches, privilege management, and application control, to arrest the advance of the attack.
The next part of incident management is to restore your assets to a working state. Previously, we've taken some of these systems offline and we might be in a position where these systems are severely compromised and we have a lot of work we need to do. This is part of incident management, so what you're trying to do first is really a triage. Make sure you're restoring systems to a working state that must be restored to restore operating services. We have several capabilities in this space that Eran is going to talk about a little deeper as far as being able to restore your environment to a working state.
Eran: When talking to customers, re-image is one of the most common ways they deal with problems. Most of us are not security experts and it's hard for us to go into the details and do a lot of forensic analysis, so if you re-image your machine, you'll most likely have a clean machine and you can start over. As Phil says, find those machines and use our tools. We have very sophisticated re-image capabilities. You can remotely re-image machines, redeploy applications, and redeploy software. We can do even better. We can learn which application each user is using, and when you re-image the machine, we will deploy those applications back again so the user can go back to a working state as soon as possible. Usually, re-image is the easiest route.
The next slide, which I took from a SANS report released recently, customers were asked, what is the most effective means for remediation? In this context, it means we know we have an infected machine, we know there is a machine running suspicious behavior, what are we going to do? You imagine they will scan from AV and let AV decide if it can clean the machine or not, but most customers are doing two things. They're isolating the machine from the network, making sure nothing is spread to other machines, and then they re-image it. Going back to working state is the most efficient and fast way to solve those problems, but it doesn't end there, right? We all know that after the machine is re-imaged, there is still work to give the user everything he or she needs.
Remember, our tools can help you isolate, re-image, and then put back all the software remotely, even migrating a user’s personal settings between machines. We can help you get your end user in a working state as soon as possible.
Amber: I'm going to interject here for a second. If we go back to the AV discussion you were having before, we have someone who would like to know why you would update the AV definitions using Ivanti when the product itself should be doing this all the way along.
Eran: You can decide what's the easiest route. If you're using the AV console, and there's something different that works with the console, by all means, use this console. If you're using Ivanti as your go-to console to manage your environment, it's sometimes easier to have one console. I call it one console to rule them all. You can have this console for full security and for full client management.
Amber: I think the question is why wouldn’t those AV definitions be up-to-date already?
Phil: That's a really good question. From a security officer perspective, I can tell you that, very frequently, users or servers turn off automatic updates on some of those things. One of the nice things about the Ivanti software is it will notify you when users turn off auto-update settings, or it can lock the user out of turning them off. You have the ability to monitor or to stop users from turning those services off. I know you have the capability of stopping users from doing automatic updates with antivirus products, but every one of those antivirus products has a backdoor, and they're quite easy to find. It's quite easy to turn off automatic updates for antivirus software products.
Even if your users need to turn those off, from a security officer perspective, you want to know what your environment looks like and what the threat landscape is. To do that, you need to know who's turned what off and the last time those definitions got updated. You can do this through the antivirus product, but as Eran said, and from my perspective, having one console where you can see not only the antimalware state, but also the patch state and configuration state all in one place is tremendously valuable.
Eran: It usually saves you time so you are able to respond faster.
Phil: That takes us to the end of our incident management process. The next thing we want to talk about, we're switching gears to the long-term process. How do we fix the environment so this kind of thing doesn't happen again? How do we go through our set of activities so we can close out this issue? These are often the types of activities senior managers, boards of directors, and large customer groups want to know we're taking care of. Not only resolving the short-term incident, but also making sure we've fixed our environment, fixed our company so this kind of thing can't happen again. Problem management is all about making sure your environment is better, making sure it's insulated from this kind of attack in the future and, hopefully, as many other kinds of attacks as you can.
Establishing a Management Process
We talked about patching in systems, we talked about making sure you're doing pinpoint configurations to systems, and that sort of thing. It’s now about establishing a management process or procedure around these things. Patch management and configuration management work on a schedule, on a cadence. You want to be able to demonstrate to the organization not only that you have applied current patches, but also that you have a process for applying those patches on a regular basis. What that regular basis is for your company depends on your company's tolerance for risk, and that sort of thing. If they're more tolerant for patch risk, they might have a longer patch cycle. If they're very intolerant, or if your industry is very intolerant of that sort of risk, you would have more frequent patch cycles.
Configuration management is all about the process of measuring what we in the industry call configuration drift. Machines have a habit of getting out of a standard configuration over time. Users install software, users make changes to settings, and over time, your machine drifts out of the standard configuration. Understanding and measuring that drift, and having a process for pulling those machines back into a managed state, is important. Again, that happens from a cadence perspective.
I want to talk a little about log management, which is doing two things. First, for all of your critical servers, making sure they are logging to a common location so you're saving those log files off, and second, using a process or application to crawl automatically through those log files and identify indicators of a breach or a compromise. As an example, a relatively trivial example, a log might show that a particular user has tried to log in to your service a few hundred times within a minute or so. That's what we in the industry call a brute-force attack, or an indicator of a brute-force attack. It's something you would want to look into to see who that person is, what they're trying to do, and figure out what's happening in that space. It's an indication that somebody might be trying to attack your environment. You would get that kind of information through a log-management tool.
Eran: I want to add a little here about patch management because Ivanti is the leader in patch management. We have a lot of experience in what it means to patch manage, what patch management is, and what the best practice is. Let me give you some advice. I'm not sure if all of you are patching or not, if you're using our product or not, but patching is one of the most important things you can do for your environment. The challenge in patching is to make sure you're patching in a specific cadence. You also have to decide what you want to patch because we all know that when you patch, applications, other applications or the same applications, may break. There is also the reboot problem that users hate. When you apply a patch, you need to reboot.
Our solutions are designed to help you mitigate those challenges. We have a better way for you to patch in terms of how you manage patches, how you apply patches in an efficient way, and how you make sure you don't have to reboot, or you don't annoy users by only rebooting when you need to. Also, the user can defer so he reboots only when it's convenient to him. Of course, you can cap it and say, "Hey, that's how much time you can defer the reboot."
When you talk about patching, for most customers, the best practice is to scan every day. All machines have to scan and report back to your core servers, so you have the most recent reporting of what is patched and what is not patched. You don't have to apply the patch, but at least know what you have and where you are on a daily basis. Most customers decide what cadence they want to deploy the patches. Some deploy only the critical patches, some deploy everything. It really depends on how aggressive you want to be with your security initiatives.
We recommend you patch at least every Patch Tuesday, every time Microsoft releases its patches. We emphasize the importance of patching Microsoft, but even more importantly, patching your browsers, Java, Flash, and Adobe products. Based on statistics, those are where the bad guys leverage vulnerabilities to hack your machines more than the Microsoft set of products, so it's very important to scan and patch for third-party applications and it has to be cross-platform. It's not enough to patch your Windows machine, it's very important to make sure your Mac, Linux, and everything you have in your environment is fully patched because the bad guys don't care which OS you're using, they will always find the easiest chain in your environment.
Phil: Okay, now we're going to move into application control and privilege management. Eran will talk a little about application control, and I'll talk about privilege management after that.
Eran: Application control is a big term, and people are using it differently. Basically, application control is managing the applications you have. We have what we call blacklisting, which is predefining which applications cannot run in your environment. It could be not allowing people to use games or Facebook or whatever. That's a good method for preventing DTorrent and all those Torrent types of things, but usually it's not enough. Whitelisting is a different approach and says which applications you’ll allow to run, and only those applications will run. If a ransomware, malware, or something bad is running, by definition it's not part of your whitelisting, so it will be blocked.
The challenge is how to manage all those lists. Between our Ivanti products, we have ways to help you make managing or keeping track of all those whitelists much easier. For example, you can say everything Microsoft is allowed. Everything that is a trusted application is allowed. Everything that runs from a trusted user, by a trusted user is allowed. There are many ways to streamline and make sure you apply whitelisting more efficiently, but application control doesn't stop with those two methods. There's a common attack right now called a fileless attack, which means the bad guys are attacking your machine by running a script without calling any file, simply by calling PowerShell. You need methods to prevent those types of attacks, because it will bypass most of your AV and other solutions.
As part of our application control solution, we have a breadth of products and capabilities that can help you protect against those kinds of attacks. I'll give you an example. There's no reason for a user to run PowerShell from the Internet. When they download PowerShell from Gmail, there’s no reason to run it, so block all those things. All those capabilities are part of our agents that manage your machines.
Phil: The idea behind privilege management is to make sure users have enough privilege to get their job done. When I say users, I'm not only talking about the people, but also about the processes. Make sure they have enough privilege to complete their task, their job, but not so much that threat actors or bad guys can get into the environment and use those accounts for things other than what they were designed for. It sounds easy, but it's very challenging to actually do in environments where you have hundreds or thousands of accounts that might be managed through Active Directory and other systems that you have deployed throughout your environment. Trying to do an assessment of each individual account can be very challenging.
At Ivanti, we have software that will make that process more streamlined. We have software that will inventory all of your accounts and the types of activities they would do under normal circumstances, and then tailor the privileges directly to those accounts. That software also has the ability to allow individuals to escalate their own privileges for specific one-time-use-only activities, or to require that an administrator escalate those privileges for the users. However you want to configure it, we have multiple ways of solving that problem. Rather than having administrators spend huge amounts of time trying to tailor privileges for specific users and user groups, that can happen automatically through Ivanti's privilege management software.
Eran: Okay. Recovery. We've talked about recovery before, but at the end of the day, we want our users to have machines they can use. If we have points of sale and other business-critical machines, we want them to be operative, and we don't want security issues to bother us, because those are the machines that make us money. It's very important to prepare for an attack. It can be a direct attack like ransomware or malware, or it can be a more sophisticated attack that is reported by the FBI, for example. Image management is the core here. If you look at CIS, which is standard best practice for security, the first thing they say is make sure you have images. Make sure you have something you can trust to start your deployment from.
If you're using our tool, you can build those images, and the important part is you can deploy them very quickly and efficiently. You spend time on building this image, the process, the deployment after you have this image, and the end result is, when you're hacked, you can very easily go remotely, re-image your machine, and have it back in a running state very quickly. That saves you tons of time and saves headaches for your users who don't have to use those backup machines everybody usually hates. Use our image technology to deploy those images remotely. We can also redeploy applications automatically, we can deploy user settings and user applications, so the user will have almost the same machine he or she had before the re-imaging.
It's important to plan ahead. You have to make sure you have all those in place, because when you are hit by an attack, you’ll want to use the tool as efficiently and quickly as possible.
That leads me to automation. We all believe that's what will make life even easier, right? I'll give you an example of how we see automation and what our product can do for you. Let's take ransomware as an example because this is something everybody can relate to. Let's say you have ransomware in your environment. Let's say it’s WannaCry. We or other tools you have in your environment can detect the ransomware. Once the ransomware is detected, we need to take action immediately because it will try to encrypt all of your network drives. It will try to find other machines in your network and infect those. You need to take this machine out of the network as soon as you can because if you do it manually, it's too late.
Using automation, you can take the action automatically. You can say, "If it's ransomware, let's take it out of the network immediately." When it's out of the network, you can still control it. You can connect to this machine and maybe even salvage several files. Maybe you can take some actions on this machine, maybe you need to run some software, some forensic software, to get more information about this specific ransomware. You can do it remotely, and then re-image remotely. So you can re-image the machine and go back to a working state as soon as you can.
Once the machine is up and running after you re-image it, we will provision it, install all of your software back to it, and install all of your settings back to it, all automatically. If you're hit by ransomware, 30 minutes later you have a running machine almost in a state where the user can start using it. More importantly, you saved yourself the hassle of fixing other machines that may be affected by this ransomware because you've isolated it as soon as you can. After creating the processes, taking action faster is very important from a security context.
Incident Response Plan
Phil: Okay. Thanks, Eran. To summarize, the things we've talked about today are part and parcel of what we call an Incident Response Plan. It's important to put together an Incident Response Plan, which is a document you then use to communicate how you handle incidents to all of the key parties in your organization. You want to do that well before you get that horrible phone call from the FBI saying you've been compromised, because when that phone call comes, everybody's in panic mode, and that's not the time to put together a plan. That's the time to execute on a plan that already exists.
If you do nothing else, my advice today would be to start by putting together a plan that divides your activities into short-term things you want to take care of and long-term things you want to take care of. Make sure you write those things down and share them with the organization so when the bad thing happens, everybody has a job, everybody knows what kinds of activities will take place, and you can start working the problem directly from your plan.
Short-term activities include discovery and making sure all of your endpoints are managed; being able to isolate and apply surgical configuration and antivirus, antimalware capabilities to those endpoints; pinpointing specific places where you want to patch and perform application control and privilege lockdown; and re-imaging and redeploying the specific components that have been infected.
From a long-term perspective, you want to make sure you have a process, a cadence around patching and configuration and log management. Make sure you have good hygiene around application control and privilege management, and make sure you're not only performing backups, but also performing test restores. A backup is an important thing to have, but too often, companies spend time learning how to and performing backups without ever testing to see if they can actually restore from their backup. That sometimes is where they get into trouble because if you can't do a restore, you might as well not do the backup. As Eran said, you want to make sure these processes are automated. Automation allows those rules to trigger automatically, and it makes your environment respond and react to these kind of activities faster, which limits the extent of the damage.
Amber: We had someone who asked earlier about user education. Phil, can you spend a few minutes talking about the importance of that.
Phil: Yes. Here at Ivanti, we have a few activities we do from a user-education perspective. I believe user education is very important. We make sure all of our developers go through a secure application development training based on the OWASP's Top-10. We also make sure all of the individuals in the company go through security awareness training, which involves a lot of different aspects of security training. We also go through email phishing recognition training, which includes periodic testing of our users. My team creates phishing emails that we send out to the user population to determine how effective our company is at recognizing malicious or nefarious emails and not responding to them. We try to make that fun. We make a contest out of it, make it a game with prizes and that kind of thing. The point is, you want to make sure your users are educated. Users who understand security concepts are better positioned to help your organization defend itself against attacks.
Amber: Okay. We have some time left. If anyone has questions, go ahead and ask those. In the meantime, I'd ask if Eran or Phil have anything else they'd like to say?
Phil: Let me talk a little about patching. One of the things we talked about is the importance of the patch management process, and I want to talk about that a little from the perspective of having worked at a regulated company. I spent some time with regulators, and we talked about our patch management process. One of the things a regulator asked me once, he asked me two questions actually when we were in an examination. The first one was: "Are you patching your systems?" We were. We spent quite a bit of time making sure we were patching our systems on a regular basis, so I said, "Yes, we're patching our systems." The second question he asked took a lot longer to answer, and it was more difficult to answer. The question was: "How do you know?" Implied in that question are other questions such as, how do you know you have all of the right patches? How do you know you're patching all of your software? How do you know you're applying all of the patches, that you have all of the machines identified that need patching?
This goes to another core competency of the Ivanti software, which is making sure you are accurate in terms of discovering your applications and your endpoints. We talked about the importance of discovery services, and this is where that comes in. You can have the ability to say, I know I'm patching all the systems because I actively go out and scan for new systems. I know when new systems come into the environment. I know when a workstation moves from one city to another because the person is traveling. I know what my environment looks like. Having that knowledge gives you more confidence to say, “I know I'm patching all my systems.”
Amber: All right. It doesn't look like we have any further questions, so I think we can wrap it up. Thank you for attending and this will be available soon. I will send you the link so you can return to it and gather more information moving forward. Thank you.