Patch Management Best Practices

May 30, 2018

Chris Goettl | Director, Product Management, Security | Ivanti

Todd Schell | Product Manager for Patch | Ivanti

Patch management is critical to reducing your attack surface and keeping your endpoints and business running smoothly. Unfortunately, it's also a process that must be repeated weekly, monthly, quarterly, and whenever critical fixes have been identified for your environment. The good news is: with the right tools and some advance planning, this process can run smoothly and leave your IT team with more time to support core business goals.

Join us to learn about trends in patch management, including the latest ways Ivanti is helping Security and IT teams work together like a well-oiled machine.

Transcript:

Chris: Good morning, and welcome to the presentation of Ivanti's Patch Management Best Practices. My name is Chris Goettl, many of you may have heard, or seen, or taken in one of our, "Patch Tuesday Webinars" from time to time. Those are hosted monthly by myself and Todd Schell who's joining me here today as well. Good morning Todd, how are you doing?
 
Todd: Morning Chris, I'm doing great.
 
Chris: Excellent, and, you know, supporting us as always we've got Erica who is helping us manage the call here today. And she's sitting there waiting, making sure that any questions that come in are answered and taken care of. And also we're joined today by our Product Marketing Manager Amber who will also be assisting on Q&A, and other related topics like that. So thanks, ladies, for joining us as well. All right, so Todd, let's get started. So this was [crosstalk 00:00:55].
 
So this was a presentation we did at the Interchange Showdown in Dallas just last week, and we wanted to do an additional run of this for those who could not make it down to Interchange itself. You know, where this was such a popular session that we actually had to run it twice. And by the volume of people we've got joining up here today, it's definitely warranted that we decided to do it again. So what we're gonna do today is we're gonna talk a little bit about the approaches to security, in general, but patch management especially.
 
These are some tips and tricks, you know, some of these are gonna be basic recommendations, others are gonna be, you know, recommendations that, hey, if you're doing this but you're not doing it to the right level, you need to take it up a notch. And we're gonna talk about, you know, some of the challenges that you may run into, some of the challenges you should, you know, start to take more head on politically even within your organization. But as we go through this, we'll kind of talk through a few of the things that will help to smooth over some of the challenges that many companies face in patching.
 
Now, Ivanti's security strategy is always based on kind of three basic principles. You need to be able to discover, provide insight, and be able to take action. And all three of these principles come into play when you're dealing with patch management as well. So these things need to be done, they need to be done well, otherwise your patching strategy will be a bit of a challenge. So let's talk a little bit about discovery and planning first.
 
So the first thing that we highly recommend is you need to make sure the right people are involved. There's obviously the immediate team that's responsible for patch management, there's going to be an extended team. So as patch maintenance occurs you're going to have to make sure that the right teams are included in the overall process. And, you know, that would be your helpdesk team, making sure that they're aware of which systems are being patched, when they're being patched, and that they're prepared to be able to support any additional volume that may occur over the course of the next days, two weeks, depending on how you run your patch cycles.
 
You want your network support teams to be in the know, you want...even if they're not directly relating to the patching itself. I hear too many times where, you know, the people responsible for the immediate patching team the day after or couple of days after patching has occurred, all of a sudden everybody starts saying, "Well, the guys...you know, the patching team was patching so it must be their fault that this is broken."
 
A lot of times people start reaching for that patch straw and saying, "No, it's because things were patched, that's the problem." Well, if you keep these teams properly informed, you know which systems were updated, what was updated on those systems, many more of those teams should start to be more aware of what was being updated, so that they don't immediately push the blame off onto the people responsible for patching.
 
You also wanna make sure that the proper business line teams are in communication as well. So any application teams, you know, business line teams that are responsible for an application layer when you may be responsible for the OS layer, you wanna make sure that those teams are part of this extended team, well aware of when maintenance will be running. What will be updated, and even participating in the testing process.
 
Management, this one can't be stressed enough. You need to have the functional and organizational managers. If I'm patching, you know, a server group and that server group happens to be where one of our critical business apps reside, I need to have the manager responsible for that critical application be involved and invested in this process. They need to understand, you know, why it's necessary, the security challenges, the compliance, and regulatory requirements. The fact that if you don't do these things effectively, there's a level of responsibility there that is on those functional managers as well.
 
And then you also want to get your C-level executive suite involved, you know, backing the patch process as well. Your CISO, or CSO in a case where you've got one, is obviously a strong proponent for this. But in the broader executive suite, it's good to have those teams buying in on this. You never want to have... You know, you get into a disagreement with one of the functional managers that you need to update a critical vulnerability, they resist, say, "No, absolutely not." Well, at that point, you need to have the C-level executive over each of your roles be able to come in and arbitrate that and enforce these types of policies. So having that management tier both the functional and the C-level is very important to that process.
 
And the other one that you wanna make sure to include are end users, during the testing process especially, the end users need to be involved more. You know, a lot of times the IT team will be able to, you know, get login access to a critical application. After you're done patching you go log in, you'd pull a record, "Okay, everything looks fine." Well, a lot of times that part of it may be working fine, but somewhere further into the application, something else may have broken.
 
If you updated, you know, the OS updates, there's been a lot of them lately that have GDI+ updates. Well, if that broke how images or, you know, some type of document was being rendered in the application that is critical to the business, you may not have tested those things and seen that it was working. But as soon as users start to get in there they will find, "Oh, printing broke," or, "Image rendering broke," or something else deeper in the application. Some type of job they have to run.
 
So as you test the updates that you're pushing out, it's good to engage with, again, that functional manager in those critical business apps, and say, "We need a couple of your best users." The people that the rest of your team turn to, and if there's a question, or a need for training, or somebody has to learn how to do something new that's the person that you would turn to. That's the person we need as part of this pilot program. That way we can roll out patches, they can do their functional testing and get a quicker turnaround to get to that next gate in the process.
 
So the people involved in the patch process is very important, it's not just the people responsible for patching, there is a very broad, extended team that needs to be involved throughout the process to make this work smoothly.
 
The next piece here is knowing your environment. Discovery, going back to that...discovery is a very critical piece of any security program. One saying that I like to use quite often is, "If you can't see it, if you don't know about it, you can't secure it." And that comes down to a lot of the way that vulnerabilities are exposed and exploited to get into an environment.
 
So in 2017, there was a...a news article came out about a North American casino that got breached, and the way that this environment got breached was actually through the fish tank. They had a fish tank, it was hooked up to a very nice IoT system that was monitoring tank conditions, doling out, you know, food at different intervals to keep the fish nice and healthy. Basically monitoring everything about that tank. Well, that was running on a system that was exposed to a vulnerability, because it wasn't properly being managed, that's how the attackers were able to get a foothold into the environment. And from there they were able to work their way deeper and deeper into the environment and be able to finally get access to some critical data.
 
So discovery, understanding everything in your environment and making sure that you can capture that information, and make sure that each of those systems are managed is very important. So you wanna make sure to talk to all the different teams throughout IT, understand what systems are out there, who's responsible for patching the OS layer, the application layer. Sometimes, you know, the application teams want to maintain the application layer themselves, but make sure that you've gotten an understanding of those systems, how they're being used, and who's responsible for the OS and the application layers.
 
Network wise, you know, how to make sure that you're covering everything throughout your environment. That you know everything that's mapped on the physical networks. That you're able to scan that network and be able to identify additional systems that are coming on or off. But also, you know, if systems are moving off network, making sure that you can also support those systems as well. You know, this is a principle that we often refer to as following the user, if any security control only works up to the network boundary that security control is, you know, fundamentally going to be short-sighted if that system happens to be a laptop. If it can go off network it's now exposed to many more things that my network security was taking care of.
 
So as it goes out there it's at higher risk, you wanna make sure that you've got the ability to know where systems can be in your environment. Another example, one of the customers of ours…this happens to be the Naval arm of the armed forces of this country. They have ships that will come into harbor. And when they come in depending on where they are in the patch cycle, within a few days, patching needs to begin for the systems that were on that ship. And they need to execute in a very specific fashion once that system comes, you know, back into the Naval Yard.
 
You know, another case would be law enforcement. A lot of times when you've got a squad that goes out, you know, it's going to come back in at some point, connect up to an internal network, and need to be able to get updates and things like that as well. So making sure that you understand the gaps that need to be overcome in your environment, and making sure that you have consistent coverage no matter where those systems go, or making sure that when they do come back in that they do you get scanned and updated.
 
And then you also wanna be aware of existing procedures. If you've got, you know, restrictions, let's say the HR system runs a quarterly job at the end of each quarter, so every three patch cycles you may need to, you know, have that particular set of systems be delayed by a week. So making sure you know which systems have different restrictions or challenges like that that you need to overcome. And enabling those groups to be able to notify you and work out when they will be patched, so that you don't miss a full cycle you just offset that cycle by a short period.
 
So talking a little bit more about identifying all the systems in your environment, you should be aware of all the different operating systems throughout your environment. You know, every flavor of every OS is vulnerable. There's no operating system out there that has no security flaws. Doesn't matter if you're on a Mac, a Windows system, a Linux system, you do need to make sure that you've got equal coverage for all these different systems.
 
One of the particular attacks that's happening right now, many of you have probably heard of the SamSam ransomware attacks that have been happening. These are very targeted sort of attacks that they are not like a typical ransomware where they start out with a phishing scam. These guys are actually targeting things like JBoss development servers. This is a Java environment that runs on Linux distro, and it's highly targeted by these attackers because it's not often updated.
 
So it doesn't matter again what flavor of OS that you're on, if it's exposed, if there's, you know, applications on there that are vulnerable, that is a vulnerability you need to be concerned about and make sure that you've got coverage for. Same thing with your applications, you know, there is obviously the main vendors, you know, Windows, Mac, many of the Linux distros have a very effective patching of their own OS, and even some of their applications.
 
But many of the other vendors out there don't release on a regular cadence. If you're looking at a lot of the browser vendors, so let's take Chrome and Firefox, these guys are on a continuous deployment model. As soon as they get to a kind of a critical mass they're going to release an update, and that's gonna come at any time.
 
You may have two to three releases in a single month for Chrome. And that may be because they had their regular feature release, they did another one that had a set of security vulnerabilities, and then they had a critical vulnerability that needed to be released very quickly because of either a disclosure that was coming up. Where a security researcher was holding them to a date where if they haven't resolved the vulnerability they were gonna do a disclosure. Those types of things happen pretty regularly.
 
So a lot of times there will be application releases throughout the month. What we refer to a lot on our, "Patch Tuesday Webinars" as between the patch Tuesdays. And a lot of times there will be security vulnerabilities plugged. If you're not aware of these applications, if you're not keeping those up to date, you can be exposed to several vulnerabilities that could be out there for quite some time.
 
There's also many applications that don't manage vulnerabilities the same way. You know, there's a lot of vendors who do a very good job, you know, VMware has got a fairly robust communication around their vulnerability patching, Google a very…you know, a strong program as well. Oracle and Adobe, but there's others that they never open a CVE. They don't notify you that there's actually security vulnerabilities being resolved.
 
So a lot of these products while they're not saying there's vulnerabilities there, they may be plugging security issues, and those pretty much go unnoticed just because they're not being disclosed. So applications, in general, should never be left to stagnate and not be updated for too long. So making sure you understand the breadth of the applications in your environment, and that you're managing those effectively is important.
 
So talking a little bit about some of those vendor release cycles, you know, Microsoft has a fairly consistent release cycle, every patch Tuesday you expect their security patches. Adobe has gotten on that patch Tuesday model with Microsoft. They've partnered up very strongly, and things like Flash Player are updated pretty much every patch cycle. Others like Adobe Reader and Acrobats they'll come out probably every two to three months.
 
Other vendors like Oracle do have a regular cadence that happens to fall on the Tuesday closest to the 17th every month. So a couple of times a year it'll be on patch Tuesday, a couple of times a year it will be the week after Microsoft's Patch Tuesday event.
 
And then there's other vendors who again may be on a continuous release cycle, you know, you may need to understand different subscription models to access the updates from these vendors. So there's a lot of complexity to each of the vendors that are out there. And one of the things that you can do is depending on the systems in your environment, you may have certain systems that, okay, we're on a monthly patch cadence for our servers. Maybe certain server groups might be on a quarterly cadence because they're critical assets, they're locked down tighter, but they also get less maintenance windows because of high availability, things like that.
 
So making sure that you understand what vendors are on each of these platforms you're running on, if you've got things like laptop users, those you may want to update on a more frequent cadence. Just to make sure that since they can go in and out of the environment, since they're exposed to more risk, they've got a higher frequency of being updated.
 
Ivanti content, we release twice a week on Tuesdays and Thursdays for our Windows patch catalog. And that is...for a lot of our customers, they're laptop users, will be patched once a week or even twice a week on Wednesdays and Fridays, to make sure that any security updates that come out within any patch release get covered.
 
So when it comes down to tracking a lot of these vendor releases, it can be difficult to keep track of all these. So we've provided a number of different sources for our patch customers, many of you of which are using our patching solution. So many of you are probably familiar with our, "Patch Tuesday Webinars" or even our content announcements for our Windows…patch for Windows and input manager products. We are working to standardize how we're doing that across all of our products and across more than just the Windows platform here.
 
You're gonna see some additional changes coming forward, but these are good places to tune into to get additional information. From, you know, regular emails that tell you when new content has been released, to, you know, massive amounts of content around Patch Tuesday, that give you kind of an inside look to, "Here's the known issues you should worry about. Here's the high profile vulnerabilities and disclosures that you're gonna need to be concerned about," and where to focus your attention. So that's a lot of really good content that helps you to get better at tracking those vendors and tracking release cadences.
 
Now, many of you are familiar at this point with how Windows 10 and Server 2016 are operating. Office 365 is also going on, you know, a similar kind of semiannual channel model or branch model. So the way that patching in general is kind of changing. This Windows as a service rollout makes it so that we've got a regular cadence of patching still. The patching model has changed obviously. We've got these monthly roll-ups, and it's a cumulative model, so everything that you push out includes everything that came before now on Windows 10 and Server 2016.
 
But we also now have to deal with this branch upgrade, so we've got the monthly and cumulative security and non-security updates for older products yet, so anything... You know, Windows 8.1 and earlier, Server 2012 and earlier. You can still choose to do a security only bundle each month for the earlier operating systems. But Windows 10 and forward, you have the monthly cumulative roll-up model.
 
The branch upgrades, they are a semiannual channel, so roughly twice a year Microsoft will be releasing a new branch for Windows 10, and the Windows Server semiannual channel. And with that, you know, you've got basically a 18-month cycle, or a 24-month cycle depending on which edition of Windows you're running to be able to move up to that next branch. And then there's the long-term service channel.
 
Now, initially, many more companies were looking at the long-term service channel and trying to identify if that would work for them. Microsoft has recommended strongly that this is meant for kiosks or, you know, not really a knowledge worker type configuration. It's more limited to certain capabilities and not very flexible. So if you're gonna be putting this in for a knowledge worker, the long-term service channel may not be ideal there, they don't recommend...or they don't guarantee that all browser functionality, or Office suite functionality, or things like that will work the same as it would on the semiannual branch or the semiannual channel.
 
So, you know, as you're getting into this for companies who are still investigating and looking to do that rollout, the semiannual channel is the most flexible user experience. The long-term service channel is ideal for when you've got something hooked up to a piece of manufacturing equipment, or medical equipment. Or even, you know, a kiosk where you might have some user interaction but it's limited to a certain set. That's where the long-term service channel may be ideal.
 
So getting a little bit more into the branches that are currently out there, for Windows 10 you can see here we've got a couple of end-of-life dates or end-of-service dates for these already. So out in circulation, you know, there was 1507, 1511, those two have had end-of-service dates already. 1607, 1703, and 1709, are still available out there. Now, if you go to Microsoft's lifecycle page, you will see that from the point where the 1607 branch released, it has an 18-month cycle before it hits end-of-service.
 
At that point, if you're running on the enterprise or education editions, you get an additional six months of support past that. So 1607 for pro and home users would end after 18 months. And you have…1803 isn't added on there yet, we're gonna add that one on there as well going forward. But that is supporting our products already just a matter of it...when we created the deck it was not fully released yet.
 
So server 2016, we've got that release now and you're gonna start to see a similar cadence. So when the Windows 10 branch releases the Windows Server branch should be releasing roughly around the same time frames. So Windows Server we've got 1709 right now, and the 1803, and going forward we should see that every...or, twice a year we should see a similar cadence for the Windows Server branch.
 
For the legacy operating systems, again, for Windows Server and Server 2012 R2, anything up to Server 2012 R2, and up to Windows 8.1, you have the regular patch Tuesday release which is security focused. That security only quality update will only have the security updates for that month, it's not cumulative. So you won't get all of the non-security fixes that came in between or anything like that. So that gives you a little bit more selective patch capability. And Internet Explorer, in this case, is also broken out. So if you're doing the security only quality update, you're gonna have that plus the IE update on Patch Tuesday.
 
And the reason Microsoft did this was to try to give some flexibility around the different challenges if you need to keep a browser at a certain locked in level. Or if one of the security only bundles did conflict with something, next month you may be able to patch it as long as that same issue didn't get superseded and fixed in the next quality bundle.
 
The security monthly quality roll-up, this is basically following the Windows 10 model, but for all the older operating systems. So you can go with a cumulative month to month update that includes everything including the non-security updates that will come out between Patch Tuesdays as well. So for the security monthly quality roll-up, there's two releases each month. There's the security monthly quality roll-up, and then there's the monthly preview that comes out later in the month.
 
So that brings in the non-security fixes at the end of the month, and then next patch Tuesday's security monthly quality roll-up…that's a mouthful, will include the preview, the non-security fixes as well in that patch Tuesday release. So that works exactly like Windows 10 where each update you get the cumulative of everything that came before.
 
So in this case, a lot of times what we recommend is stick to the security monthly quality roll-up where possible. If you look at the image here when Microsoft is doing testing, if you're on that cumulative quality roll-up model you've got everything on there and you're testing...you're getting exactly what was tested in Microsoft's environment. If you're doing, you know, especially the older model where you can do individual patches, if you're piecemeal adding in just the updates that you choose, Microsoft is not testing things in those types of configurations as cleanly. And that's where they were saying quality will be better if you follow that cumulative roll-up model. So that's the recommendation.
 
Now, where you run into exceptions, again, if I'm in an environment where I need to keep IE at a certain version, that puts me into that quality update model instead, so that I can break IE out and patch the OS individual from Internet Explorer. Or if those non-security fixes are intending to break something in a critical environment, that security only quality update again is the way to go there to keep those non-security fixes separate from the security. So those are the recommendations there.
 
So as you're going through, you wanna make sure to try to group systems together, identify large groups of systems that you can patch in the same maintenance window with the same policies. You know, exceptions again should be an exception. So you shouldn't have, you know, exceptions across the board, exception should only come into play when necessary. And you wanna make sure that you've got a fairly common patch policy for environments across your organization.
 
Operational requirements you wanna make sure that patching and availability... It has a function of risk assessment, not just software updating. This needs to be treated like a security topic. There are security vulnerabilities in these updates that is the main reason why patching is necessary. So make sure that people don't think of patching as just an operational issue, it is a security issue.
 
Pre-deployment systems, impact and validation testing. It's good to have as good of a test environment as possible to replicate critical systems that you wanna make sure you patch those first, and then you start rolling out to the next group. As part of that validation testing, you also want to include some regular users. Some of our most effective customers... You know, one in particular that I can think of, these guys had 88,000 endpoints and they patched them in under 2 weeks.
 
The way that they do that is they have fully engaged with the organization and they have a set of test systems even if it's a small number, but they also include a lot of users that are active in the test cycle. So that in the first couple of days after Microsoft's Patch Tuesday release, they've got validation coming back from end-users, and they know a general lay of the land as far as how it's gonna impact the broader environment. And with that, they can quickly move through each gate in that process, expanding out to larger and larger groups.
 
So these things can be done if you engage fully with the management and with the users in your organization, that's where we've seen the best successes in our customers in patching. So that one was actually a large manufacturer, we've also got another one I can think of as well that is a large healthcare provider, these guys are patching again in under two weeks. And the way that they're able to do that is they had a new CISO come in a few years back and her first initiative was making sure that topics like patching, asset management, and other things were being taken seriously as a security function, not as just normal asset management or operational updating. 
 
So from that standpoint with buy-in from the top, making sure that you've got these groups involved in the testing process, they've been much more successful and can do these things in a much faster timeframe. So getting in the maintenance windows, making sure that the organization understands, "Okay, here's the maintenance Windows you're gonna have." You're gonna have some environments that are gonna be more restricted than others.
 
We've got several customers who run in test labs, where they're testing out critical systems that they're developing for the market or for a contract. Those environments have very strict maintenance windows. You may have 24/7 parts of your operation, those types of maintenance windows, yes, absolutely understand those. Make sure that you are finding time to get to those systems though. If it's not an HA system and it's 24/7, that is a highly risky situation to be in. So making sure that you either get maintenance windows set up and you're able to maintain those systems from the security perspective. Or making sure that you get, you know, a high availability system put in place so that you can, while keeping it online, be able to patch portions of it and get those back online and transition through them all effectively.
 
So maintaining those maintenance windows are critical to understanding those operational requirements. Where you don't have a maintenance window available you need to figure out how to get one. And those conversations are never easy, but again, with management involved with the security perspective, those conversations aren't just discarded as, "No, we can't do that. It's not gonna happen." They need to be taken seriously.
 
Policy types, so there's a couple of different ways that policies could be handled. Again, we've talked about security only versus monthly roll-ups, but there's also, you know, what mentality are you using in this? With a baseline, you're basically saying, "Okay, everything will be approved after it's been tested." And then there's, you know, others that are going with that kind of continuous roll-outs. Certain applications, it doesn't matter, you know, what it is that released in there. It's gonna be patched, and it's gonna be patched quickly.
 
Things like Flash player often fall under that model, browsers, there's kind of a certain group of applications that need to be in that zero tolerance type of mode. If it's on an end-user machine and it's one of those critical applications that's highly targeted, it always gets patched and it gets patched frequently. And, you know, that's, you know, one of the things where as you're going through this and defining your policy or updating your policy, those are important decisions to make and to come to... You know, there will be some arguments over that.
 
I've been in many organizations where parts of the organization are completely resistant to updating certain components. Again, because they're not thinking about the security ramifications or the compliance ramifications, they're thinking just about, "Oh, this might break something. When's the last time it broke something?" "Well, I can't remember but it's got a high risk of breaking something." Well, those types of arguments need to be had and need to be overcome to be able to better manage your environment.
 
All right, Todd, we're gonna switch over to talking a little bit about configuring systems within your organization and understanding how the content and everything updates, and things get delivered to the environment.
 
Todd: Thanks Chris, yeah, so Chris has kinda set the stage around the background of, you know, making sure you know what's in your environment, making sure you know the people that are involved, knowing what applications, operating systems, etc., that you have out there. So let's start digging into a little bit of...we'll say implementation or configuring your systems at this point.
 
So the next set of the slides that I put together here were really around, you know, what happens when a patch is released and what does Ivanti do, and what do you need to do, you know, basically out of your side. So really the first step in this from a patch data flow perspective is, you know, producing what we call content. And really what content is all about is when the manufacturer releases their patch we download those binaries to Ivanti.
 
We run them through a process whereby we look at the patches themselves, we put together a series of metadata that our products will recognize in terms of detection rules. We identify the location where the binary is being stored by the manufacturer, we provide all that information that you see in our products, when you select the patch you can see the CVEs, you can see whatever information the manufacturer has provided about that patch. We provide file size, we validate that it's, you know, properly signed by that vendor, and we make sure that we can validate the patch once it's deployed down to the endpoint.
 
So really we put together all of this content information. And, you know, this is kind of interesting in that, you know, we get a lot of requests saying, "Where are these patches for my SAP system?" Or, "Where is the latest patch for my medical applications?" And the reality is we have access to, you know, the publicly available patches that the vendors provide. We do not subscribe to obviously every application that's out there as a company, we obviously just can't afford to do that. So there are always going to be a large number of patches released by these other vendors that we just can't support in our environment.
 
Typically, what we're pulling down is that list that Chris showed you on the slide earlier with all the applications we support, all the openly available patches and updates that are available. This is kind of the first step that we go through. We pull this down, and typically within 24 hours often much less than that, we'll have this content available for distribution. 
 
So that's really kind of the next step in the process as far as the data flow goes, it's the content distribution. So here at Ivanti, what we do then is we publish this, we post it up on our website, content.ivanti.com. And the products know where to reach out and grab the information…this content information about the patches. So at this point, they will pull down information from the Internet about all these patches that are now available, and that information is stored down, you know, in our products in your environment.
 
And so then the next step obviously from a data flow perspective is to scan and detect the missing patches. So the products take that information about how to detect if a patch is available. It'll scan all these endpoints where either an agent is located, or agent lists, or whatever configuration you happen to have in your environment. And they'll pull this information back, and you'll collect information about what patches are missing in your environment. And at this point, the next step will be to actually determine and pull down the missing patches.
 
So you actually pull down the binaries at this point and store them in your system. And this often is a kind of an interesting question, and we get asked this a lot. You know, why doesn't Ivanti just grab all these patches and provide them for distribution? Well, there's a number of reasons, a big one is sheer volume obviously, you know, grabbing all the latest updates from Microsoft and across all these third-party vendors is a huge amount of information. Second of all, there's also a large liability associated with that. You know, we just can't keep up with making sure that we have the latest patches from every vendor stored somewhere and available, and we rely on the vendors to do that. Nor do we wanna have these in, you know, a location where they could possibly be compromised.
 
You might remember the Petya ransomware attack from last year, that was a result of a patch being compromised, and then being distributed out to, you know, to all those endpoint applications. In this case, I think it was patch software, and that spread like wildfire after that point. So you wanna make sure, you know, that from a liability perspective that you're getting the patches directly from the vendor. And one of the things that we do obviously as part of that content production is make sure that we have checks and balances in the product, that when it goes out there...when our products go out there to the Internet, pull down these patches, they are properly signed, the hashes are correct, and they are a valid patch. So very important from that perspective.
 
So obviously at that point, we now have the binaries down and available on the endpoints, so the final step, you know, at this point obviously is to deploy those patches out to your endpoint. So depending upon what your policy is, you know, Chris talked quite a bit about different policy implications here, these would be, you know, pushed out to the endpoints. The patches would be effectively installed on the endpoints, and then, you know, depending upon whether there's a reboot required or not, the vulnerability will be remediated through that patch on the endpoint.
 
So I provided this to kinda give you the big picture of, you know, what's actually happening out there. So from a content production all the way through pulling down patches, and then deploying patches on your endpoint. So you can start thinking about, you know, where do I need to, for example, install my patch servers? Getting back to your network discovery and knowing your environment that Chris talked about earlier, you can start thinking in terms of, you know, I have this patch product from a scalability perspective, where do I need to locate my servers so that I have network access to all the endpoints.
 
Are there geographic distribution requirements that I have to take into effect? For example, maybe I have…you know, follow the sun patching for example with different patch teams if I'm a very large organization. So you need to take those kinds of things into consideration. Also, from an Internet access standpoint, you saw earlier that you need access to get the content from Ivanti obviously, pull down the latest information on all those patches. And of course, the system has to be able to go out and grab the binaries from the third-party vendors as well.
 
And finally, considering the big picture once again, you know, how are you set up from an administration standpoint? Who's responsible for using these systems to deploy patches? Are some people responsible for scan only and detection and another group responsible for testing and distribution? You could think about that in terms of, you know, your organization. We see a lot of different setups with all of our customers, in some cases, they may be a very small organization, and one person is responsible for everything from start to finish from a patch perspective.
 
In other organizations there are a lot of times deeper checks and balances in place where, you know, an administrator may be responsible for pulling down patches and doing initial testing. Then that information is handed off to another, you know, configuration control board, and then another group is responsible for deploying those patches. So there can be a number of different approaches to this for you to consider.
 
Chris talked about that testing process, you know, we see different organizations once again do different things depending upon what they need. Typical, you know, configuration control process which we're mostly familiar with, if you're in a very regulated environment where you have data centers with strict maintenance windows, you know, we'll see the release update come out. There will be a download period where you will pull the patches in and do extensive testing to make sure that they're approved, and they work properly on the endpoints. And, you know, you're getting the results that you expect. And then finally, they will be rolled out to production.
 
You might have different success criteria in place for, you know, different steps in this process, but, you know, once again this is a kind of a typical roll-out process. And another time, this process might be very quick, like for example, Chris was talking about browsers and how browsers are updated. You might not go through quite so formal process for browser updates because, you know, there have typically not been very many problems with browser updates in the past. So maybe a very light testing process and it be rolled out very quickly. So something to think about in your organization and how, you know, you would do your testing.
 
So let's talk a little bit more about actual setup within, you know, the patch products themselves. And I have a couple of examples here from the patch for Windows products and organizing information here on the right-hand side, this happens to be from the former HEAT product over here. Chris talked about organizing your systems and how you would go through patching those, and the process there. Typically we see, for example, like systems in a data center may be organized into one group and those will be patched as a group. Whereas, you know, laptops will be set up in a separate group for a particular business unit, or something like that.
 
So the key thing here is mostly around...the key thing for your initial set up is organizing typically into patch groups, and this is organizations of machines and how they would be patched. So you kinda think in that process.
 
The next step has to do with patch selection and actual patch management. This gets into... We talked earlier about the discovery phase, you know, what applications do I have in my environment that need to be patched? What operating systems do I have in my environment that need to be patched? And all of our patch products and every patch product on the market today really, has the ability for you to select and choose, you know, how you would like to go about patching.
 
I realize this is a screenshot with a little bit of small text here but you can see here all the vendors and the applications that we would support for testing. And then you would choose, you know, what kind of security patches you're really interested in. We have the ability in our products...and by the way, this is the patch for Windows products, some people are asking what this is. This is the configuration screen for patch for Windows, the current release is 9.3 that these were taken from.
 
And in this case, you know, I've selected here from a security patch perspective, I'm only interested in critical and important patches. I'm not interested in the non-security patches. I'm not interested in some of the other security tools. Just an example of how you would go about selecting and identifying what applications you want to patch in your environment. And like I said, this is just kind of a general discussion that we're having here today, but you would go through and selectively choose, you know, how you wanna go about doing this.
 
If you're going through and you are working with more strict configuration control in an environment, for example, in a data center with critical servers, you may go through and select rather than just selecting the applications themselves, you would go through and select the specific patches. We have the abilities to create what we call a patch group in all the products. In this case, I've created an old one, an old set of...I grabbed an old set of patches here from last year and created a patch group for 2008 servers. I go through and identify those, and I'm gonna use those essentially as a test group and as a baseline in the future here. So I'll then be applying only these patches during my next test run.
 
So just an example of how you would go about doing this. In this case, you can see that I've selected this and created a baseline, I've chosen these patches. And I've gone through for example, now and going back to that test model that I showed earlier, I've gone through I've selected these patches. I've tested them on a series of systems that are representative of my production environment. I've gone off possibly and gotten approval from an external group if there is a configuration control board maybe, or, you know, maybe I have the authority to do this myself. And now I have the ability to go through and, you know, apply those on an endpoint.
 
Another thing is, you know, we talked about maybe more of a rolling model, and I wanna scan for everything and identify what patches are out there. In this case, I'm scanning for everything but Java, I don't care about Java, you know, I handle that separately in maybe another patch process or something like that. So you have the ability to not only scan for selective things, you have the ability to exclude things from scan. And once again, you know, we're digging down into kind of maybe some specifics here, but just I wanted to show you some examples of different approaches you could take in your patch model depending upon what you're trying to achieve in your particular environment. Like I said, in this case, we're scanning for everything but were excluding Java.
 
So we have a lot of configurability within our products depending upon what you would like to achieve during a given scan cycle. Getting to that scan cycle, let's talk a little about scheduling. We have different approaches to how you scan and how you deploy patches out to endpoints, for example, in this particular configuration we have the flexibility to just basically scan. I can choose that template that I just created a minute ago called, "Scan everything but Java," and I'm gonna scan it, you know, on a recurring basis. And I have the option, do I want to immediately patch after the scan or do I just wanna stop.
 
Okay so you can see that you can choose the option to just identify the vulnerability of the missing patch, or, I have the ability to immediately deploy packages or patches afterwards. So, in this case, getting back to that model that I showed you earlier with a little bit of animation there, in this case, the scan would kick off based on the content that we've downloaded. And because I've set this particular one up to immediately patch afterwards, it would go through and pull down those patches and deploy them out to the endpoints immediately and patch them. And I'll talk more about reboot here in just a second.
 
And we have a third part to this which is our deployment template, and this has to do with, you know, when do I wanna install the patches? Do I wanna schedule them at a certain time? You know, do I want them just to correspond to the maintenance windows that I have in place. Once again, getting to the overall process that I'll talk about here in just a second, but really what you wanna do is make sure that, you know, whatever you've set up from a process perspective, you know, it matches up with the way you do patching in your organization.
 
Final screenshot that I have here, this is again from the patch for Windows product, you can see here that I've scanned patches... I mean, I've scanned endpoints and I see that I have a number of patches missing. Maybe I've just done a scan and I've stopped there, I haven't deployed anything at this point, what I can do is possibly if this is a very critical server, I can go through and manually apply patches at this point and push them down basically in real time as opposed to doing them during a maintenance window, or something like that.
 
So you can see that you have a lot of flexibility within the products themselves, we could spend probably hours just talking about different configurations and different ways that our customers use the products. But just wanted to give you some examples of kinda what's available and what can be done out there.
 
So let's talk more about actually running patch operations. Kind of the steps in a normal patch operation process, you know, you monitor content meaning you're keeping an eye on what's happening out there. You know, there's a lot of vendors to keep an eye on, we do that for you. We provide, like Chris said, regular updates that you can subscribe to. You can see that for example, we release patches on Tuesdays and Thursdays, information about patches on Tuesdays and Thursdays on everything that's happened, you know, between those breaks.
 
So for example, if patches are released over the weekend or something like that, they would show up in our content on Tuesday. So you don't have to be on the lookout for that, you would be notified in the products themselves. From that point, you would build a baseline possibly, and do your testing. Or maybe you would just test the latest release patches very quickly, obtain configuration control like I mentioned if required. The next step would be, you know, involving all those groups that Chris talked about on his introductory slide there about who's involved in the patch process.
 
This is where you would send notification out to employees, users, your helpdesk, etc., once again, depending upon what your process is internally. And finally, you know, you actually do it. I mean, this is all prep work up to scan the planned patch right. And then finally, you know, make sure you monitor results. There are always going to be systems that are maybe offline when patching occurs, or, you know, there are problems applying patches because of environmental variables, you wanna make sure you monitor those results as well.
 
Occasionally, you will have need to conduct some emergency patch operations, you know, maybe there's been a breach and all of a sudden you have to respond, and then provide patches based on a downward directed…you know, things like that. So you would build your response patch set, immediately send notifications out, and very quickly, you know, basically you're shortening this process down to a matter of maybe hours instead of maybe, you know, a day or two.
 
Chris talked earlier about the importance of, you know, communicating patch activity out to employees. This is an example of what we do here at Ivanti, this is a little bit dated, but you can see that we send out information. This is done actually through our email notification system from our IT office saying that a patching window is coming up and we're gonna be updating our systems, this is from last year obviously.
 
Another thing to have in place because, you know, problems are going to occur, it's not an if, it's when things go wrong, is to make sure you have an incident response plan in place. You can test everything and make sure, you know, everything kinda, you know, works properly in your test environment, but when it comes to actually deploying patches things happen. That's just a reality of it. You know, make sure that you have, you know, cross-team processes in place, you…
 
[00:51:54]
 
[Silence]
 
[00:52:17]
 
Chris: [crosstalk 00:52:17].
 
Todd: Sorry.
 
Chris: It sounded like we lost you there for a little bit.
 
Todd: Where did I drop out?
 
Chris: I think you were just talking about the...you had just gotten past the, not if, it's when, and then going into the things to consider.
 
Todd: Okay, sorry about that, am I good now, you can hear me?
 
Chris: Yeah.
 
Todd: Yeah, so, you know, you need to make sure you have a plan in place when things do go wrong. And make sure that you have the right teams involved in this process. And I put down a couple of notes here for things to consider. You know, there's different types of events that will occur and make sure that you have escalation procedures associated with each of those events. For example, it may be you might have an insider issue, or you might have an outsider attack type of event. And the procedures may be different, you know, from your patch team based on those attacks.
 
And again, I showed earlier that there were standard procedures versus emergency procedures and how you would shorten those down. And finally, something to consider also is, you know, public disclosure versus information control. And we've heard this quite a bit, I mean, you know, when you have problems how much do you share with if the press gets involved, if outsiders get involved versus your executive team gets involved. You wanna make sure about how you're going to handle disclosure of, you know, what you've done, exactly how you've patched, exactly what you fixed, how you responded to the emergency. So you want to have that as part of your incident response plan as well.
 
And finally, you know, I can't stress this enough, you know, you really wanna make sure that this is not something that just sits on the shelf and you never take a look at it after you've put it in place. You wanna make sure that you do exercise the emergency plan, kind of, like the fire drill, right? You can do the best plan, put it in place, if you never exercise the fire drill plan you're not gonna come up with, and you're not gonna find, you know, where the flaws are in your plan until the actual emergency occurs. So very important that you exercise and continue to improve this plan over time.
 
Chris: But the way, Todd, Ross, suggests that it might have been a patch that caused your audio fade out there…
 
Todd: There you go, nice. Chris, were you patching me while I was doing this?
 
Chris: Yes, I patched your system, so no.
 
Todd: No problem. I think this is my last slide here, we're getting near to the end. You know, there are times when you're not gonna be able to patch the system for whatever reason, and you need to make sure that if you can't patch…if you can't remediate you need to have mitigation procedures in place. Listed quite a few different things here for you to consider. Reducing user privileges, you know, removing direct access to personal information, or, you know, things like that. You know, virtualizing some systems to make sure that you can recover very quickly. You know, reducing exposure to neighboring systems, this comes down to, you know, how is your network configured and things like that.
 
Removing direct Internet access should that be an option that's available for a particular system. We've seen that time and time again for critical internal systems. For example, you know, HR, or something like that, where, you know, certain billing systems or something like that are maintained internally. Implement application whitelisting, again, another capability to limit if there is a vulnerability that can be exploited and, you know, malware installed. At least with application whitelisting you can keep that malware from running.
 
And finally, we're hearing more and more about containerization, you know, and making sure that... This is once again, you know, an isolation process to make sure that different applications or different systems can be managed more effectively when you can't patch. So with that, I'll turn it back over to you Chris, for a little bit.
 
Chris: All right Todd, thanks. All right so just a couple of things, you know, there's some trends that are happening in the industry, some challenges that are being introduced into the overall process as well. Many of you are dealing with things like DevOps now, and the problem with DevOps is you need to make sure that security is part of that overall process as well. So DevSecOps is a term that's being used by a lot of the analysts in the industry as well, to make sure that you don't forget that security aspect as you're rolling out these new build.
 
So the whole point of DevOps is I can update in tests and then validate everything is good, and then promote that to production. If you include your patch process in that DevOps flow, you can make it so that everything goes out into production fully patched and ready to go, and has passed, you know, a level of testing there. If you actually, you know, use services like Amazon, or Azure, or any of those types of environments, a lot of the infrastructure that they're managing, they don't actually patch production. What they do is they patch in the DevOps flow and then promote the patched version to production, and the current production version goes away.
 
So this is a model that's coming up more and more, and to try to make it so that we can fit into that process as well we wanna be able to open up, you know, APIs within our products to be able to allow you to integrate those steps into your DevOps flow. So, you know, you've got the ability to, you know, with Chef, or Puppet, or vRealize, or, you know, there's a lot of different orchestration tools.
 
Ivanti he has our own automation platform as well, we are opening up APIs throughout many of our products to be able to integrate into these types of flows. But you wanna make sure that your applications and OS provisioning are all…you know, making sure that everything gets out to production fully up to date. That you can integrate with many of these different tools out here quickly and effectively.
 
Another challenge that a lot of companies have are, you know, many of you are running, on the security side of the house, you're running some type of a vulnerability assessment solution as well. Whether it's Qualys, Rapid7, BeyondTrust, Tenable, Nessus, it doesn't matter which one. Each of these vulnerability tools goes out and looks through...looks at the world from a slightly different perspective. They're gonna come back with a list of vulnerabilities and that list of vulnerabilities is gonna get passed over to you. And that could be anywhere from hundreds to thousands to tens of thousands of vulnerabilities that were discovered in that single pass, and now you've gotta go and research that.
 
So that same API strategy here, we want to facilitate the integration between the vulnerability vendors and our patch solutions as well. So, you know, we've already got some examples of integrations like this with the patch for Windows product, and we're gonna be expanding out endpoint managers, API feature set later this year. And the patch for Windows APIs will change from just being PowerShell based into being RES interface as well. But you're gonna see more and more of this API strategy expanding out to facilitate integrations like this.
 
So we do have working examples of Qualys, Rapid7, and BeyondTrust integrations with a patch for Windows product today that show you how to grab a CVE and pass that in and build up a list of software titles that need to be deployed to resolve that. And we're gonna be expanding this out over time as well.
 
But these are a couple of areas where if you build in your patch management solution into and automate these steps when you're... You know, you can eliminate a day at a time on average when your security team does a vulnerability scan. Just in the amount of research you're gonna spend to try to deduplicate and validate which CVEs are gonna be resolved with which software updates. And you may find that, oh, because Reader was three versions out of date, there were actually 150 vulnerabilities that are gonna be resolved by just pushing the latest Adobe Reader.
 
So these types of things can be automated and taken from hours down to minutes as far as the overall process would take to do those steps. So those are a couple of things as well to start to look towards to make sure to integrate into your overall process to make things easier on you and your teams.
 
So we do have some questions that came in and I know we're right up against the hour here, so we're gonna try to get through some of these fairly quickly.
 
As far as the screenshots you guys were seeing before, we do have a number of patch management solutions at Ivanti and those we're working to kinda make those more similar over time. But a lot of the screenshots you were seeing were from the patch for Windows product. That has some of the richer interfaces for looking at a patch and seeing all the different data about it. So that's why we chose those for displaying that. It's just the information is more tangible in that particular interface that's why we showed that one.
 
There was a question actually from James...this is actually a really good point, James had a comment earlier about, you know, why do they still call it patch Tuesday when really, you know, Microsoft has two or three releases of updates each months? Well, you know, James they still kind of refer to that Patch Tuesday event as the security-focused event. They may have additional security fixes that come out through the...you know, the next week and a half or two weeks there, just because they're…you know, the way they're doing this, the massive amount of updates they're providing. They, you know, maybe a version of a patch had to be rereleased or they didn't include a certain platform update on Patch Tuesday it came a few days later. So that happens a lot.
 
And then there are the non-security updates that happen at the end of the month as well. So depending on where you wanna put your focus, I definitely recognize a challenge, it's hard to keep into a consistent maintenance window when you've got all these additional updates constantly coming out. So one of the things that we often suggest and I kind of mentioned it a little bit before is getting to a point where, you know, you classify your assets. If it's a server, that's gotta go into a more consistent kinda maintenance window once a month. If needed once a quarter because of extenuating circumstances, but once a month preferable.
 
With end-user machines, machines where users could be targeted, could be phished, could be…you know, vulnerabilities are much, you know, more low hanging fruit on a user system. It's much more critical to try to get those into a more frequent cadence. That's especially laptop users, that's where I often suggest getting into a weekly or twice a week cadence. And maybe you set up a policy that only does the like Flash Player, Office, and browsers, you know, once a week, but everything else like the OS updates will just be once a month. So there's different ways you could strategize on that and do that based on your risk tolerance.
 
There was another question from Suzanne that, you know, would phishing emails... You know, wouldn't a phishing attempt be stopped by a patch? And the answer to that is in many cases, yes, oftentimes a user targeted attack is going to exploit a vulnerability first, then it's going to launch a payload that's going to do something additional. Now, there's fileless attack methods as well, and for those, you do need additional security measures, that's why defense in depth is so important. But many of the vulnerabilities that a phishing email would target are exploiting an Office vulnerability, a browser vulnerability, or something like that first, to be able to launch the attack. Then it's gonna download an additional payload whatever it is, whether it's crypto, or a rootkit or, you know, some other type of command and control software.
 
But many of those phishing attempts are stopped by making sure that user targeted vulnerabilities are addressed. Actually, that's one thing that... And where's my mouse? There we go. Let me show... Sorry, I'm trying to bring up another screen here. It's hard to do with so many windows on my other screen that are floating in front of everything.
 
So if you go to... We mentioned our patch Tuesday page, if you go to ivanti.com and go into our patch Tuesday page under resources. We actually have a fairly cool graphic here that we release every patch Tuesday as well, but this shows updates that have user targeted vulnerabilities in them. So you can see here many of the updates that released last month included... You know, these are vulnerabilities that an attacker would basically target a user and... Oh, apparently Lenovo have released another update.
 
That they would target a user to exploit them either through a browser attack method, through an email attachment, through chat messaging. So you can see here user targeted vulnerabilities are pretty common, you know, Flash, the OS, IE, many of these had user targeted vulnerabilities this last month. So that's some of the information we put out and helps you to prioritize, if you're on an end-user system these are all the updates that those types of user targeted attacks could be a risk for. So that's good information to utilize there. Let's see, I think we've gotten... Todd, are you seeing any more questions here that we wanna try to grab?
 
Todd: Yeah, there's one Chris, let's talk a little bit about patch application and firmware application and rollback, and kind of best practices around that. So there have been a couple of questions around, you know, when I apply patches on key servers and like that, can I do rollback? Well, on the patch side we are dependent upon what the vendor supports, okay, so if rollback is available for a patch we will support that in our products. We do not try to do any additional testing or provide any functionality for rollback that does not exist on top of what the vendor does. So that's for patching specifically.
 
On the firmware side, this gets a little more dicey, okay? The problem with firmware, and when you update firmware on devices, we can do that with some of our products, for example Endpoint Manager does that very well. In a test environment because you cannot typically roll back a firmware update you can apply it once, make sure it works properly, but then you're kinda stuck at that point, you can't go back and do multiple tests, okay.
 
So, you know, we're kind of in the same boat. We have a very limited test environment for some types of hardware and, you know, we'll apply the firmware update and make sure it works the first time. But we can't do it multiple times and create very complex environments sometimes to do that. So firmware is much more…you know, provides a much more limited test environment, you have to be very careful about how you apply those. And you usually get one shot at applying them and make sure they work, and that's about it, before you take it to production. Anything you wanna add on that Chris?
 
Chris: Yeah. I would say that I think there's a shift coming around the firmware side of things. I think Meltdown and Spectre has opened a lot of eyes. There have obviously been other security vulnerabilities. One of the few vendors that have taken firmware updates very seriously has been Nvidia. There have been a number of security vulnerabilities identified at the GPU level and those guys have taken updating of their firmware for security issues very seriously over the years. But I think more of the vendors are starting to realize that.
 
So Shawn right now, if you go to Dell site, or Lenovo site, or HP site and you try to drill into and figure out, okay, what exactly, you know, needs to be updated to make sure that there's no security vulnerabilities exposed on my system. They've got a very limited amount of information actually available to understand that, and even less when you're trying to manage things as far as like, you know, our Endpoint Management solution can roll up driver updates. But, you know, being able to go in and say, "Give me only the ones that are security related." That's types of metadata that just don't exist from the vendor on outright now.
 
You know, so there are some challenges there, I think the industry, in general, is coming to a shakedown on the firmware side, and we've been talking with a number of vendors to try to see if there's more that could be done there. So it's definitely a topic of interest.
 
Now, a couple other questions that came in, so there was some interest around the vulnerability management vendor integrations. So Kevin your question around how do you have a discussion around that Qualys integration? Depending on the product, if you're on patch for Windows we have a KB article out on the community that shows how to do that for that particular product. Again, these are happening in certain product areas specifically right now. I would say you can reach out to your TAM, express interest there. Have them get you in touch with the product manager for your product line.
 
So if you're on Endpoint Manager, getting in touch with Eran and getting an idea of, you know, what would be ideal there. It's something where those APIs exist within an Endpoint Manager today, but they're not very widely publicized. They're more for partner use than end-user use, we're changing that model going forward, there's gonna be more documentation, more availability to just our general customer base for those APIs going forward as well.
 
But that would be a good way to go about it, either contact your TAM or you can either contact me, or I can just contact your product manager for the specific product line you're on. So Todd, in this case, if you're on patch for Windows or our patch for SCCM products, Todd is the product manager for those.
 
Mahesh, you had the same question but about Rapid7. So Rapid7, their older versions didn't have an API that let us do this very efficiently. So we've got a crude model where you have to export an XML formatted report, we can send you details on how to do that. If you're up to the latest version of Rapid7 we're actually in conversations with those guys. We just met up at RSA, they're gonna get us a license for their latest version and we're going to do a similar documented version of that integration for the patch for Windows products, I know which are on Mahesh. We're gonna be doing that with their new RES interface in the near future here.
 
There was a question about do our products support custom patching? So yes, Endpoint Manager or patch for Windows, you know, the patch and remediation product from the legacy HEAT side as well. Each of those have a custom patch module in there, you can basically package up and deliver custom software updates. The difference there is that's going to be something that you have to monitor and add in each new package as they release. If it's publicly available or a commercially available software that doesn't require like gated access to it, that's one where, again, I would say suggest a feature request to us and we will prioritize that to get added into our content stream.
 
Aside from Microsoft, we've got over 150 non-Microsoft product families supported within our Windows patch catalog today and growing. But as we move forward again here we're gonna get to the point where we're gonna start to add some more products again. We kind of took a pause in the early part of the year as we were working on some consolidation projects but we're gonna get back to adding new vendor and product support again. So getting any suggestions in there that are available for us to be able to support, we can definitely prioritize those. Let's see...we talked about firmware.
 
So Anthony had a question about deploying the latest Windows feature release when using third-party encryption. That's something where maybe contact us and let's have a discussion about that one. I'm not sure we have a specific… I'd have to talk to somebody in the support organization to see if we have specific cases where those branch upgrades have been done that way. My guess is that you're using the drive encryption where you need to put in a key before it even gets to the boot process.
 
That's a little bit different than if the encryption...if you don't have to do the encryption key every time you turn the system off. So reach out to us we'll probably need to get a bit more conversation around that one. Let's see, I think we've got most of the questions...
 
Todd: There was one question, Chris, about total number of reboots and if we can tell how many there are going to be.
 
Chris: Got it, that's kind of a moving target. One thing that we always try to do is we prioritize patches in a way where... And this is one of the reasons why we've been switching...like, Endpoint Manager has recently switched over to the Shavlik Windows patch engine for those of you who knew that that was going on there. The reason that we've chosen that engine is it optimizes things so that only the minimal number of patches needed get onto the system, which helps reduce the reboots.
 
So rather than doing package by package and not having a relationship between those, this engine knows that, oh, hey, if you've approved 10 patches and 7 out of 10 of those are in the same superseding stream. Meaning each one of those was replaced by the newer one, we only need to push the latest of those seven patches, so really, in reality, only three patches would need to be delivered.
 
Things like that help reduce the number of reboots, but also, you know, disabling the patch level, you know, switches to say the patch doesn't control reboot, it's controlled at a higher level. We can install things in order and in a way that we can reduce reboots as well. It starts to get into kind of a convoluted state of the machine kind of situation at that point. I've taken like a Windows 7 machine out of the box, never been patched before, and loaded it up with third-party apps, and patched dozens of applications that were very out of date, all in one shot with a single reboot. I've had a machine that's been updated much more frequently but had a number of user state kind of issues happening on there so I did some installs, uninstalls, and things like that.
 
There's a point where, like, on a Windows system you can hit a point where Windows installer will just say, "No, I'm done. You need to reboot me before I'll continue on." So circumstances like that can happen. We've had some cases where we've introduced a feature called pre-deployment reboot, if you've got cases like that that happened frequently, doing a reboot before you install patches. Make it so that you'll get down to two reboots only. And then we have the Windows 10 branch upgrades. There is no way to control how many reboots that's going to need, that's completely dependent on the branch upgrade. So those ones are completely out of our hands, but we try to optimize as much as possible to try to reduce the number of reboots.
 
Todd: I might mention, Chris, that this just came up in the last month, Microsoft released an update on...especially on the Office side around reboots. There are certain upgrades right now where Office has to reboot twice, and with one of those updates are not even notifying individual users that this update is coming and the reboot is coming. So we can send you a link to that web page or you can do a search on Microsoft's TechNet and find it. So it's something we've seen recently.
 
Chris: So Rick had a question around...more of a product-specific question. So Rick, you're on patch for Windows it sounds like, and it sounds like the way you're scanning you're running into some performance issues. So that's one where... Why don't you shoot an email to myself and Todd, we'll probably need to ask a few questions to figure out exactly what you're doing, and we could probably help you optimize that a little bit better. Based on your configuration you might be making the engine do more than it needs to, to get to the point where you're updating things. So let's talk about that one, reach out to us and we'll follow up with you.
 
I think we're getting down rather into the weeds now and we are already 20 minutes over, so for those of you who do have more product specific questions, you know, Kevin, you know, Rick, a few of you, I would recommend reach out to your TAM if you have one, or in the case where, you know, the product manager for your product line, Eran for the patch for the Endpoint Manager side. Todd or myself as well on, like, patch for Windows, patch for SCCM. Go ahead and reach out to us and we can answer any more direct product related questions.
 
I think... Oh, there was another question that got sent to Erica as the host about any support coming for Windows cluster patching. To answer your question there, Luke, using that API model that we talked about, there is actually...so like in the patch for Windows API document, one of the examples I created in there was specifically interfacing with not only our APIs but also the windows cluster APIs to be able to patch a SQL cluster. So that's an example script in there. Functionality like that is ideally suited for that automation level.
 
If you reach out to me I can send you the link for that particular document. But again, it will depend on which product, if you're on Endpoint Manager we probably need to get you in touch with Eran and get some more of the API documentation as it becomes available. So you can see how to automate those steps more. Now, we do have...and I try to avoid getting into the whole product pitch here, but we do have an addition to our product family through the RES acquisition that happened late last year.
 
RES has an automation platform, this is being expanded out to support many of our products, so we're creating connectors for many of the different products. There is a connector already for the patch for Windows product. There is another connector in the works right now for Endpoint Manager. So those connectors can interface with our products using those APIs we've talked about and can help you to automate things like this. Standard editions of that automation product that have our own product connectors available there are available to our customer base.
 
Right now it's kind of as a per-request basis, but as we get to the point where it's more fully integrated we're gonna be expanding that out and informing more and more people of that. So, you know, for those of you who are interested in looking at our automation platform, that's an option that...reach out to your rep, reach out to your TAM, or even the support team, and ask the question, we're glad to get engaged and talk to you about what we have available there.
 
All right, I think that's a good point to wrap. Thank you, everybody, for joining us, here. Again, I hope we got to most of the major like patch related questions that, you know, weren't too product specific here today. And thanks for joining us.
 
Todd: Thanks a lot.