Developing Best Practices to Patch Management: Technical Webcast
April 25, 2012
In today’s complex networking environment using patch and vulnerability management as the principal component of your risk mitigation strategy, and taking prudent measures to establish a best practices approach, can help reduce costs and risks in the long term. Patch and vulnerability management continues to be the first and last line of defense against existing and newest exploits. With the sophistication and sheer volume of exploits targeting major applications and operating systems, the speed of assessment and deployment of security patches across your complex IT infrastructure is key to mitigating risks and remediating vulnerabilities.
Chris: Hello, everyone and welcome to this Lumension in-depth technical webcast on developing patch management best practices. My name is Chris [inaudible 00:00:13] and I'll be your moderator for today's event.
Patch and vulnerability management is a principal and I'd argue core component of your risk mitigation strategy. It is the first and last line of defense against existing and new exploits laying the foundation from which your AV and other technologies work. As the sophistication and sheer volume of exploits targeting operating systems and other major applications increases, the speed of assessment and deployment of security patches is key to mitigating risks and remediating vulnerabilities and reducing costs.
In today's webinar, we're gonna take a deep dive into a best practice process for patch and vulnerability management developed by Lumension over thousands of customer engagements and get a real-world perspective from the trenches. This process which is flexible and simple enough to be adapted into your environment revolves around Patch Tuesday and includes laying the groundwork for successful patch process. The week before Patch Tuesday, the week of Patch Tuesday and the week after Patch Tuesday.
Before we move on, let me tell everyone that we'll be sending you a link to the slides and the recording of this webinar tomorrow. We'll also send you a copy of our best practices for patch management white paper. So be looking for that tomorrow.
We're lucky enough to have a distinguished panel to engage in this conversation. Jim Czyzewski is the clinical desktop support supervisor at MidMichigan Medical Center and I hope I pronounced his name correctly. He has been working in IT for over 20 years and has led the endpoint security team at MidMichigan since 2003. Jim's expertise in endpoint security is continually tapped to provide real world knowledge and insights at speaking engagements and patch management and desktop virtualization conferences.
Russ Ernst is a group product manager for vulnerability management products here at Lumension including Lumension patch and remediation. Since joining Lumension in 2008, Russ has expanded the platforms and applications supported for vulnerability remediation content including a migration of all supported Linux platforms to a new content architecture. Welcome, gentlemen and thanks for participating in today's discussion.
Jim: Thank you.
Russ: Morning, Chris. Thanks.
Chris: One housekeeping point. If you have any questions or comments at any point during this webcast, please submit them via the question tab at the top of your screen. We'll answer as many of them as we can during the conversation. We want this to be as interactive as possible with Russ presenting a generalized view of the best practices gleaned from the thousands of customer engagements and Jim's gonna be weighing in with his real-world perspectives on how these best practices work in the field. It's our hope you walk away from this webinar with some actionable ideas on how to hone your patch management process. So your questions and comments are key to guiding this conversation.
Now, we're gonna try to get the slides to match up with what we're doing here and I'm gonna try to conduct a quick poll of the audience if I can find it. There we go. So the question I have for you is how would you describe your patch management process?
Is anyone seeing this? Okay, there seems to be some votes coming in. This is great. I can't see it on my screen but we'll hope that everyone else is seeing them. So the options we give you here are robust, operational, modest, ad hoc or kind of to the purpose and then nonexistent. If you have any questions or comments on that, please use the questions tab so we can follow that through. All right, great. I'm seeing votes coming in so obviously everybody else can see it even if I can't. Okay so...
Before I hand it over to the panelists, I wanna set the stage a little bit. Why we're here today. The endpoint is the primary entryway for malware today. We know that it's under attack from physical attacks via rogue USB devices to network attacks via viruses and malware and of course, the vulnerability problem of applications which only seems to be escalating every day. So endpoint is vulnerable and in fact, John Pescatore from Gartner says that 95% of all exploits go after misconfigured and mismatched endpoints. So only 5% of endpoints vulnerability comes from so-called zero-day malware.
Recent data breach study of 500 breaches found that 90$ of exploits used for entry into the endpoint had patches available for six months or longer. The same study went on to point out that 50% of systems have 10 or more vulnerabilities for which patches are currently available. And these vulnerabilities are increasingly in the applications. It's not a Microsoft world anymore. Recent study shows that a typical 50 application endpoint...only 20% of the vulnerabilities are in the Microsoft products including the OS and the applications. The rest come from third-party apps and what's more, you need to have 12 different updating mechanisms to keep up with them all and you still wouldn't have all the visibility and assurance that everything is okay that you'd like to have.
And that's why we say that patching is the cornerstone of your defense-in-depth protection strategy. We're seeing a new endpoint security model now. We're patching configuration as the bedrock upon which all other technologies rest.
So what are the benefits of a well-designed, well-implemented patch and vulnerability management program? Here's how I see it. Every piece of malware that gets into your network costs you money and hits your bottom line via the impact of everyday operational costs and wasted bandwidth. And that's not even looking at the cost associated with a data breach. So when we improve our patching process, we reduce the target area for the bad guys to aim at and what they have to hit which not only protects your data but increases your operational efficiency and improves your cost effectiveness which allows you to shift your focus of your manpower and budgetary resources to other strategic business initiatives. The sorts of things that CEOs and CFOs and folks like that really understand and care about.
Let me close by saying that we have no single...no single technology is a silver bullet to magically protect against malware. It takes several overlapping technologies working well together to protect against today's threats. And this defense-in-depth approach to your security does not begin and end with AV but rather it starts on the foundation of a solid patch management process. Okay. With that said, let's close the voting and view the results.
So what we're seeing is that a vast majority of you all out there, 75% say that your current patch process is either operational or modest. With only 5% saying robust or nonexistent. How does that match up with your experience, Russ?
Russ: That's actually pretty encouraging news to see. We really like to see that companies do have an operational aspect of their patch management process. Fourty-four percent is still a little bit low but it's good to see that the folks that are on the line here take this seriously enough to get this into an operational standpoint. How about you, Jim? What do you think about those numbers?
Jim: I'm surprised that it's not greater, to be honest with you. Patch management has been an integral part of I think security for a number of years. We first realized the need for that almost 10 years ago now and the number of attacks that come against existing...where existing patches are available, it was so high back then that I think I would've thought that maybe more people would already be on a patch management solution by now.
Chris: Yeah, absolutely. Well, hopefully, this webcast will help folks move it to the next level. Okay, so let's get into the meat of the presentation. How to develop and implement a solid patch management process? Russ, take it away.
Russ: All right. Thanks a lot, Chris. So as Chris mentioned, we're gonna talk about the patch management best practices in four parts and that's gonna be laying the groundwork before Patch Tuesday, on Patch Tuesday, and after Patch Tuesday. And we understand that every company's patch management process is going to be a little bit different. What's important about these best practices are that it's number one, it's a repeatable cycle. Number two, it's based on calendar events and in this case what we're saying is that we're gonna be talking about best practices based off of Microsoft's Patch Tuesday and really depending on each organization, you can choose a different calendar event but what we're saying is important is that it's repeatable whether you're doing it on a 30-day cycle, a 90-day cycle, quarterly, whathaveyou, but it's repeatable on that calendar cycle. Third, I say that it's iterative. It could be tweaked based on what's learned from previous patch cycles and finally, it's measurable. And so, by putting a process into your environment and documenting that process for the organization, it's really the best way to communicate the importance of that patch process to the rest of the organization to both the C-level employees as well as to those folks that are gonna be impacted by deploying out patches to your environment.
So with that, let's go right into the next section here that we're calling laying the groundwork and really, this section is all about gaining an understanding of the machines under management and preparing that patch management process. So at a high level, this really means identifying those systems to be managed and then defining your patch rollout plan and finally training your organization on the patch and remediation process.
So the first step we talk about here, laying the groundwork is discovering assets and, you know, there is all different ways that you could discover assets in your environment. You can identify hardware and software on the network, you can categorize them by platform and application, you can categorize them by department, but really this first step is gaining an understanding of the machines that you choose to manage in your patch and management process.
And so, we know that there are many tools you can use to help you achieve this but the important part is to know it's out there. it's gonna be helpful to start with a network topology if one exists but we actually find that it's better to start by scanning your network and see what's actually connecting to your network.
So some tools actually even have the ability to schedule recurring scans of the network. We think that's a good idea and when first rolling out the patch management process, it may be best to set up those scans more frequently but as the process matures, it may make sense to have those scans only on a monthly basis or so.
So in the next part I wanna talk about agent maintenance and in this case, we actually prefer to have an agent on the endpoint that you're choosing to manage but this step is all about deciding which machines in the network are gonna be managed by the patch management process. So at this point, you've already found out what's connecting and...but at this point, you're saying, "Here's what I'm actually going to choose to manage as part of my process."
So Jim, when we talked a little bit before, you had said that you have some logging scripts in your environment. You wanna talk a little bit about that?
Jim: Sure. The way we get our agent out to our endpoints is through a logging script. What it essentially does is it checks for the existence of the agent. If the agent does not exist, it will install it automatically during that process. But of course, not all endpoints can have the agents installed and then there's also some that do not run the logging script because they may not be on our domain for whatever reason.
So we also do the asset identification component to identify those agents that should be managed by Lumension and get that agent installed to them and also, we need to peruse through that list to make sure that we are not touching any FDA certified devices. Being in healthcare, there's ultrasound machines and various CT scanners and so forth that are running on Windows operating system but yet we can't install other software on them. It has to be approved in as it is state by the FDA. So we can't be installing patch management agents or antivirus agents...or antivirus software on them. So we kind of have to rely on the vendor to take care of that for us.
Russ: Yeah. So we're definitely in favor of an agent-based solution especially when it comes to managing Windows machines in the environment and really so that we can be sure that the machine is managed regardless if it's in a workgroup or if it's across domain or even if it's spun up as a virtual image. So I'm actually curious for our...people that are on the call here too about your experience versus agent-based or agentless based experiences and feel free to put your questions into the question panel.
Chris: So Russ, one question's come in. The question, why not deploy the agent via group policy or with group policy?
Russ: We support that as well and again, we also understand that group policy is a very powerful way to manage policies in a Windows environment but there are also...with group policy you are limited to what's within that specific domain. We have a lot of customers that are managing multiple domains in their environment and having the centralized management infrastructure or centralized console to really understand which machines are being managed from the patch management process is critically important. But GPO is a very powerful tool and we approve of that rollout method as well.
Jim: And I'd like to comment as well. when we first implemented it was PatchLink [SP] back then which is now Lumension, we used multiple methods to get the agent out to make sure we hit all our endpoints. We monitored it quite regularly to make sure we were getting all of them. But like Russ said, we do...GPO is one of the options that is used. The logging script's kind of what we use now to catch any new clients that may have gone out and get them...the agent installed on those.
Russ: Very cool. So with that, I wanna move on to classify your value and risk. So we mentioned a little bit about your network topology and building out and understanding of what machines are in the environment. And at this point, in building out your process, it's really important to determine who are most critical to protect based on the assets in-house and or the function that they provide.
So you can define the level of risk by criticality of the system or how prone it is to the attack but what's important here is really it's gaining an understanding of the machines that are in your environment so you can start to define what your pipe rollout process is gonna be, you start to think about some test machines for your test deployments but this is about classifying the value and risk to the organization. I'm really glad, Jim, you brought up earlier too some of those FDA-approved machines but you just need to have an understanding within your own environment what your patch process is to your own environment.
Jim: Yes, and by monitoring those endpoints that we don't...we can't control with Lumension, we're at least able to see where their vulnerabilities are and we can chase after the vendor and make them...get them up-to-date especially if we were to...there's a known exploit within our environment. We wanna make sure that the vendors are aware so these medical devices are not susceptible to that exploit.
Russ: That's a great point. You know, you almost...as the internal advocate of your path process, you become the proactive security controller out to your own vendors to say, "Hey, we know about this exploit that's out there on these machines. We know that you're running this OS for these machines, you know. What's your timeline for getting these rolled out?" That's great.
So at this point, we wanna talk about establishing your workflow and establishing your groups. And this is all about determining ownership, seeing which permissions are needed, the responsibilities for the thread identification, starting to think about the testing and remediation process across the security IT and business units. So understanding who's gonna be doing the patch management, who's gonna be responsible for it whether it's operational or whether it's your security team. Then starting to set up the groups in your environment to say if you wanna set them up by desktop or server or you wanna set them up by functional unit, whether it's for your accounting group or your development group. You wanna talk a little bit about...Jim, about how you have your environment configured?
Jim: Sure. We wanted to keep our...from a group perspective, we wanted to keep it simple, keep it as simple as possible understanding that the more groups you have, the more management it requires. And we also went into it understanding that if we do need to segregate any groups out based on business unit, we could still do that but we wanted to approach all business units as equals as much as possible. And we've been able to maintain that group structure since we really began, there's been remote instances where we've had to segregate a business unit out but it's been very rare that we've had to do that. Our groups are structured strictly based on test groups for right now and we'll talk more about that in the next slide, so.
Russ: Yep, that's great. So and again, this is also important where within a good patch management tool, you'll usually have some role based access controls so you can set up those groups and you can set up even the people who are gonna be logging into the...the users who'll be logging into that console so you can make sure that they have an understanding of what those individuals can do while they're in there whether it's from an executive view where you really want them to have more of a reporting view to what's in there or down to the actual administrator who's gonna be rolling out the content.
Jim: We use three...I'm sorry. We use three roles within our organization. The administrator, who pretty much manages all security patches and deployments and we also have a desktop role. We use Lumension, patch mediation solution for software deployments as well. So we wanna be able to give that ability to our desktop staff when need to deploy a particular software package out to a desktop, they have the ability to go in and do that as well. And it goes down to the operator role who just has the ability to look at a particular computer's vulnerability assessment at that point.
Chris: Yep, yep. So I have question from the audience, guys. Is it possible to have group, custom group created for systems that contain dynamic members added automatically? So an example would be all servers that are added to an OU specific group.
Russ: So within the Lumension patch remediation product itself, if you're talking about on a system level like that, it is supported. So from a system level, if you're talking about machines that come onto the network that are server based or if they're by OS then those are automatically added to those system groups. From a custom group perspective, that's not currently supported but that is a...it's a great idea for how we could have machines come into the network. What's your experience, Jim, with adding new machines to your groups?
Jim: There was one point and I don't know if it's supported in the newest version. I turned over the admin roles to one of my staff so I haven't worked with the newest version as of late but there used to be a way in the executable where you could actually assign it to a custom group. So we always assign it to a default group, default deployment group that we use and if we happen to notice one that needed to be reassigned to one of our test groups, we'd be able to go in and pull it out and move it to a different group.
Russ: Very cool. So next I wanna move to...and Jim, you had mentioned about identifying test groups and this is really one of those critical aspects of laying the groundwork is starting to identify which machines are gonna get the patches first and then identifying those test machines. So, Jim, I wanna turn this over to you for a little bit to talk about some real-world experience you have with your testing process.
Jim: Okay. We didn't really have the...when we first started this process, we didn't really have the luxury a lot of people talked about of having a test environment internally. So we had to figure out a way where we could test it on the multitude of applications that we have in our environment with having as little impact as possible to production. So we decided to go with three levels of testing. Our first test group, we have our IT desktop folks in this group and it's essentially for them to validate that it doesn't really screw up Windows in the process. It's something close to home to where the deployment's occurring from since we're...from within IT and the people that are getting the deployments are able to resolve the issues on their own, they know what's happening. So once we go through that process, we then move it on to a second level of testers and what we do is we select a super user from all our applications and have them in this group so they can go through and make sure that it doesn't impact their application during normal day-to-day use.
After a week within that group, we have a third testing group which is essentially a broader user scope with the same apps. This way we have more eyes on the deployment, maybe use different functionality from software than maybe the super user would've the first time through just to make sure there are no problems.
And then we also have them...each level of testers responsible for reporting back on our intranet. We have a database set up where they can select the current month's patch and then report back any problems or if there are no problems found. Over time, the test users have decided to not report when they don't have problems as most people will probably experience. The only time we ever really see anything in there is if there is a problem reported. It can be anywhere from that they happen to notice that their computer may have been slower which we may rule out as any correlation to the patch that was deployed to where it happened...it totally breaks their application. We do also encourage our users to report back to our call center if it impacts them significantly so we can get on it right away and either roll back the patch or remediate the problem in another fashion.
Russ: Yeah, that's so important. So what...to kinda summarize those points and I think those are really excellent points...really, as the best practice we recommend building your test groups based on a representative sample of machine type and the applications used in the environment. Especially including those endpoints that have custom built or may be industry specific applications in your environment to making sure that as you do your rollout to those test groups, you're getting...like that representative...like you mentioned, Jim, getting enough eyeballs on those machines to say that, "Yep, this is really gonna...it's gonna work for my environment specifically and it's gonna...it's much better for your full rollout."
We've got one more area in laying the groundwork and that comes into staff training and this is really all about training your applicable staff on the vulnerability of monitoring and remediation techniques but it's also about training your...the organization about the importance of the patch management process. So we talked about here is there are some...definitely, there are some product specific tools that you could use for training your internal IT staff but it's also about, you know, building your own internal learning portals or building your own intranet site so that you can have all employees go out and really find out why it's important to do this and that's building that confidence and that trust within the organization to say, you know, "We're doing this to protect our machines, to protect our data and to protect ourselves from all that...the malware that's out there."
So with that, I'm gonna move onto the next phase in the patch management best practices and that's before Patch Tuesday and really, this section is all about preparing your environment for that patch deployment cycle whether it's monthly or quarterly, whatever works for your environment and then include some industry research on what is expected to be released by Microsoft and other application vendors and assess the impact of those planned releases to your managed machines.
Chris: So before we go into that, Russ, let's open up another poll and it should be open for everybody. The question is what group within your organization drives these patch assessments? Is it the security group, network operations, general IT admin, executive IT management or other? And as I mentioned before, if you have some comments or you wanna fill in with other means, that'd be great. Just use the question tab. We'll leave this open for a little bit and let Russ carry on.
Russ: Thanks, Chris. So the first part of this before Patch Tuesday is really about preparing your resources. So this is scheduling your resources and reserving your downtime for your servers if you are patching servers in this case. And this is all about allocating the IT resources for Patch Tuesday, reviewing the patching needs of any internally developed applications and or custom patches and consider deploying those patches as part of the monthly patch cycle.
Really, this is all about making sure that you have enough manpower in place and that you have the downtime scheduled in your environment if that's required in order to effectively get your patches rolled out.
And also, this week before, you wanna be thinking about the preannouncements. You wanna be...as an IT security administrator, you wanna be thinking about those resources you can look to out on the Internet to say, "What's important? What is really gonna be impacting my environment?" You know, a couple of things that we recommend is the SANS Storm Center is a great resource for finding out information. Definitely, the week before Patch Tuesday we should all be aware the Thursday before Patch Tuesday Microsoft does a good job of sending out their advanced notification which gives a lot of background information of what's going to be released the following Tuesday and then there's also the Lumension Endpoint Intelligence Center where you can learn more about the different threats or vulnerabilities that are available.
Chris: So a couple of questions from the audience. The discussion is kinda framed around both Microsoft and third-party apps and yet we're devising this...or we're discussing this in terms of Patch Tuesday. What's the correlation or what's...why is it that you decided to go that route?
Russ: Oh, that's a great question and really, it's all about...like I said at the beginning, it's all about getting onto our repeatable cycle. And some companies will choose to release their software updates for third-party supported applications when they're made available and that's totally fine. What we're saying is the best practice though is that if you get onto a repeatable calendar based cycle that you can include all of your third party and your Microsoft updates within the same deployments. And we found that customers will do one or the other but what we're saying is that it's most important just to have that repeatable process so that you can have, you know, that time that you're sending out those updates to your environment and include your third-party applications. Jim, what's been your experience with updating your third-party software?
Jim: We do the same. We try and schedule it all within one...based on the Microsoft Patch Tuesday event just so users have a set expectation and it minimizes the impact to users as well. They know that they'll receive their reboot notifications on a...only once a month unless for some reason a critical high priority patch becomes available and there's an exploit available or out in the wild that requires us to move a little more quickly on it. We even try to schedule our other software deployments around it as well.
Russ: Yep, absolutely. So next we're gonna talk about confirming that your reporting is up to date. And so again, this is all about...in that week before, you just wanna make sure that you can review your last deployment reports, make sure all of your computers are still being regularly scanned, validating that your application server is actively communicating with...and getting your supported updates and this is all about just ensuring that your systems are operational and working as designed before the Patch Tuesday event.
And then finally, as part of this section, we talk about deploying any missing updates or any prerequisites. So this is really about...and again, understanding of the machines that you have under management in your environment and that if you have some missing service packs or if you have missing updates or any Microsoft Fix-its or any other non-security updates that you wanna go out and have your machines at a known baseline level prior to the Patch Tuesday, it might be a good idea to do some of those pre-deployments before the actual Patch Tuesday gets out. All right. So actually, let's go ahead and I'll turn this back over to Chris so we can go through the votes from the last poll.
Chris: All right, great. Thanks, Russ. I'm gonna stop the poll there. Got a good number of folks voting. So the question was about who's involved in your patch assessments and what we find is 34% of you all have a security group that's involved and then 37% have general IT admin. Only 6% have executive IT management and then 13% network operations. What do you think of that, Jim and Russ? Go ahead, Jim.
Jim: Personally, we use...we just...our security team leader assesses the patches and deploys...and emails [inaudible 00:34:13] security team with the assessment of the patches that are available and makes a recommendation based on that as far as what we're gonna deploy and the schedule for the deployment.
Russ: Yeah, this is a...this breakdown here is...it's actually not uncommon from what we've seen from our customers too is that we typically see a general IT operational side as well as a security side. I'm actually a little bit surprised that there's a...that security's a little bit less than the general IT administrator but there might be, you know, maybe general economy reasons for that and people are looking to downsize areas. But I would say that having those two being about the same, 37% for the general IT administration and 34% for the security is not uncommon and the fact that those are so similar, that's typically what we've seen with our customers too.
Chris: All right. Thanks, guys. So this next section is...we call on Patch Tuesday and this section really outlines the steps to prioritize the security patches released by Microsoft and your other application vendors and then actually do those deployments and deploy those patches out to the machines managed in your environment.
So the first step behind this is to really study the information and study the security briefings behind those release patches. And so, I've got a couple of...there are a couple of sources here of course that we could check. Again, the Microsoft Security Resource Center is a great place to check for Microsoft releases but definitely for other third-party supported vendors, Adobe is a great example. Adobe has a pretty good security site. You wanna check that out as well. And this is all about getting an understanding of exactly what is being remediated for that vulnerability for that patch. And then understanding what that impact is to your environment.
So, Jim, you talked a little bit about your security administrator sending out a notification. Is that also triggered by studying the information that's available from when these...when the patches are released?
Jim: Correct. Leading up to Patch Tuesday, we get the notification from Microsoft as far as what's coming out and doing the assessment not only on those but also looking at what the other third-party vendors have out there at the same time so we do try and revolve it around the auto deployments on Patch Tuesday.
Chris: Yeah, and some of the things you wanna be thinking about when you're studying this information and you wanna be considering when you're thinking about the impact to your environment, you wanna be thinking about the bulletin severity for each of the bulletins and each vendor has their own version of a bulletin severity. Whether or not that vulnerability was known or publicly disclosed at the time of the release, did the vendor know, did Microsoft know of any active exploits at the time of the release and then how easily can that vulnerability be exploited once that bulletin's been released.
And, you know, there's actually some...again, Microsoft does a good job of presenting a lot of this information on their Security Resource Center and other vendors have started to take note of this. Adobe's catching up. Adobe's doing a pretty good job but you also have other third-party vendors like Oracle's Java content that's out there. They're starting to do a better job of providing better insights to the criticality of the bulletin that's being released for that vulnerability and then also giving a good measure of how easily it can be exploited and whether it's been exploited out there in the environment.
Yeah, there was a study done by an analyst group that had some really interesting results on what sources of information that their participants used and vendor email list, third-party email list and the US [inaudible 00:38:26] were top in that survey result.
Russ: So after doing all that research, then you wanna start prioritizing those potential patches. And this is all about...again, you've done the assessment about the impact of those patches to your environment. Now it's about which ones are the most critical, which ones have to get out immediately or which ones followed my normal patch and remediation process. And again, we talked about prioritizing based off of the impact to the environment but also you wanna think about what that threat level is, whether there are known active exploits in the wild, the risk of any kind of compromise to your environment and maybe the consequences of a compromise. So for example, if you have a vulnerability that's targeted to a server based machine and you know on that specific type of server class machine in your environment where you have customer data for example or you have patient data, you wanna make sure that that...if there's a known exploit for that kind of vulnerability, you wanna make sure that that's accelerated in your environment.
So Jim, we talked a little bit offline about an expedited process and this is really...I think where this comes into play...when you're looking at your prioritized patches, it kind of falls into a couple of buckets and one is it's gonna fall into my normal patch deployment process or it's gonna fall under the expedited process. Do you wanna talk a little bit about that?
Jim: Sure. Looking at...you know, we can look at threat level, the known [inaudible 00:40:05] of exploits, the risk of compromise and the consequence of a compromise as you mentioned. We don't take the threat level into a whole lot of consideration initially. We know Microsoft rates them between critical and important and whatever their lowest level is. Anyway, we treat all those equally. We just wanna get those out to make sure that we're patched regardless of what the threat level is. This way we can all squeeze it into the same Patch Tuesday event. But we do look at if there's any active exploits. We kinda start at step two in that list. Starting at the known active exploits and if there are, what's our risk and what are the consequences that go along with that?
Depending on how those rate through our discussions, we may decide to elevate that. If it's something imminent, we will definitely push that we have a 24-hour process built in as far as still stepping it through our test groups on a quicker basis. We contact the...we have a call tree set up to contact our testers and say, "Look, you have an hour to test this. Run this through as much as you possibly can in the next hour." And we're gonna push out to the next group until we get it out to the rest of the group within a 24-hour process. And we do expand that from a 24-hour up to a week if need be. Or if we feel the threat's there but not as...we're not as at risk as much as possible...other people may be.
Chris: So do you follow the same kind of expedited process if it's a like an Adobe patch or whether it's a Microsoft patch if you know that there are known exploits against that vulnerability?
Jim: Yes, absolutely. And I wanna add to that too we do proactively monitor for exploits because many times we'll start a normal patch process because there's no active exploits out there. So we'll start through our normal three, four-week cycle and if then something pops up maybe a few days later or a week or two later and we determine that we need to expedite it because of this new threat so then we evaluate it and escalate it beyond...based on that information.
Chris: That's great. Yeah, so we had a question about your different...using different processes depending on the level of risk and I think you kinda answered that one, Jim. But we have another question. What sort of approach to patching should organizations that are delivering streamed non-persistent desktops...so VDIs I guess. And should that match the same framework as your physical approach?
Russ: I think that's a fantastic question especially with how predominant virtual images are becoming in the enterprise today. And really, the patch management process is the same regardless of whether it's a physical or a virtual machine. Of course, how you actually roll those out's gonna be a little bit different but following that same process and going through that same criticality check and then doing the same rollout is gonna be...pulling those same processes is really gonna be about the same. What we say from a best practice in our standpoint is that you still need to understand the impact of those patches to those virtual machines and there are different methods you can use about patching those virtual machines but it's critically important to make sure that those virtual machines are up to the same patch levels as your physical machines because what we've seen too is that a lot of times you'll have...customers will have virtual machines in their environment that go for months and months that are being turned off and then they come onboard, they come online and they're months and months out of date from their patches and if any of those machines were to connect to the Internet or start browsing to some suspect websites, those machines are just prime targets to be exploited and then that's an instant foothold into your environment.
Jim: We all...we use virtual. I'm sorry. Go ahead.
Russ: Yeah, go ahead. What's your experience?
Jim: We happen to have a virtual desktop infrastructure in place with about 900 virtual desktops being used and we do use the same process. We have test pools set up so we have an offline mechanism to run the patches through prior to and into our production pools.
Russ: Excellent. So the next...the last part of this on Patch Tuesday or the week of Patch Tuesday section is to actually test and install the patches. We talked a little bit about this already about following that tough process, following your...rolling this out to your test groups, doing your stage testing. This is also about following your internal change control processes. You know, we talked a little bit about having that week before where you're making sure those change control processes are in place and...but this is really about...okay, now you know your patches have been released, you know you need to follow the paperwork with the certain deployment groups in your environment or if you have maybe a standing change control order with your...as part of your security rollout process. But this is all about that stage testing, deploying your applicable bulletins to your test groups, ensuring that those deployments are successful before rolling out to additional groups and then again paying special attention to the...any endpoints that have custom developed or internal applications or may be industry specific applications in your environment. Is there anything you wanted to add to that part of it, Jim?
Jim: Not too much. We do have a standing change. I don't know what we had to do for change control as we just supplied a one-time event and said that we'll be doing this on a monthly basis, explained our process involved and they just approved that process. So we had a standing item. We don't have to go back to change control of every item beyond that. And as I think I said before, we step it through our three test groups one week at a time and they report back to us with any information that may have occurred during the testing process.
Russ: Very good. And so, I think with that, we're gonna open up another poll, Chris?
Chris: Yeah, let's do another poll. Question I'm gonna put...let's see. Hopefully, you guys can see this. What metrics do you use to assess the success of your patch management process? So answers we have here are time to patch, the adherence to policy, the number of systems or percentage of systems in compliance, doing vulnerability scans to assess vulnerability or other. And as always, if you answer other, feel free to put something in the question tab to explain that one. So we'll leave that open for a little bit but before we move on I'd like to interject a question from the audience. How long are you allowing machines not to check in either through manual intervention or wake on LAN? Why wouldn't you force a check in? So I think that's directed to Jim but [inaudible 00:47:39]
We can't force a wake on LAN because, for many years now, we've used an assigned node and wake on LAN really relies on a hardware MAC. Otherwise, we'd be using that on a lot more frequent basis. We've tried doing this in the past and we have not had any success. We have since changed our process of using assigned nodes to going with the hardware MAC address so we can start to...taking more advantage of a wake on land function.
And as far as monitoring, agents not being online, we have several training rooms. So sometimes these training rooms remain dormant for months or so at a time. So we understand based on their computer ID what devices those are so we don't proactively go after those. Knowing that they'll catch up the next time they do get turned on. But as far as other agents, we do try to minimize their check-in points of less than 30 days.
Russ: Yeah, that's what I was gonna mention too, Jim, is that we recommend that you have machines check in within at least 30 days and...but it's really...it's all about knowing your environment too. I mean, you make a great point about knowing that you have training rooms that are in your environment and knowing that those can be...that are gonna have to catch up at a certain point and it's knowing about what those areas are within your environment so can normally go out to them and proactively make sure they're fully patched.
So with that, that takes us to our last group and that's after Patch Tuesday and this is really all about assessing the success of the patch and remediation deployments in your environment and then also monitoring for any machines that come online during that time and following your success metrics. So the first thing we're gonna talk about is looking at your deployment history and that's about maintaining your accurate records of all the patches that have been deployed and then validating that any necessary...any reboots occurred and that your endpoints don't...or whether or not your endpoints do require a reboot. So this is all about, again, monitoring the success of the deployments and Jim, do you wanna talk a little bit about how you guys monitor that during the...or just after doing your patch rollouts?
Jim: Sure. We just use the Lumension product to keep track of how far along the deployment has gone. Usually, within a couple of days, we'll reach 95% of all our devices. And we would...understanding that most of that 5% is probably infrequently used devices that are probably left set off or turned off at the time.
Russ: Yeah, and it's really important too that if you are monitoring why...if you're monitoring any failures, any deployment failures to start...to have that visibility so you can dig into exactly why the deployment...what didn't go out there, were they running something else at the same time, were they...for whatever reason, if they initiated a reboot in the middle of the deployment or if there's any other reason why you had any machines that didn't deploy correctly so you can get that fast follow up.
Jim: Correct. We haven't really had any incidents lately. We always monitored for deployment failures and if we saw an excessive number of failures in a particular patch, we would do a deep dive into it. Otherwise, what we do now is we just let it go, catch down the baseline deployment and if we notice that it still fails on a particular device, then we'll look further into it.
Russ: So that really gets into some of the metrics that we recommend and one of the metrics that we've seen a lot of our customers use is calculating that time to deploy and this is all about measuring how long it takes to get all your servers, desktops and laptops fully patched in your organization and this is a great measure...or metric to measure against so you can remain vigilant for...against those...maybe some of those laptops that connect in periodically or some of those VPN connected systems that may connect days or weeks after the initial deployment. So Jim, do you wanna talk a little bit about what your...the metric that you guys use internally for success?
Jim: For the most part, it's the time. The time it takes to get the patch out. Our acceptable volume is 95% and we've maintained that since we started doing patching however long ago that was now. And we've never had a problem reaching that percentage within a week's time.
Russ: That's excellent and...
Chris: Why don't we look at the results from the audience? I'm gonna go ahead and stop it and look at the results. What we see from the audience, 51% are looking at the percentage of systems in compliance. Next, they do vulnerability scans at 22%. That's very interesting to me. And then finally, the next one is time to patch, 15%. What do you guys think of that?
Russ: I think that's pretty interesting and, you know, what we're...we've seen systems in compliance. That's an interesting stat. So I would also think that by following up with vulnerability scans, that's actually a great way to do a third-party validation that the machines you already have patched are now no longer vulnerable for what's being...for what's known in the industry.
Chris: That's actually...that's a good lead-in from what we talked...we're talking about next is this monitoring for compliance. And this really...this section is all about how do we make sure that those machines that were not online when we were doing our patch rollouts pick up those required bulletins once they do come online. And one of the tools that we talk about is a baseline and making sure you have a baseline of the required patches that have to be installed for every machine in your environment and usually these are the required security updates. And this is really all about when a machine does come online that it's gonna pick up this baseline of required bulletins and deploy them right away.
So Jim, do you wanna talk a little bit about how you guys use baselines? Because when we talked about this offline, I was just really intrigued about how you have this really built into your process.
Jim: Yeah, this was built in when we first started our process as well is that after 30 days of having the initial patch available, we'll put it into our baseline so we can stop the deployment. We don't have to keep the deployment continually out there checking and running for these computers and we'll just move it over to the baseline. This protects us...for those computers that did not get it during the patch process and it also helps us get our base images for PCs caught up as well so when we do deploy a brand-new computer out there, it'll catch up with all its patches that it hasn't...that weren't built into that base image.
Russ: That's great. So the last part of what we wanna talk about here in this section is this after Patch Tuesday is really the continuous improvement. I mean, this is all about making sure that...is there any lesson that's been learned? This is saying that this is an iterative process. What can we do better next time? So this is all about reviewing the effectiveness of the Patch Tuesday remediations. Did you get any increase in IT support tickets? You know, is there any kind of issue that was...that resulted from the rollout of the bulletins? You know, sometimes what we hear too from our customers is that if you have the same end user that's reporting a lot of tickets about the rollout, maybe they're a good candidate to be in your test group the next time, right?
So you wanna think about changing your test groups for your rollouts possibly and then it's about your metrics improvement, you know. If you're...if you only hit 80% of patch machines within a week of the deployment, what can you do to get that number higher the next time and start...continue to raise the bar. So is it about...you know, do you need to modify any system settings, do you need to think about WAN optimization, do you need to start thinking about putting a proxy in place for your deployments? And start to look for those computers that didn't receive those updates at all or look for those computers that took an unusually long time and...you know, again, this is all about knowing your environment. So maybe you know that that's a training room that's out there or is there a new batch of machines that hasn't come on in a while that you need to start proactively going out there as a security admin to go out there and say, "You know, how come these machines aren't coming online really? Who owns those machines and how come they're not coming online to get their bulletin updates?" So with that...I mean, Chris, looks like maybe we had...you wanna move right into the Q&A section?
Chris: Yeah, I know that Jim, you have a hard stop at the top of the hour and I certainly want to take advantage of your being here. We're gonna keep on the line to answer questions after you have to leave but let's dive right into it if you've gotta a few minutes still.
Jim: Yep, I've gotta couple more to go.
Chris: Okay. So one question came in. What's the best way to measure compliance? Also, is anyone having any problems with their patching on those devices that hibernate and what's the best way to resolve it? So I think you kind of addressed that a little bit but maybe it deserves a little going over again.
Jim: Yeah, we...
Chris: How do you guys...
Jim: Go ahead.
Chris: I was just gonna say how do you guys monitor for compliance in your environment? Do you use any other third-party validation tools outside of the patch and remediation tool?
Jim: We use the patch and remediation tools exclusively for this. We're able to see...I mean, if a particular device didn't get a particular patch, you can look...use Lumension software to be able to go in and look at a particular patch and see what percentage of those devices have actually receded and drill down further to see what particular devices those are. So if we really feel there's a high threat...a vulnerability out there...maybe it comes out two, three months after the patch was initially deployed, we wanna validate that we are 100% compliant with that patch. We can go into the Lumension product, look at that...the patch and see who's still left if any.
Chris: So Jim, you have a...I mean, it sounds like you have a really...a good control over the security operations in your environment with patch really at the core of that defense in depth strategy. Do you use any other tools to help with that defense-in-depth strategy?
Jim: Yes, we also do...as I mentioned, application control product which gives us another...an additional level and gives us a little more breathing room I think as far as having to get the patches out knowing that any exploits that may hit our devices wouldn't really be able to execute since they're not an approved application in our environment.
Russ: That's excellent.
Chris: Great. All right. One more question. It's getting short on time for you, Jim, but one more question before you leave. How do you patch critical production servers that are running 24 by 7? Is that in your area?
Jim: Our server management team handles that. we do have maintenance windows for all our servers but we also do have to address the concerns of the vendor. We have to validate that the vendor software is validated with that patch. The last thing we wanna do is patch a system where they're gonna say, "Well, you put this patch on here and we didn't authorize it within our system which is why your system is now not working particularly." And especially in healthcare and its reliance, various government approvals for these various software products.
Russ: Yeah, again, this is all about really knowing your environment and assessing the risk of that vulnerability to your managed machines and a lot of the customers that I've spoken with that do manage server class machines...there is...you know, there's a different level of questions that they ask when they're doing those vulnerability assessments. You know, is that server...is it internet facing? Is there any...what's the risk of actually being exploited, of that vulnerability being exploited on those machines, you know? Is it on a quarterly cycle in that case? If it's really...it has to be up 24 by 7, there's other things that those customers will use whether it's a load balanced system where they can take one machine offline and still maintain the 24 by 7 uptime for the overall system. So there's a lot of other considerations that come into play but definitely having that process in place where you are monitoring it on a more frequent or on a timely basis. That's really what's most important here.
Jim: I do have to go right now, guys. I apologize.
Chris: Oh, well, thanks very much, Jim, and for our audience, we're gonna keep the phone lines open and continue to answer questions as long as they come in. So thanks again, Jim.
Russ: Thanks, Jim.
Jim: Thank you. Thank you. Bye.
Chris: Okay, Russ. Let's take some questions from the audience. Let me...before we get into some of the details, let me just tell our audience that there is more of information available on many of the topics that Russ and Jim talked about today. You can get a free vulnerability scanner tool from our website. Run that and see what vulnerabilities exist in your environment. So some of that post event management that you were talking about, Russ, you can deal with via that tool.
So let's get into some of these questions. Why is it a best practice to apply third party software patches at the same time as Windows updates? Doesn't that create more variables? So presumably, more avenues for failure or something.
Russ: Yeah. From what we're seeing from the best practice standpoint is that you really just have this repeatable process in your environment. You don't necessarily have to base it on Patch Tuesday but this is one area where Microsoft has been really a great leader in the industry of saying that once a month on a predictable cycle they're going to have updates available for their machines that are still under...within their lifecycle. And you've got...actually seen other vendors sort of piggyback on top of that where you now have Adobe that's trying to be on a more repeatable quarterly cycle and it just so happens that the dates that Adobe chooses always happen to fall on a Microsoft Patch Tuesday.
Now, we also see that on a monthly basis, of course, there are vulnerability remediations that are released outside of Microsoft's Patch Tuesday but what we're saying is that within your organization, if you choose a repeatable monthly cycle, this is just what we've seen. Some companies will do it on a weekly basis, some companies do it on a quarterly basis but just choose that repeatable cycle and follow that. Then it makes sense to have all of your updates rolled out around that same time. And again, I don't wanna under stress too that the importance of...understanding the impact of that vulnerability to your own environment. And so, you can have an expedited release process.
So regardless of whether it's an Adobe patch, whether it's a Microsoft patch or whether it's a Java patch, if it's a zero-day vulnerability remediation that impacts a large number of machines in your organizations...for example, if it's a reader patch that's gonna be prevalent throughout your organization and that you know there are known active exploits against it, that doesn't matter if it's not a Microsoft patch. It still needs to go through your expedited process to get out to those machines.
Chris: And presumably also high-risk machines, right? I mean, if there's a vulnerability against the high-risk machine, then that would be expedited?
Russ: Yeah, absolutely and again, it's all about knowing your environment, right? So if you're a company that's migrated everything over to Windows 7 and you're on the leading edge of all those...the latest operating system and there's a Microsoft Patch Tuesday that comes out and it's only impacting XP, hey, you're free and clear. But that's completely different for another organization that might be still in their upgrade process and they still have a large number of XP machines in their environment. And so, they...you have to weigh the risk of those bulletins that are released to your own...the impact to your own environment.
Chris: Great. So next question. We have limited test servers and nothing dedicated to testing patches, zero interest from management in setting up regular patching because it might impact production availability. So the question is are there any tools for summarizing how far their environment is behind in terms of patching?
Russ: That is a great question and that's really one of the things that we pride ourselves on with the Lumension patch and remediation product is that being able to show to your executive management what...we call that patch lag or basically how long...how out of date those machines are. And so, you can run a vulnerability analysis report, show that to your superiors and say, "Listen, here is how far out of date we are with our managed machines. You know, these specific bulletins are ones that have active exploits against them." And you really start to build that case internally about the importance of having regular patch maintenance windows within your environment. It really is so critical. Getting back to the statistics, again, I think we were talking about earlier at the beginning of this webcast about how many of those exploits are really targeted at vulnerabilities that already have a known patch and the bad guys are really taking advantage of the fact that end users take so long to patch their machines.
Chris: Yeah, there are some studies that show that in today's environment, 72% of vulnerabilities have a patch available within 24 hours of disclosure. So that...I mean, right there you see that, you know, the vendors are taking this seriously. And that 77% of vulnerabilities have a patch available within 30 days. So, you know, they're doing their half...part of the task so...
Chris: Okay, let's move on to the next question. When you suggest measuring success based on the percentage of devices successfully receiving a particular patch or percent of devices not missing critical patches. So a little nuance in the metrics question.
Russ: I mean, again, that is a bit of a nuance to the metric itself and how you choose to measure the success of the end bulletins. It's gonna be a little bit tailored to your own specific environment. What we recommend as a best practice is having those machines fully patched regardless of the criticality and really that's because you could have a moderate exploit that's out there that, you know, maybe it's not the easiest way to exploit but if you put it on a lower priority and you never patch your machines for that, you're just opening up that window for the bad guys to take the time to develop that exploit and so that they can still target that machine. And really, it's all about maintaining the complete patch status of your environment. You might wanna think about rolling out the critical patches earlier but we still recommend as a best practice to keep those machines fully patched.
Chris: Great. All right. Next question. Let's see. A lot of talk about Patch Tuesday. We keep talking about deploying Microsoft patches. So what...is the purpose of the Lumension product to replace a working WSUS environment or does it work hand in hand? What added value is Lumension providing over a given...or a functional WSUS it's offering?
Russ: That's a good question. You know, Microsoft's WSUS tool is an effective tool but really what we're seeing with the Lumension patch and remediation product is that we're giving visibility across the entire organization, across all third-party applications beyond just Microsoft. It's no longer just a Microsoft world out there and it's really about knowing the complete picture from a security standpoint, from a security patching standpoint of your environment.
And really, the benefit of being able to provide that visibility into the third-party application exploits that are out there in addition to, you know, side by side with your Microsoft updates is really...it's a very powerful combination with Lumension patch and remediation product.
Chris: So kind of as a follow-on, there's a question do we have a Linux agent for Lumension patch and remediation?
Russ: Yeah, absolutely and that's one of the strengths also of the Lumension patch and remediation product is that you get a view of your entire heterogeneous environment. So you see the patch status of your Windows machine side by side with the patch status of your Red Hat machines or your CentOS machines, your Mac machines. Of course, there's a huge prevalence now of Macs in your environment. Being able to respond to... I'm not sure if the callers on the phone have heard of the Mac malware that's been out there in the news in the last couple of weeks but, you know, being able to prove to your executive management that those Mac users are just as fully patched as your Windows users, all within the same reporting infrastructure.
Chris: And certainly, also, any auditors, any compliance regulations that you're beholden to.
Chris: So we have a number of questions that are specific to our...to the Lumension product. Before I get into some of those, let's take some of the higher-level questions. One, I'm curious about patching schedule. How often is the patch cycle and would it not make sense to patch on a weekend rather than a weekday in case system problems occur?
Russ: So there's pluses and minuses to that. Again, it's kinda dependent on how many of your managed machines are desktops or how many of them are laptops. And if you have a large number of desktop machines or servers that are in your environment that you know are with the...that stay within the environment on the weekend hours, then maybe it does make more sense to start to schedule those rollouts on a weekend where you have less user impact. But if you have, say, more of a mobile workforce and you have a lot of laptops that are connecting in, maybe it makes more sense to do that during the week and choosing a time zone for those rollouts that has the least amount of impact to all of your users whether those users are spread out across different time zones. So it's gonna be dependent on the number of users that are connecting in with your environment.
Chris: Great. All right. Questions keep rolling in. Keep them coming, guys. We're here for as long as we need to be or until they cut the phone lines. The next question. How do you deal...let's see. I think we've kinda dealt with this one. How do you deal with operating systems like Mac OS? I think we've done that one.
Russ: Actually, I just wanna touch on that again too. I mean, that's a great question and especially since we've actually seen, you know, more Mac endpoints entering the environment, that really is critical for end users or for administrators to have that view of the fully patched status of their Windows machines as well as their Mac machines within that same reporting infrastructure. I can't really...I can't sell that enough. It's so important to be able to prove to your C-level executives to say, "Yep, my Mac machines, they're not at risk for that flashback malware. They're fully patched. We've got the Java fully patched and we're ready to go."
Chris: Next question. How do we determine systems which don't have a specific patch installed? Like we just had a critically released MS12-020RDP patch. Is there a way to assess if a specific system doesn't have a specific patch?
Russ: Yeah, absolutely. As long as that machine is under management, you can dive into a view based off of the vulnerability itself so you could pivot the view based off of the MS12020. You could pivot the view based off of groups whether you can say how many of those machines in that group are applicable to that known vulnerability or you could dive right into the specific endpoint itself to say, "That specific machine, what is that machine applicable to?"
Chris: Great. Okay. Another question on scheduling. You stated we should come up with a schedule for deploying patches monthly, quarterly etc. but you also say we have to plan to deploy within 72 hours of Patch Tuesday. When do you expect to be done within those 72 hours?
Russ: Again, that's going to be dependent on, again, the machines that you have under management in your environment. We've heard Jim tell his...about how his rollout plan is actually tiered over three weeks. And so, it's really all about maintaining a repeatable process to get those machines patched in a metric that works for your specific environment. Again, as a best practice, we would like to say that you'd like to have all your machines patched within 72 hours and we...I know of many customers that we have that push to have upwards of 80, 90% of their machines patched within the week of Patch Tuesday, by the Thursday of Patch Tuesday and that's a very accelerated path but you also have other customers where as long as you have a repeatable process where you say, "My first tier's gonna be within the first week. My second tier is within the second week. My third tier is within the third week and I expect to be fully patched by that third week." It's really what works best for your own environment. Again, we talked about our own best practices of saying that as long as you have a process to get those machines fully patched, that's what's most important.
Chris: Yeah, and I think that's what auditors are looking for also when they're looking at compliance regulators. You know, just having that process and adhering to that process is the key point.
Russ: Yeah, I mean, if you're running an environment that's all desktops or all servers, all those machines that are within your four walls, you're running...you know, you're running plain Microsoft OS and you're running plain Office applications. There really is no reason why you shouldn't be patched within a week. But if you have more of a distributed environment, you're running a lot of custom applications, a lot of in-house applications that are maybe built off of Java, you wanna understand what the impact of releasing a Java update is to those custom-built applications, you might wanna be a little bit more cautious about doing a full rollout and just having kind of a set it, forget it kind of attitude.
Chris: So the next couple of questions here are kind of a little specific. Regarding the agent, is one agent...is there one agent per OS or can your agent be installed on multiple OSs? Is the agent bandwidth sensitive? What's the footprint?
Russ: Those are all really good questions. I would actually...I would say that we have some limitations based off of the type of the OS that we're installing onto but pretty much, we have an agent for Windows machines, we have an agent for non-Windows machines and we try to keep that agent as simple as possible and yet retain as much of the power that it has in managing your environment. I'd like to say we have a...we do have a small footprint and we wanna make sure that there is...that agent is as performant as possible. But again, that's gonna be dependent on the type of machine that you're managing and what specifically you want to manage on that endpoint itself too whether it's specifically just patch remediation or if you have patch remediation within any virus module installed as well.
Chris: Great, thanks. Let's go back to...you mentioned Java and somebody asked the question how do you propose dealing with frequent Java updates vulnerability given that many applications require back level versions?
Russ: That is a fantastic question and Java is a...it's a very interesting application and an application framework. I know of a lot of different companies that use Java as their internal...as their framework to build internal applications. I know that when you install Java you can't have multiple versions of Java on the endpoint at the same time. I also know that Java's very sensitive, that if you have internally built applications, if you upgrade to that newer version it could break those internal applications. But again, this all comes back to having that process where you can do the assessment of the risk of that vulnerability to your own environment. And once you understand the amount of risk or impact to your environment, that's what you start to use to build a business case for updating those internal applications, for being able to roll out those updates.
Again, I go back to that example that Jim gave to where this sounds very similar to those FDA machines that he has in his environment. Although he's not allowed to update them directly, if he's armed with the information to know to say, "Hey, this machine, it might be running a version of XP to control my scanner..." Or whatever it is in your environment. To go back to that vendor and say, "You need to provide the update so that I'm not...I don't have an exploit window in my environment."
It's very much the same thing. So if I'm armed with that information about the Java update, I can go to my internal application developers and say, "This is a huge security hole. What are you doing to fix it? Because I need to upgrade my Java."
Chris: Yeah, that's really important. Information or knowledge is power, right? Okay. A couple of questions about reporting. As we know, the job's not done till you've finished the paperwork. Can you talk a little bit about publishing reports, making them available to management, using SharePoint, that sort of thing?
Russ: Yeah, absolutely. We have a whole other module around what we call Lumension reporting services and I agree entirely that the job isn't done until everyone knows that it's done. You know, you could be fully patched and if the execs don't know or if your manager doesn't know that those machines are fully patched or that the job is done, then there could be an assumption that there's still work that's left to do be done. So reporting is an incredibly important part of this process to circle back and to have that suite of reports to go back to your superiors or out to the organization or to your compliance officer, to your auditor and say, "Yep, we're fully patched." Or, "Here are the machines that are..."
Again, there's many different views to this but the number one view is gonna be the patch status. So whether it's...if you're doing a vulnerability assessment and you're fully patched for those targeted machines, then that's the most important measure. After that, you're starting to look at agent check in times, you have reports around how many machines have checked in within the last time period, how many of those machines have agents that are offline or how many of those machines are offline? Making sure you have a better understanding of using those reports to understand your environment.
Chris: Great. Okay, let's see here. Audience members written in that they have problems with the laptops being slow and logging in in the morning, all sorts of things going on all at the same time. They currently deploy the patches at 11:00 p.m. However, most of the laptops are not on the network and so they receive the patches in the morning. So he's trying to convince management to deploy it 12:30 p.m, so presumably that's early afternoon. Then things wouldn't be so slow in the morning. He's wondering...it would help users that hibernate.
Russ: Is that 12:30 p.m East Coast time or is that 12:30 p.m Pacific? So I mean, we...so that's...it's a great question and really it's a challenge that all customers have, that all end users have and that all IT administrators have and really, again, it comes down to making sure that those users understand the importance of having a fully patched machine and understanding...you know, getting that understanding out to the entire organization to say that on your repeatable cycle, on a monthly basis you're going to have these patch windows and to proactively notify those users that, yep, you've got a deployment scheduled. We have an end user notification that we can pop onto the endpoint to say, "You know, we've got a patch scheduled that's gonna include these updates. Here's why it's important."
And to just proactively notify those users to say, "You're about to go through a patch update." Whether it requires a reboot or not and just to set those expectations to the end users and to say, you know, "This is important because your machine holds critical company information. The information is critical and vital to stay within the organization and we need to make sure those machines are fully patched to protect that data."
Chris: So another question came in a little earlier. Does the patching process include approvals for the owner of a service that is updating the servers that are running?
Russ: Yeah. This is where we talk about the workflows too, right, where you have a patch approval process, where you can set up your deployments, set up your test environments, make sure you have those approvals in place. This is critically important for...we see this, again, with server administrators to where you have multiple owners that are in charge of those machines and making sure you can go through those owners and say, "Here's what needs to be deployed. You know, let's schedule my downtime, let's schedule my maintenance window, whether it needs to go a clustered environment and starting to go through each machine at the time." So yeah, definitely. Those are all vital to this process.
Chris: Great. Okay, another question. How to minimize the patch deployment time taking over...compare the patch deployment time, I guess. How do you minimize the deployment time and what's the comparison to other...between tools?
Russ: That's really gonna be dependent about...on the type of deployment that you're scheduling and how many bulletins are included in that, whether you've got a service pack that's in there. I mean, there are a lot of factors that are involved in that. You could have a Patch Tuesday that has just a single bulletin or you can have a Patch Tuesday that has 16 bulletins, that it has, you know, two Adobe bulletins on top of that and Java update on top of that. Oh, and by the way, maybe you've got your Windows 7 service pack that was close to 900 gigabytes, you know, so you...or 900 megabytes, excuse me. I mean, you're talking about some huge deployments across your environment.
So all those things need to take into consideration in terms of how to measure the success of your deployments and again, it's about are you deploying to a lot of remote locations, is it all in-house, is there something you can do from a proxy standpoint to put a proxy into your remote locations to locally cache those updates before they get deployed out? There are a lot of things that you can do to minimize the impact from a bandwidth consideration. We have a featurette in our product that we call Fast Path and it's something that's used to...from an endpoint perspective, especially from mobile endpoints depending on where they're out in the country, it'll choose the next closest location internally within your organization say if you're connecting via VPN. So it knows which place to get its cached content from. There's a lot of things that we have built into the product to that help with these considerations.
Chris: Great, great. Well, it looks like we're gonna get kicked off in about four or five minutes so I just wanna get through a couple more questions, Russ. We have a question. Might have been aimed at Jim but hopefully, you can look at this. If you're using a logging script, how do you account for admin access to install the agent? How do you account for unsuccessful installation of the agent?
Russ: Oh, that's a great question and one of the things that we can use internally...we actually have a tool that we call the Lumension Content Wizard where you can build a custom patch to actually...that accounts for some of these logging scripts as well. Using logging scripts is a powerful tool but it can also have a performance consideration for those endpoints as they're logging on in the morning. So there's a number of different ways we can look at this but if we have an...if you're actually using this Lumension Content Wizard to deploy out these updates or to set up some of those scripts on your machines, you can set up reports within the application that state that for that specific patch type, if you use it as a policy type, is that policy enabled or disabled on that endpoint. So there are a lot of different monitoring techniques we have within the application itself.
Chris: Now, moving into a slightly different direction. How do you go about pruning the database of old patches and service packs or how do you deal with that baseline?
Russ: I mean, that's kind of a couple of different questions but it's very important to make sure that your baseline is up to date because really that's your last best defense about those machines that check in outside of your normal patch windows to get up to date and so they're not susceptible to those known active exploits. I really like the example that Jim gave about building the baselines into his overall process and when we talk about kind of that week before Patch Tuesday versus the week of Patch Tuesday, you're talking about, you know, three weeks into your process is when you're gonna be adding those bulletins to your baseline and that leads perfectly right into the next month. So as you're doing your maintenance on a monthly basis, you start adding in the bulletins from the previous month at the time that you're researching the bulletins for this current month. So you're picking up any of those machines from...that may have come off...may come online anytime during the month to pick up those baseline items and so they're ready to pick up the next month's patches.
Chris: Okay. We only have about a minute. I want to thank everybody for spending your time with us and I hope this was useful. We're gonna do one more question and if we didn't get to your questions, we'll get back with you after the webcast. As a reminder, remember that tomorrow we'll be sending a link to the slides and a recording of this webinar along with a copy of the patch management best practice guide white paper. So be looking for that tomorrow. Russ, let's see. For last question. Yeah, what is a particular server to client ratio? Does that...okay, for example, say that we have 500 workstations in a particular region and 500 machines in another region. Do I need one server or two?
Russ: You'd only need one server in that situation. We can have thousands and thousands of endpoints connected to a single server regardless of where those endpoints are located. We have a lot of customers that...they can choose to install multiple servers to manage regional offices but realistically, you can have a single server manage remote offices very easily. We have other things that we recommend. For example, I wanna go back to having the...the example of having a proxy in place for each of those local offices. Definitely for bandwidth considerations and for those endpoints connecting to the proxy to pick up their cached content as opposed to going directly to the server if it's remote. But absolutely, we can connect thousands and thousands of endpoints to a specific managed application server.
Chris: Great. Okay. Well, thanks, Russ. We've run out of time. Thanks again and thanks to Jim who's in a meeting now. We'll be following up with folks whose questions we couldn't get to. Thanks very much.
Russ: Thank you, Chris.