October Patch Tuesday
October 11, 2017
Chris Goettl | Director, Product Management, Security | Ivanti
Todd Schell | Product Manager for Patch | Ivanti
Join us this month as we recap the Microsoft and 3rd Party security patches released on Patch Tuesday. We will discuss things to watch out for, products to be sure to test adequately, and which patches should be highest priority to roll out.
Chris: Good morning everyone. This is Chris Goettl, and with me today is Todd Schell. Todd, how are you doing today?
Todd: I'm good. Chris how about yourself?
Chris: Doing really well. Welcome to Wednesday, October 11, 2017, Patch Tuesday webinar. It's been an interesting morning so far. I'm on the road again this week. I'm in Dallas and unfortunately, there's a little construction going on at the hotel I'm in. To give you an idea of what rehearsal time was like a few minutes ago, if you remember that scene from the movie Scrooged where Bill Murray is trying to have a conversation with all the hammering going on in the background, and he's continually shouting, ''Would you please hold the hammering?'' That was pretty much going on here a few minutes ago. I've been told they've asked the team in the room directly above me to stop using the hammer drill to drill in the floor right above my head. If that picks up too much again, I'll have to go on mute and Todd’ll take it until I can jump in again. But we’ll make it through.
All right. We'll go ahead and go through an overview of what we've seen so far this Patch Tuesday and talk about some of the things happening in the news right now. There's a lot going on, as usual. Then we'll get into a bulletin-by-bulletin blow of exactly what came out this Patch Tuesday and field any questions you might have. As we're going through, go ahead and use the Q&A feature to send any questions our way. We'll take a look at those throughout the webinar and toward the end, as well, to try to answer as many as possible.
Jumping right in, this month Microsoft has 10 individual updates that have released. Nine of those are Critical, one is rated Important. All 10 include vulnerabilities that could be used in a user-targeted scenario, meaning typical scenarios that could end up as a phishing scam, a drive-by download, or attacks like that where a user could be duped into allowing an attacker to exploit a vulnerability through crafted files, a specially crafted webpage, a URL in a link or chats, or other features like that. Adobe has an update this month for Flash Player, but you'll notice it is not rated Critical or Important. In fact, it doesn't include any security vulnerabilities at all. This is a first in quite awhile. We have an Adobe Flash update that is completely bug fix, no security vulnerabilities to this one. We'll talk about that and some other updates that are coming out. We do have one zero day that's being resolved this month and that is in a Microsoft Update, which we'll talk about in a moment.
In the News
Let's talk about some of the news going on right now. We have a continuation of the Equifax drama. Let me get over to a couple of articles. The biggest thing is the Equifax breach is extending, as far as what they know about who was affected. There were more non-US people affected by this, it was reported recently. There's this article that came out from Bloomberg, which this is what we want to try to touch on today because Equifax at this point has been beaten to death. You can't go through the headlines any day without tripping over another article. The biggest thing here is they're talking about the impact of this breach, the speed of the disclosure. Most importantly, though, is this section where Representative Joe Barton was wondering if companies like Equifax might do a better job of protecting their customer data if there were federal fines for breaches. There's a distinct possibility with the number, size, and scope of these breaches that there will be more regulation around and fines enforced in cases like this where due diligence may not have been done. That's a little more development in the aftermath of the Equifax breach. I think it's not directly related to the Equifax breach, it's more in general, but the Equifax breach being so large and widespread has drawn the attention of lawmakers and more of a regulatory enforcement. That's a situation that may develop further.
There are other things regarding continued use of the Eternal Blue exploits in a number of attacks that have happened. There was a banking Trojan using that SNB exploit, and some hotels were attacked in Europe and the Middle East using those exploits. I'm sending you links to the articles for those. The moral of the story on that one is there's been an update available to plug those vulnerabilities since March, so identify where you're vulnerable to that, and get those vulnerabilities plugged as quickly as possible. They're still being utilized out there. It's too much of a risk to leave it. If you can't plug that vulnerability for some reason, you need to make sure other security controls are put in place to mitigate risk. Reduced privileges, application control, device control, whatever you need to put in place to ensure those environments are protected. Keep that in mind for environments that may have made it through the first WannaCry attack or haven't seen exploits of those SNB vulnerabilities yet. There were a couple of car manufacturers that had plants get shut down because after the SNB or WannaCry attack was behind us, a version of WannaCry malware was introduced to those environments through a device like a USB stick. It is possible to get into environments even if you made it through the first round of attacks. Those vulnerabilities can still be exploited and you can have a very disruptive incident if that occurs, so we’re reinforcing that you make sure you have those critical, known-to-be exploited vulnerabilities plugged.
Last one here. This is actually not a patch-related incident, but I've seen a number of articles around this. Brian Krebs had an article about this not too long ago. He says, ''When you're looking at things like web services, Cloud services in general, it's very easy for a misconfiguration to leave some of those services exposed to the public.'' This last article is relating to a case like that. Over 100,000 medical patients’ data was exposed because they had stored data in an Amazon web service that was left accessible to public-facing access. A simple misconfiguration led to a breach, in this case. It's going back to and reinforcing the fact that the majority of instances we run into are preventable. It's understanding where and how they can get in, and making sure we're plugging the methods used to get in.
Vulerability Public Disclosures—Linux and SharePoint
Let's talk a little about the vulnerabilities that were resolved this month. There were a couple of public disclosures we'll go through first, and then we'll get into the exploit in the wild, the zero day we have this month. The first one, CVE- 2017-8703, is a public disclosure that came out regarding a vulnerability in the Windows subsystems for Linux. It's a denial-of-service vulnerability. The attacker, if they exploit this, could cause the local system on the affected system to become unresponsive, making that system useless until you clean up that attack. This is the second public disclosure on that Linux service in the past few months. I think it was two months ago there was another one, so something to watch out for and make sure you get plugged. This is affecting Windows 10 build 1703, nothing older than that. It was a service introduced at that time to create cross-platform support for Linux systems as well as the Windows 10 platform. It's only the one platform that's affected.
The reason we call out a public disclosure, and this is not being exploited in the wild, there are no known exploits currently. A public disclosure means that either through a researcher documenting and publishing their findings or some other means of information being leaked to the public, this disclosure has allowed enough information out so an attacker has a jump start on developing an exploit. This is one of those indicators of risk we keep track of to help identify where you should focus some attention.
The other public disclosure this month is in Microsoft Office SharePoint, and this allows for a cross-site scripting. That vulnerability could allow an elevation of privilege in SharePoint. If an attacker successfully exploits this, they could perform cross-site scripting attacks on the affected systems. They can run scripts in the security context of the user, but more importantly, the attacker could read content they should not be allowed to access, use the victim's identity to take actions, and be able to change permissions or leak content. They can even inject malicious content into the browser of the user. This is again a public disclosure. Enough information is out there for an attacker to put together an attack around that vulnerability, so it’s one to watch out for. You definitely don't want to leave that one hanging for too long as it may get exposed in the near future.
In the Wild Exploit—Office
The next one here is CVE- 2017-11826. This is the vulnerability this month that has been exploited in the wild. It's a vulnerability in Office. It involves memory corruption. There have been documented cases where this has been used in the wild to exploit systems already. An attacker could create a specially crafted document and have the user open the created content by sending it via email or hosting it in a web-based scenario where the user could be convinced to go to a website and click on content there to exploit it. That specially crafted file could allow the attacker to gain rights equal to the user who was attacked.
This is a good example of use of least privilege mitigating the impact of a vulnerability like this. If the attacker exploited a user who is a full administrator, the attacker gets full access to that system. The attacker would have the ability to create user accounts; add, change, move applications; delete data; even jump from that system to others where the user has access.
If it was a regular user, the vulnerability would only allow them equal rights to that user, and the attacker’s ability to do anything would be diffused. The attacker would have to find additional ways, like an elevation of privilege attack or other means, to escalate the privilege level before the attacker could effectively do anything that had value to him or her. Least privilege would mitigate the impact if it were exposed. In general, having better privilege management control in your environment makes it so, in cases like this, even when there's a zero day, the potential impact is reduced.
Now, getting into known issues, there are, unfortunately, a lot of known issues this month. We'll go through them, and as Todd gives us a bulletin-by-bulletin breakdown, he'll go through some OS-specific issues, but there are a couple we want to point out in advance. The first one is Windows 10 version 1511. This is it. The final update was this Patch Tuesday. There are no more security updates for 1511. You’ll want to make sure you identify any 1507 or 1511 branches still in your environment and upgrade them to a later branch as quickly as possible. October 10 is the end of extended support for Office 2007-Outlook 2007. Any of you affected by these end-of-lifes, take a look at your environment, make sure you get a good inventory of everything, and identify where you have these products. These products are now a security liability.
Todd: But it's only 10 years Chris.
Chris: I know Todd. It's one of those things where there are still XP systems out there. We still have customers running Server 2003. I know there are sometimes a lot of reasons you have to keep legacy systems around. The one thing I will say, Todd, to that point of it's been 10 years, it has been 10 years, but if you still have things running on those platforms that you haven't been able to take off, make sure that, knowing they're exposed from a security standpoint, you know what steps you're going to take. This is one thing we go through quite a bit when we share best practices around patching. An exception to a patch that needs to be put in place is not the end of your process. If there's an exception, you need to have documented mitigation to ensure you've done the right things to reduce exposure.
If you have a system that continues to use Office 2007, it might be time to remove the application from the end user's machine, put it into a VDI environment, and give the user access to it through some other means. A means that reduces direct Internet exposure, reduces the user’s control in that environment so the user has a reduced level of privileges, and provides stronger application control policies. There are many ways you can balance the security on a system like that if you have to keep it. If there's really no reason to have it still in your environment, get it cleaned up. It's too much of a risk. Too many vulnerabilities are exploited for years after updates are made available or after end of life of a product is established because the system continued to run.
There are a couple of other things to be aware of. There is a vulnerability in TPM that could allow for a security feature bypass. This is a long, drawn out, complicated KB article we're about to look at, so let me give you the high level on this one. The Trusted Platform Module chipsets is a set of chipsets that are affected by a vulnerability. If you have those families of chipsets, there is a firmware update coming from the OEM, and there is a software update from Microsoft. The issue here is if you're affected by this, the key strength of the trusted platform module, the encryption strength is weakened. It's important to note this is a firmware vulnerability, not a vulnerability in the operating system or a specific application. After you install the software update, you'll need to re-enroll any security services you are running to remediate those services.
There are a lot of steps. It has a wealth of information, but I'm going to point out a few key things quickly. First, there is a list of OEM vendor sites, the TPM/OEM page here, but also, if you're running Fujitsu, HP, or Lenovo hardware, there are links to their pages that document what products are affected. A couple of things to keep in mind: They've given a few options for identifying which systems are affected. None of them are spectacular. You have the use of event logs. If you have some kind of log-monitoring system, you may be able to gather that data more automatically and identify which systems are affected. Option two is to use a script that would allow you to check which systems are vulnerable. The third option would be to manually check the TPM management snap-in on those systems and see what's affected.
One thing they are recommending, however, is to apply the Windows operating system updates first, and then apply the firmware update. If you don't do it in that order, apparently you'll lose visibility on determining if the machine is still affected or not, so order is a concern. That's something you’ll want to take a look at. Make sure that if you have any affected hardware, you follow up and get the firmware update. Make sure the Windows Updates are fully up-to-date first, and then apply the firmware update.
Todd: Chris, you might add that this is specific to a chipset by a company called Infineon, and not all manufacturers use that chipset. There are a lot of TPM chip manufacturers out there, so your hardware may not need to be patched. That's one of the reasons you have to use the detection method they show there.
Chris: Absolutely. Those vendor pages in the article will be critical to helping you know if you have affected systems. All right, last piece of news this month is Oracle's Critical Patch Update. Next week on Tuesday, October 17, Oracle will release a host of critical updates that will definitely include updates for Java, JRE, and JDK, so make sure that if you are running those in your environment and you need to update those, which you most likely will as they rarely have a handful of critical updates, make sure you have that built into your deployment plan this month. Also, look for an announcement on the future of Solaris and SPARK hardware. Todd, if I'm not mistaken, we're coming up on a potential end-of-life soon, right?
Todd: Yes. Oracle laid off about 2,500 employees on September 5, and they were primarily people doing Solaris support and SPARK hardware support. The Oracle website says they're going to support this through something like 2024 or some really long, far-out date, but they may be pulling that in because probably nobody'll be using it by that point if they're going to stop supporting it.
Chris: Right, so potentially impacting anybody using those platforms. All right Todd, let's talk bulletins.
Monthly Bulletins—Windows 10
Todd: Okay. Let's go through this month's bulletins. As usual, the bulletin for Windows 10 continues to be the largest in terms of the set of vulnerabilities that were resolved this month. This bulletin covers everything from the last release for 1511 through server 2016 and Microsoft Edge. There were four KB articles that covered the releases. The one vulnerability you talked about earlier, Chris, was publicly disclosed, the 8703. I didn't list all of the vulnerabilities here only because they don't fit very well on the slide, but you can go into the security update guide and get a complete list. As I said, this is the final security update release for version 1511, so do be aware of that.
On the next slide, we'll see the Known Issues this month. There were quite a few of them. I've included the KB articles and the versions of the operating system under the Windows 10 bulletin that are addressed. You can see the first one is going to impact your Express installation files in the future. If you take a look at the bulletin, there is a temporary work around. It has to go in and change some registry settings. For all of these it always says Microsoft is working on a fix and it'll show up in a future release. In addition to the Express installation files may fail, there may be a duplicate of KB numbers if you installed the Delta update package, so be aware of that. It's something you'll see in the information that isn't anything that is affecting the update directly.
Finally, this last item will show up across multiple operating systems this month. When you're closing down some applications, you may get an error dialogue that an application exception has occurred. It's caused by this .dll that's listed here, this MSHTML. This is a problem across multiple operating systems this month, and you'll see it addressed several times. For details on this particular bulletin, you can click on the link that's shown here. You can see it not only affects that particular bulletin but is also covered under the 42895. The next one is the same Known Issue when applications are shutting down. It's also addressed separately under that bulletin.
The last one is something we've seen for a couple of months now. This first one has to do with language changes and you can see this references a much-earlier bulletin that was installed. I think it's carried over for two months now, so they still haven't resolved this one under Microsoft Edge. There is a new issue, however, that has to do with the Type C connectors. Unfortunately, one of the things they say to do to get around this problem temporarily is turn off the UCSI in the BIOS. That's a temporary workaround. Obviously, if you're having a blue screen or a black screen, you need to do that. They are working on this one aggressively, as well, and all the details are covered in this particular bulletin. There are these three KBs that are affected with Known Issues under Windows 10 this month.
Chris: This is probably a good time to pause and talk about another issue we've seen floating around out there. I know we have a lot of non-Ivanti customers on here, as well. Some of you may be using WSUS. There was a Reddit thread that came up yesterday saying a number of people were running into blue screens. The source of that is a Known Issue. Apparently, the WSS auto-approved rules allowed the cumulative rollup and the Delta to be approved at the same time, which makes it so the two of those were allowed to get down to the same system and run in the same window, resulting in that blue screen.
This is a good time―we have another of our product experts online with us today. Brian Seacrest, not to be mixed up with Ryan Seacrest, Brian is with our content team. He does a lot of our testing in the team and has jumped on with us today to talk a little about the steps we've taken in our Windows content team to prevent this type of issue from happening in our environment. Brian.
Brian: Hey. On our content right now, we ran into that issue when the Deltas were first released, and the way we approach it is if you are running the previous security for the September one, we offer only the Delta and we don't offer the full file. Regardless, if you stand with our product and deployed to everything, simply went CTRL-A deploy, you won't be vulnerable to your systems not booting, which requires a lot of manual intervention and causes a lot of headaches.
Chris: The beauty of that is you approve this month's cumulative rollout for Windows 10, and we take care of whether the Delta or the full should be applied, depending on the state of the machine. All that guesswork is off of you, and we handle it with several checks and balances all the way down to the end-user system executing the patch to make sure only one or the other gets applied, right?
Brian: That's correct.
Chris: For those of you who were using the auto-approved rules, you might want to double-check that and make sure both the full file and the Delta didn't get approved. You can see here the Delta, if approved, the two of those together could cause this blue-screen scenario. In this Reddit, it looks like several people found the Delta was pulled, and they may have republished again without the chance of the two of them going together, but there was a window last night, so if you're one of those who were kickstarting things right away, the first round of test systems or the groups you might have had going first could potentially be affected. That's a heads-up here that that particular issue may be affecting Windows 10 or those of you running WSUS and doing the auto approve. All right. Todd, back to you.
Todd: Okay. Moving on to Office, as Chris mentioned earlier, there were some publicly disclosed as well as zero-day exploits this month, and I've highlighted those in red down below. The publicly disclosed one was 777, and 826 was the one that was both disclosed and exploited this month. After last month's huge release of updates for Office, this month was a little quieter, although there were still 32 KB articles released, a lot of which were focused on specific service packs. You’ll want to make sure you have the latest service pack for each Office application installed because these patches will only apply to those.
Once again this month, Microsoft has released their defense-in-depth package. They don't say a lot about what they fixed under the covers in there other than it addresses a large number of vulnerabilities and is kind of a widespread approach to as they say defense-in-depth across these applications, so be aware of that. We've seen some of the older applications supported this month. For example, Word 2007 got an update and Outlook 2010 also got an update this month.
Monthly Bulletins—Internet Explorer
Next is Internet Explorer. There are only a few vulnerabilities addressed this month. There were five across Explorer 9, 10, and 11. Be aware that the cumulative updates for Explorer are included in the monthly quality rollups, so when you apply those, you're getting all these Internet Explorer updates for free, or included in that package. There were eight KB articles this month, which is standard for the way they release updates for Internet Explorer. They are rated Critical, so you’ll want to continue to update your versions of Internet Explorer. Server 2008 also had a number of releases. There were 20 vulnerabilities resolved, none of which were publicly disclosed or exploited. Like the previous package, there was a defense-in-depth released this month that is Advisory 170016. You can read through that to be aware of what they've done there. As I said, this is a regular update for Server 2008 as they continue to support this.
Monthly Bulletins—Windows 7, Server 2008 R2
Next is the monthly rollup for Windows 7 and Server 2008 R2. This includes updates for Internet Explorer. It includes the improvements and fixes that were part of the preview released on September 9. It’s a single bulletin, 40401681, that fixes the 20 CVEs I've listed here and the previous five Internet Explorer vulnerabilities. You'll see on the next slide that from a Known Issues perspective, we have this problem related to applications closing down and throwing up an error dialogue message, the same one I talked about earlier. The link here specifically refers to the KB that addresses Windows 7 Service Pack 1 and the Windows Server 2008 R2 Service Pack 1, so if you’re seeing error dialogue boxes showing up on your machines, you'll know what's causing it.
Monthly Bulletins—Server 2012, Windows 8.1
The monthly rollup for Server 2012 this month addresses 21 vulnerabilities and includes five Internet Explorer vulnerabilities. It’s pretty straightforward. It is Critical because of the remote code execution possibilities covered by this particular set of patches.
Finally, we have the monthly rollup for Windows 8.1 and Server 2012 R2. Each of these fixes a few more vulnerabilities. This one covers 23 vulnerabilities, which are listed here. It includes not only the base operating system, but also Internet Explorer for this particular one. On the Known Issues for this, you'll see the error dialogue related to applications closing once again. As I said earlier, this stretches across quite a few.
Getting into the security-only updates for the month, and to reiterate what the difference is, the cumulative rollups are a growing set of patches that started in October of last year. We've now come up on the one-year anniversary of monthly rollups, so we have one complete year of updated patches in one bundle. You can apply that in one fell swoop and update your entire system. The downside, of course, is that it's growing and getting bigger. It's a very big patch. With the security-only updates, these are the security releases that came out during the month and they're rolled together for a rollup at the end of the month. The security-only update is only for what came out in September. You would have to go back and individually apply the ones for previous months. They are not cumulative as expected. They are cumulative for the month, but not from the beginning of the rollup period, which was October of last year.
This month, under Windows 7 and Server 2008 R2, you can see a number of components were addressed from a security standpoint. This security-only rollup is 4041678. There were 20 vulnerabilities addressed. As we would expect, they were the same 20 vulnerabilities addressed and listed previously for the cumulative rollup. Interestingly, unlike the application error dialogue I was talking about, this is not a result of the patches that were done as part of the security-only updates, so there are no reported Known Issues for this bundle of security patches.
Next we talk about Server 2012. Once again, 21 vulnerabilities addressed as part of this security-only rollup. There’s a lot of commonality across these rollup patches for the months, as usual. Notice that this one addresses the Windows SMB server, so be aware of that. It’s single bulletin 4041679 and again, no reported issues for this security-only monthly update.
Last is the security-only update for Windows 8.1 and Server 2012 R2 addressing 23 vulnerabilities. We have the TPM issue listed here that Chris talked about earlier, the ADV1700012. It's covered with this update. The only one, I don't know if Chris mentioned it earlier, but the only operating system that isn't covered by the TPM vulnerability is Windows 7. The reason for that is because of the way it had implemented BitLocker as part of its key generation routine. All other operating systems except Windows 7 are affected by that issue Chris talked about. Again, there are no reported issues for this security-only update.
Getting into our non-security updates, we did release two other updates for other applications this month, Opera and Firefox ESR. These did not have security fixes, as Adobe Flash Player did not. However, it did have some critical bug fixes and updates for some of the features available in these products.
Between Patch Tuesdays
Chris: All right. That wraps up our Patch Tuesday bulletin run through. There are several questions coming in, and we'll go to those shortly. One thing we do cover each month is what we call ''between the Patch Tuesdays.” There are a number of products we’ve added new support for, a number of security updates that came out, and a number of non-security updates that came out, so you can see there's a lot of activity between the Patch Tuesdays. One thing we often recommend, especially for systems that have laptops that go in and out of the environment, is to get to a more than once-a-month patch cycle. If possible, getting to weekly is ideal, twice a month good. Once a month―there's a lot of activity that happens where those third-party products could have vulnerabilities that needed to be resolved.
Here are some of the vulnerabilities that were plugged in between. Brian and the team on the content side were kind enough to give us a breakdown of the vulnerabilities. You can see this Nitro Pro update included two CVE fixes. The Apple iCloud updates included 22 security vulnerabilities that were resolved. Chrome, the release on 922 included two security vulnerabilities. The Apple iTunes update took care of 19 vulnerabilities, and some new CVEs were added to some past BMware bulletins. These were additions to previous documentation, other vulnerabilities that were happening in between.
Now let's get into some of the questions we have coming through. One of them was more of a request from Omar: “Loving the Patch Tuesday blog, but try to bring the Known Issues into the blog, as well.” That's something we can absolutely try to do. One of the things we do is get a lot of the initial commentary done and out the door in that blog post, but we could probably make an addition to that later in the afternoon as we pull together the Known Issue information. We do a lot of PR around Patch Tuesday and that initial blog post is part of that, but we could always try to do an addendum to that. We'll see, Omar, if we can get those Known Issues appended to the blog.
Edward had a question about the Delta updates. Brian, would you mind giving us the lowdown on the full versus the Delta patch?
Brian: Absolutely. In the Chat, I'll also put Microsoft dialogue about it, but in a nutshell, Delta updates are massive. I think the most recent ones were about 1.3 gigs, at least for the Windows 10 LTSD where they should be largest. To fix that or remediate that a little, Microsoft has created a Delta update, and it only applies to “from the previous security to the previous security.” For example, if you were running 1703 and you installed the September update and then installed the non-security release in the middle, Delta wouldn't apply. It allows you to go from security release Patch Tuesday to Patch Tuesday and not deploy so much. It also reduces deployment time a little.
Chris: Got it. Thank you, Brian. Michael had a question: “Can any of the Ivanti products detect the TPM vulnerability?” Michael, that's something we have to research a little more. We do have products that can get a lot more detail about the firmware levels of products. Any of you who use our Endpoint Management solution, that product would be ideally suited for us to identify where you may have that. We have to get a little more research done on that, and I can put out a request to the team to see if we can identify how to do that. For those of you running Microsoft System Center, you should be able to get the firmware details and identify the systems that may be affected.
Going back to the three vendor sites identified in that article, the Fujitsu, HP, and Lenovo pages are going to have details about it. Let me pull one of them up here. In the summary, here's a list of the hardware that could be affected. In a lot of cases, your asset solutions in your environment may be an ideal way to identify some of this information. The systems management side from our endpoint manager and the CCM should be able to get down to firmware versions and things like that, so that would be the best source. This information having recently come up, we haven't had people documenting all of this yet, so that's something we will definitely send over to our support and engineering teams to see if we can identify opportunities to better identify the systems affected.
There was a question about the vulnerability that affected Equifax. This is from Peter. The Equifax breach was traced back to a plug-in for Apache called Apache Struts. This was the vulnerability that was not updated that allowed the attackers for the Equifax breach to get in.
Todd: There have been a lot of comments also Chris about the timeline on that. The actual vulnerability was identified long ago, and it was fixed back in March.
Chris: The breach happened a couple of months after the update was made available, but the update had not been put in place. All right, let me see if I can open up the Q&A window and see if we can answer any other questions that may be coming in. “Is there a way to get a few people the email link for next month’s webinar?” Yes. We should get that up on the page shortly, but for anybody who needs the link sooner, Erica has kindly put her email address in there and wants you to email her all the time to let her know about anything you need. I'm just kidding Erica. Yes, we'll try to get next month's Patch Tuesday webinar up as soon as possible. There are often bouncebacks or things like that if you've noticed you don't have the email. We do kind of an email drip where anybody who's signed up previously should get notified when the next one is available, but that could be getting blocked if your spam filters, clutter, and other things are getting hold of it. There are a lot of complications as we make things more strict in all of our environments.
Todd: All right. Paul had a question: “Are there any extra steps on these Microsoft Tuesday patches like changing the registry key, like happened in the previous Microsoft Patch Tuesday?”
Chris: Todd, other than the TPM vulnerability, I don't think we saw anything that required additional registry keys, did we?
Todd: No, I did not see anything specific to any registry key changes that had to be manually changed. No Known Issues.
Chris: Paul, there could be a KB somewhere that had something deeper down that wasn't in the high-level Known Issues list. Typically, Microsoft makes those things known up, but there can be cases where it's not. We haven't seen any yet, however.
Sean had a question: “The last two months we had issues with Outlook 2016 crashing after updates were applied. Removing 401191 from those machines fixed the issue. Has this been fixed in this month’s Office 2016 update?” I don't know of that specific issue being resolved in what they were talking about this month. Let’s go back to the monthly rollup, there we go. Yeah, I don't know if they resolved a specific crash issue or not Sean.
Todd: I think Brian was taking a look to see what he could find, too.
Chris: Okay. Brian, if you dig anything up, let us know.
Brian: I'm looking at it right now. I haven't noticed anything specifically yet. I'll let you know if I do find anything. I know in late September, Microsoft wrote an article I can post about custom form scripts, which is where a lot of these issues were happening. They're now disabled completely by default, which would require a reg key to turn it back on. Other than that, I'll look further into it, and I'll let you know.
Chris: Okay. James had a question about the recording being available. Yes, after the session is over, we'll convert the recording. It takes a little while, and then Erica will get everything posted to the website for us, so sometime later today that will be available. I believe we still do the follow-up email, as well. Later tonight or early tomorrow, that will notify everybody the recording is available. Oh, Erica said within the next three hours. She responded to that, as well.
“How can I confirm Endpoint Manager is detecting endpoints need Office updates?” Depending on how you're configured Eddie, that could vary. Because of all the options that can be turned on or off, I would say reach out to the support team and tell them specifically you want to ensure you have things configured correctly to detect Office. I think that would be the best way to do it. We have several patch products and each allows you to configure things a slightly different way, so I would say the best way is to have the support team do a health check with you and make sure you're configured properly to detect Office updates. Have them walk you through a few things. Steve is saying he’s been experiencing similar 2016 patch issues since June.
Oliver had a question: “In terms of in-place upgrades for 1607-1703, for example, what about vulnerabilities for features on demand and all other languages you support?” Let's see here. I'm wondering if I understood your question correctly Oliver. You're saying you've done the branch upgrade from 1607 to 1703, what about all the other vulnerabilities on that system? Once that branch upgrade is complete, you want to make sure you have a patch cycle immediately after. That will detect any vulnerabilities that might have been regressed as part of the upgrade process. A lot of the other applications and things should be retained at the version they were at. If you plug vulnerabilities for a non-OS issue, those should be carried over, but it would be ideal to do that follow-up scan after the branch upgrade. After doing that, you'll need to at least apply the latest cumulative update, security update to that system. and that assessment should pick up any additional vulnerabilities detected.
Todd: Another point, Chris, is we don't cover Microsoft every month, but if you do an upgrade and apply language packs, they recommend you apply the update after you've applied the language updates because they won't be detected if you do the language pack update afterwards.
Chris: Right, so you want to do the branch upgrade, language pack, patch assessment, and deploy the latest cumulative.
Chris: Okay. James is saying his Office 2016 is still crashing after the updates on Sunday. There's still that open question that there's still crashing going on.
Chad had a question about Patch for Windows, about the Ivanti Patch for Windows product specifically: “Can it update Windows version 1511 to 1703?” Yes, Chad, we do have support for branch upgrades. There is a knowledgebase article, which talks about how to download the ISO and set it up so it can be detected and then deployed as the equivalent of the legacy-style service pack deployment. Yes, that's very possible to do. There's a knowledgebase article for that, let's see if I can find it quickly. Find it, yes. Quickly, maybe. It might be easier to contact the support team to grab that KB article. It's there, it’s just a matter of being able to find it quickly.
Brian: I'm adding it to Chat right now.
Chris: Thank you, Brian. Having Brian around is so nice. Here is the document. This shows deployment support, and it works for, in the old brand name, Protect 9.2 and later, but under Patch for Windows, the same process still works. This shows how to obtain the right ISO, drop that in place, configure everything you need to, and then from the product, you'd be able to deploy that service pack level to the level you need to. That's all documented there. Chad, you should have that link in the Chat window now.
Todd: We might mention, Chris, that the fall Creators update is available to insiders as part of the preview stuff, if anybody's interested .
Brian: I can add to that. According to Microsoft, it should be out next week, the fall Creators update.
Todd: There we go.
Chris: All right, I was wondering when that would be arriving, so that's good to note. You'll want to try and get your hands on that some time in the next couple of months. Start testing that and that way you can get ahead of 1603 end-of-life before that comes around. Start upgrading those ones to the latest by getting your testing done on the Creators build sooner rather than later. With 1511 end-of-life being now, you want to get all of those systems up to the latest possible branch. That way that treadmill is a little bit longer before you have to take that next step. All of your Windows 10 systems are on about a 12-month upgrade cycle at this point. It's 18 months from release until end-of life for each of those. Is that correct? Todd, am I remembering that right?
Todd: That's correct.
Chris: Eighteen months for the total lifecycle of a branch. If you have six months of researching, rolling out to small groups, and testing, by the time you hit the six-month mark, you should be rolling out the branch in large chunks to the rest of your Windows 10 systems before the end-of-life of the current branch comes around. That way you keep each of those Windows 10 systems on the branch about a year before you have to upgrade again. Based on Microsoft guidance and what we've seen as a best practice, that gives adequate test time and a large enough window so you're only doing major branch upgrades once a year.
All right. We had a couple of other questions. Mahesh was saying it’s been indicated there are a number of Microsoft KBs having issues. “What is the workaround, or what should we install or not?” That's a very good question. Let me go back to one of those. For some of these, unfortunately, the answer is there is no good answer.
Todd: Most of these this month, unfortunately, it only says Microsoft is investigating.
Chris: Yeah, you've got a Known Issue, there's a symptom, and there's the symptom. A lot of these you can hold off on, but there's a risk to holding off. As Todd pointed out, if you do the cumulative, this is on the pre-Windows 10 systems, if you do the cumulative, many of those Known Issues existed. If you do the security-only bundle, those Known Issues did not exist in the security-only, When you're doing this, the cumulatives often include feature changes, functionality, fixes, and things like that, which is often where the issues come into play.
If any of you are familiar with the quality preview, which happens at the end of the month, Microsoft had a cadence for security of everything releasing on Patch Tuesday, and non-security releasing later in the month. The quality preview is that later in the month non-security update, and that's where, if you see all the news articles and notices about issues happening, the majority are happening in the quality preview changes they're making. With the security updates, it seems they're doing things in a more controlled way and there's more testing around it. When it comes to those quality changes, there seem to be more quality issues on a month-to-month basis.
The guidance there is security first. If you continue to run into issues, you may want to switch to the security-only bundles. The quality preview, definitely do those for smaller groups of systems, not across the board. For the cumulative rollup, if you're not experiencing issues, you could stay there, but if you see impacts month to month, it may be a good idea to go to the security-only bundle. Mahesh, each of those links we have in the deck are going to a KB page, and for most of them this month, unfortunately, they're still researching the issues they're seeing. That mshtml.dll issue is one of those and there's no resolution to it or workaround.
Wayne had a question: “Is it recommended that you set up a test group to apply updates to, and the next week apply to the rest to your environment?” I would say it depends on each environment and what you're able to absorb bandwidth wise. We have a lot of large customers and a lot of highly security-minded customers. We've seen a lot of success with a lot of our customers, and we're talking 80,000+ endpoints patched in a two-week time span. They use a model where they roll out to a group of nonuser test systems on day one. It’s a replication of a real-world environment, but no direct user is involved. That first week, start rolling it out to an early adopters group. If you don't have the ability to do more robust automation or testing around critical applications and how they may be impacted, it's important to get representatives from each of those critical application groups as part of your pilot program. I would say you want to patch them by Wednesday or Thursday that first week to give them a couple of days to flush out any issues.
The goal there is you validate everything is working and continue the following week on track, or you identify there's an issue and you can block the rest of the group from being impacted. One way to get buy-in from the business-line owners or application owners managing those teams of people you need help from is to say, ''If we do this, there's always the risk that when we roll out the patch to the rest of the group, you'll see issues that could impact the group as a whole. If you give us one or two people from your group, they get the patch first. They can test the necessary pieces, like making sure that printing or image rendering still work, or accessing client data or exported data, whatever major tasks typically break in those applications, that user can test those things and make sure they’re good to go before the rest of the group is impacted.”
I've had those conversations with application owners several times over the years, and the majority of them are amenable to giving one of the most-effective users in their group to you for that pilot group. Their interest is in keeping the majority of their group working. If they have one person who can keep it together when something breaks and help fix the issue, that person getting impacted for a day or two is a lot better than the entire team being impacted for several days. That's a good strategy to use.
If you're using test systems only, there's always the risk that say testing login makes sure you can still log in, but what about all of the other things like printing and image rendering? There were several GDI fixes this month alone, and there are typically several each month. Any of those can break the way a document is presented or how functionality may behave in a product.
I think we got to everybody's questions. I apologize if we missed any. We're a little over time, so we're going to wrap up for today. The recording, presentation, and everything will be available on our page in a few hours. Thanks again for joining us, and thank you Todd and Brian for joining us on this month's webinar.
Todd: You bet.
Brian: You bet. Thank you.
Chris: Thanks, everyone, talk to you next month.