*This post originally appeared on the AppSense blog prior to the rebrand in January 2017, when AppSense, LANDESK, Shavlik, Wavelink, and HEAT Software merged under the new name Ivanti.

Blog_Banners_main-page[1]

I was recently investigating an issue whereby endpoints constantly showed a “Deployed (%)” status of 0% within the Management Console, yet the “Last Response” time was getting regularly updated, and (more perplexing still) upon checking the affected machines manually we found one or more agents / configurations had indeed been installed, despite the reported 0% deployed status.

To give a bit of background information, there are several key components within the OS that the Client Communications Agent (CCA) can’t live without:

  • WinHTTP – Used by the agent to poll the Management Server
  • Background Intelligent Transfer Service (BITS) – used to download agent / configuration packages and upload event files
  • Windows Installer – Like any software that performs installations, if Windows Installer has a problem, so does the CCA

With this in mind, if the agent (cca.exe) is running, but things aren’t working as expected – you need to ensure all the above are playing ball.

Back to the issue at hand – the customer had noticed events such as the following being raised in the application event logs of the affected clients:

Log Name: Application
Source: AppSense Client Communications Agent
Date: 22/10/2015 12:24:44
Event ID: 9750
Level: Error
Computer: RK-XA65-01.RK1.support.local
Description:
The Deployment Agent failed to contact the Management Server. Error '500'.

Server side, the application event logs will show an event similar to the below (a somewhat verbose error!), logged at the same time as the Error 500 is logged on the client:

Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 22/10/2015 12:24:44
Event ID: 1309
Task Category: Web Event
Level: Warning
Keywords: Classic
Computer: RK-AMS-PROD.RK1.support.local
User: N/A
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 22/10/2015 12:24:44
Event time (UTC): 22/10/2015 11:24:44
Event ID: 7b9afa6849ca4d7d875331574686562b
Event sequence: 26
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/1/ROOT/ManagementServer/Deployment-1-130899860234261398
Trust level: Full
Application Virtual Path: /ManagementServer/Deployment
Application Path: C:\Program Files\AppSense\Management Center\Server\Web Site\Deployment\
Machine name: RK-AMS-PROD
Process information:
Process ID: 3140
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at Deployment.Manifest.CreateOrUpdateMachinePackageRow(Guid machineKey, List`1 ccaPackages, PLATFORM requestPlatform, MachinePackagesDataSet machinePackages, GroupPackagesDataSet groupPackages, PackagesDataSet packages)
at Deployment.Manifest.ModifyMachinePackages(Guid machineKey, Boolean isPendingDeletion, Boolean isDeployingNativeConfigs, PLATFORM requestPlatform, List`1 ccaPackages, PackagesDataSet packages, GroupPackagesDataSet groupPackages, MachinePackagesDataSet machinePackages)
at Deployment.Manifest.SetMachinePackages(Guid machineKey, Guid groupKey, Boolean isPendingDeletion, Request request, Boolean isDeployNativeConfigs)
at Deployment.Manifest.CheckDoRefreshForManifestRequest(MachinesRow machineRow, GroupsRow groupRow)
at Deployment.Manifest.Build()
at Deployment.ManifestHandlerListener.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Request information:
Request URL: http://rk-ams-prod/managementserver/Deployment/Manifest.aspx
Request path: /managementserver/Deployment/Manifest.aspx
User host address: 10.70.17.84
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\NETWORK SERVICE
Thread information:
Thread ID: 12
Thread account name: NT AUTHORITY\NETWORK SERVICE
Is impersonating: False
Stack trace:    at Deployment.Manifest.CreateOrUpdateMachinePackageRow(Guid machineKey, List`1 ccaPackages, PLATFORM requestPlatform, MachinePackagesDataSet machinePackages, GroupPackagesDataSet groupPackages, PackagesDataSet packages)
at Deployment.Manifest.ModifyMachinePackages(Guid machineKey, Boolean isPendingDeletion, Boolean isDeployingNativeConfigs, PLATFORM requestPlatform, List`1 ccaPackages, PackagesDataSet packages, GroupPackagesDataSet groupPackages, MachinePackagesDataSet machinePackages)
at Deployment.Manifest.SetMachinePackages(Guid machineKey, Guid groupKey, Boolean isPendingDeletion, Request request, Boolean isDeployNativeConfigs)
at Deployment.Manifest.CheckDoRefreshForManifestRequest(MachinesRow machineRow, GroupsRow groupRow)
at Deployment.Manifest.Build()
at Deployment.ManifestHandlerListener.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

You can ignore most of the above – the main two lines are simply:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.

Essentially – the manifest that the client sent during its poll contained some null data, and the server isn’t expecting this.

Support gathered some CCA debug logs, and could see the point where the 500 error was returned:

L5  T6448 4548552 [WinHttpClient] INFO: Sending request
L5  T6448 4548552 [WinHttpClient] INFO: Receiving response
L5  T6448 4548599 [WinHttpClient] INFO: status code 500
L1  T6448 4548599 [WinHttpClient] ERROR: HTTP 500 server Run Time error

Thus far we know WinHTTP is working, but the server returns an error when trying to process something we sent it. Shortly before this log entry, we could see the agent generating an XML document (the manifest) to send to the Management Server, detailing the currently installed packages, preceded by calls relating to MSIInterface and CMsiPackage, suggesting we’re now in the realms of Windows Installer. This process begins with checking which packages are installed:

L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] MsiEnumRelatedProductsW returned: 259
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Checking for upgradecode {7befc17d-2c22-467a-b640-7627d91765c3}
L3  T6448 4548536 [Registry] INFO: got upgrade code type msi/agent
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Found related product {5C6FA9E4-05C9-4DB6-B51D-0A0A0DC7FE1A}
L5  T6448 4548536 [CMsiPackage] INFO: entering CMsiPackage()
L5  T6448 4548536 [CMsiPackage] INFO: entering GetUpgradeCode()
L5  T6448 4548536 [CMsiPackage] INFO: entering QueryPackage()
L5  T6448 4548536 [CMsiPackage] INFO: leaving QueryPackage()
L5  T6448 4548536 [CMsiPackage] INFO: leaving GetUpgradeCode()
L5  T6448 4548536 [CMsiPackage] INFO: entering GetVersionString()
L5  T6448 4548536 [CMsiPackage] INFO: entering GetProductName()
L5  T6448 4548536 [CMsiPackage] INFO: leaving CMsiPackage()
L5  T6448 4548536 [CPackageInfo::Action] [Info] Returning Action. Current value [99]
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Product name AppSense Application Manager Agent 8 FR7 SP1
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Product version 8.7.131.0
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Type msi/agent
L5  T6448 4548536 [CMsiPackage] INFO: entering IsInstalled()
L5  T6448 4548536 [CMsiPackage] INFO: entering GetProductCode()
L5  T6448 4548536 [CMsiPackage] INFO: leaving GetProductCode()
L5  T6448 4548536 [CMsiPackage] INFO: Checking install state for Product Code:{5C6FA9E4-05C9-4DB6-B51D-0A0A0DC7FE1A}
L5  T6448 4548536 [CMsiPackage] INFO: state is 'use default, local or source'
L5  T6448 4548536 [CMsiPackage] INFO: leaving IsInstalled() - returning bInstalled true
L7  T6448 4548536 [MSIInterface::GetPatches] [Enter] ProductCode:'{5C6FA9E4-05C9-4DB6-B51D-0A0A0DC7FE1A}' InstalledState=[99]
L9  T6448 4548536 [MSIInterface::GetPatches] [Verbose] Found Patch '{357D9DDB-BB18-47B8-84BE-21E5FB8A2219}'
L7  T6448 4548536 [MSIInterface::GetPatchName] [Enter] Product Code:'{5C6FA9E4-05C9-4DB6-B51D-0A0A0DC7FE1A}' Patch Code:'{357D9DDB-BB18-47B8-84BE-21E5FB8A2219}'
L5  T6448 4548536 [MSIInterface::GetPatchName] [Info] Name:'AppSense Application Manager 8 FR7 SP1'
L7  T6448 4548536 [MSIInterface::GetPatchName] [Exit] Success:'1' Error:'0'
L7  T6448 4548536 [MSIInterface::GetPatchFilePath] [Enter] Patch Code:'{357D9DDB-BB18-47B8-84BE-21E5FB8A2219}'
L7  T6448 4548536 [MSIInterface::GetPatchFilePath] [Exit] Success:'0'
L7  T6448 4548536 [MSIInterface::GetPatchXML] [Enter] MSP FilePath:''
L7  T6448 4548536 [MSIInterface::GetPatchXML] [Exit] Success:'0'
L5  T6448 4548536 [MSIInterfaceBase::GetAppliedPatch] [Info] Found related applied patch '{357D9DDB-BB18-47B8-84BE-21E5FB8A2219}'
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Product BASE version
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Product patch code {357D9DDB-BB18-47B8-84BE-21E5FB8A2219}
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Product patch name AppSense Application Manager 8 FR7 SP1
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Package is installed. Adding to upload list

We then add the installed (AppSense) packages to the manifest:

L5  T6448 4548552 [CPackageInfo::AsXml] [Info] productcode {6F0D01E6-D1AD-43A8-9AE3-37B4755421FD}
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] filename
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] version
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] name AppSense Environment Manager Agent 8 FR4 SP3
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] action Installed
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] patchAction Installed - 6
L7  T6448 4548552 [CPackageInfo::AsXml] [Exit]
L7  T6448 4548552 [CPackageInfo::AsXml] [Enter]
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] productcode {5C6FA9E4-05C9-4DB6-B51D-0A0A0DC7FE1A}
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] filename
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] version
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] name AppSense Application Manager 8 FR7 SP1
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] action Installed
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] patchAction Installed - 6
L7  T6448 4548552 [CPackageInfo::AsXml] [Exit]
L7  T6448 4548552 [CPackageInfo::AsXml] [Enter]
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] productcode {753AE139-6958-4071-8140-88D53E7DD572}
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] filename
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] version 8.5.431.0
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] name AppSense Client Communications Agent 8 FR5 SP1
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] action Installed
L5  T6448 4548552 [CPackageInfo::AsXml] [Info] patchAction Installed – 6

Looking at the above entries, it’s clear there are some “empty” values, for example:

L5  T6448 4548552 [CPackageInfo::AsXml] [Info] version

Working back up through the log file, you can see during with Windows Installer related entries, we seem to be missing a version number here too:

[PackageInstaller::InstalledFileList] [Info] Product BASE version

I assigned the same agents to a machine here, and captured a CCA log of my (working) agent when it polled, so that I could get some like-for-logs of it working / failing to compare the differences.
The first job was to narrow down the customers 120MB logfile to something a bit more manageable. At the start / end of the above sections of logs, I was able to determine all the relevant loglines were written between these two log entries:

L5  T6448 4548536 [CommsManager] INFO: entering CreateUploadManifest()
(a few hundred lines later)
L5  T6448 4548552 [CommsManager] INFO: leaving CreateUploadManifest()

So I copied this section from each log to new files, ie, broken.txt and working.txt.

Comparing like-for-like logs can be made very easy with the use of a file comparison tool, such as KDIFF (https://kdiff3.sourceforge.net/) – but before opening two trimmed down log files, I used find/replace to remove the thread ID and timestamp which would obviously be different, meaning all the lines in both txt files were now more like this:

L5  [PackageInstaller::InstalledFileList] [Info] Checking for upgradecode {4336b40f-68f7-43be-98b8-75fcfcef33c7}
L3  [Registry] INFO: got upgrade code type msi/agent
L5  [PackageInstaller::InstalledFileList] [Info] Found related product {6F0D01E6-D1AD-43A8-9AE3-37B4755421FD}
L5  [CMsiPackage] INFO: entering CMsiPackage()
L5  [CMsiPackage] INFO: entering GetUpgradeCode()

I was now at a good point to select both files and compare them in KDiff – this immediately led me to the area where mine log returned the file path and version, and the customers didn’t:

The good:

L5  [MSIInterface::GetPatchName] [Info] Name:'AppSense Environment Manager Agent 8 FR4 SP3'
L7  [MSIInterface::GetPatchName] [Exit] Success:'1' Error:'0'
L7  [MSIInterface::GetPatchFilePath] [Enter] Patch Code:'{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L5  [MSIInterface::GetPatchFilePath] [Info] FilePath:'C:\Windows\Installer\19eff.msp'
L7  [MSIInterface::GetPatchFilePath] [Exit] Success:'1'
L7  [MSIInterface::GetPatchXML] [Enter] MSP FilePath:'C:\Windows\Installer\19eff.msp'
L7  [MSIInterface::GetPatchXML] [Exit] Success:'1'
L7  [Patch::ParseXML] [Enter]
L5  [XMLParse] INFO: entering CreateXmlReader()
L5  [XMLParse] INFO: leaving CreateXmlReader()
L5  [Patch::ParseXML] [Info] MSI Version:'8.4.195.0' Patch Family:'AgentPatchFamily' MSP Version:'8.4.495.0'
L7  [Patch::ParseXML] [Exit] Success:'1'
L7  [MSIInterface::GetPatches] [Exit] Patches Found:'1'
L5  [MSIInterfaceBase::GetAppliedPatch] [Info] Found related applied patch '{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L5  [CPackageInfo::PatchAction] [Info] Returning PatchAction. Current value [99]
L5  [PackageInstaller::InstalledFileList] [Info] Product BASE version 8.4.195.0

And the bad (I’ve removed some duplication here as we fail and retry a number of times):

L5  [MSIInterface::GetPatchName] [Info] Name:'AppSense Environment Manager Agent 8 FR4 SP3'
L7  [MSIInterface::GetPatchName] [Exit] Success:'1' Error:'0'
L7  [MSIInterface::GetPatchFilePath] [Enter] Patch Code:'{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L7  [MSIInterface::GetPatchFilePath] [Exit] Success:'0'
L7  [MSIInterface::GetPatchFilePath] [Enter] Patch Code:'{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L7  [MSIInterface::GetPatchFilePath] [Exit] Success:'0'
L7  [MSIInterface::GetPatchXML] [Enter] MSP FilePath:''
L7  [MSIInterface::GetPatchXML] [Exit] Success:'0'

L5  [MSIInterfaceBase::GetAppliedPatch] [Info] Found related applied patch '{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L7  [MSIInterface::GetPatchXML] [Enter] MSP FilePath:''
L7  [MSIInterface::GetPatchXML] [Exit] Success:'0'
L5  [CPackageInfo::PatchAction] [Info] Returning PatchAction. Current value [99]
L5  [PackageInstaller::InstalledFileList] [Info] Product BASE version

Note - the loglines often hint at the underlying APIs that are being used, in this case:

L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] MsiEnumRelatedProductsW returned: 259

MsiEnumRelatedProducts function

https://learn.microsoft.com/en-us/windows/win32/api/msi/nf-msi-msienumrelatedproductsa

L7  T6448 4548536 [MSIInterface::GetPatchFilePath] [Exit] Success:'0'

MsiGetPatchInfo function

https://learn.microsoft.com/en-us/windows/win32/api/msi/nf-msi-msigetpatchinfoa

I wrote a small console application to call these functions and could see using Process Monitor (procmon.exe), that certain files and registry keys were being accessed. Based on this, I deleted the cached MSI file created by Windows Installer, in this case, 'C:\Windows\Installer\19eff.msp'

At each poll I did indeed then get an error 500 – although a new log showed that I was still getting the filepath returned, it was just the “version” that was missing – suggesting MsiGetPatchInfo requires access to the actual (cached) .msp / .msi files to retrieve version information:

L7 [MSIInterface::GetPatchFilePath] [Enter] Patch Code:'{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L5 [MSIInterface::GetPatchFilePath] [Info] FilePath:'C:\Windows\Installer\19eff.msp'
L7 [MSIInterface::GetPatchFilePath] [Exit] Success:'1'
L5 [PackageInstaller::InstalledFileList] [Info] Product BASE version

I searched the registry for references to one of the files within C:\Windows\Installer, which led me to the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\6E10D0F6DA1D8A34A93E734B574512DF

I’d also been searching for Windows Installer related errors, and found a few posts suggesting the above registry keys were just the MSI Product codes in reverse – however, after some further investigation it’s not quite so straight forward.

The following log lines showed the Product Code, and any applied patch codes for those products:

L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Product name AppSense Environment Manager Agent 8 FR4 SP3
L7  T6448 4548536 [MSIInterface::GetPatchName] [Enter] Product Code:'{6F0D01E6-D1AD-43A8-9AE3-37B4755421FD}' Patch Code:'{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L5  T6448 4548536 [PackageInstaller::InstalledFileList] [Info] Product patch name AppSense Environment Manager Agent 8 FR4 SP3
L7  T6448 4548536 [MSIInterface::GetPatchName] [Enter] Product Code:'{5C6FA9E4-05C9-4DB6-B51D-0A0A0DC7FE1A}' Patch Code:'{357D9DDB-BB18-47B8-84BE-21E5FB8A2219}'

Going back to the registry key where I’d found the .msi file path:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\6E10D0F6DA1D8A34A93E734B574512DF

I realised parts of the GUID are reversed, but in different ways. If you take each (hyphenated) part of the Product Code and reverse them separately - the first three parts are reversed entirely, and the second two parts are reversed in "pairs":

Product Code: 6F0D01E6-D1AD-43A8-9AE3-37B4755421FD

Products “subkey”: 6E10D0F6DA1D8A34A93E734B574512DF

If  we add some commas / spaces – this is more obvious:

Code: 6F0D01E6 - D1AD - 43A8 - 9A,E3 - 37,B4,75,54,21,FD
Key:  6E10D0F6 - DA1D - 8A34 - A9,3E - 73,4B,57,45,12,DF

I next tried deleting the entire registry key above - but this made windows installer believe the agent wasn’t installed at all, which subsequently caused the CCA to try and install it again, which subsequently failed with error code 1651 – as it was already installed, but now I’d made Windows Installer distinctly unhappy!

I realised the same behaviour also existed for the patch itself – from the logs above we could see the  Patch Code '{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}' had been applied to Product Code:'{6F0D01E6-D1AD-43A8-9AE3-37B4755421FD}’.

Reversing the patch code in the same way:

3BA7A29D – 3077 - 47FF - 9E4B - 2BA0EBE4C341
D92A7AB3 – 7703 - FF74 - E9B4 - B20ABE4E3C14

Led me to the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Patches\D92A7AB37703FF74E9B4B20ABE4E3C14

I then deleted this key, and could now get the same error:

L7  T4304 787539 [MSIInterface::GetPatchFilePath] [Enter] Patch Code:'{3BA7A29D-3077-47FF-9E4B-2BA0EBE4C341}'
L7  T4304 787539 [MSIInterface::GetPatchFilePath] [Exit] Success:'0'
L5  T4304 787539 [CPackageInfo::PatchAction] [Info] Returning PatchAction. Current value [99]
L5  T4304 787539 [PackageInstaller::InstalledFileList] [Info] Product BASE version

So, where does this leave us?

Well, from a support perspective, we can see the issue is with Windows Installer, and not the Management Server or the CCA (good news for us!), and in this case we could determine the missing/corrupt file and registry data and copy this from a working client.

From a customer perspective (or anyone else still reading this!), you now have the information you need to find the cause of many Windows Installer related issues – in this case the CCA log will tell you which product code(s) and/or patch code(s) are causing the issue, however, if you’re manually installing or uninstalling software, either an MSI log or even just the product name will be sufficient to find the related Products registry key. Once you can determine where Windows Installer should contain the details about these products within the registry, and confirm these keys exist, you can also determine from the data within these keys that the version is correct, the path to the windows installer cache file, and use Orca to ensure the file is not corrupt (Orca is a free tool from Microsoft to inspect msi / msp files).

I’ve since installed different agents – but the details are consistent, ie, for AM 8.9 SP1 HF7 there will be a key that contains the product name and version, and the path to the installer cache file, as well as the original path to the MSI that was launched (InstallSource):

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\51E60D152D9747647907C9BFB8404400\InstallProperties
DisplayName = AppSense Application Manager Agent 8.9 SP1 HF7
DisplayVersion = 8.9.451.11
InstallSource = C:\Program Files\AppSense\Management Center\Communications Agent\download\{51d06e15-79d2-4674-9770-9cfb8b044400}\
LocalPackage = C:\Windows\Installer\1c81a5.msi

This also has two subkeys listed the (slightly reversed – see earlier comment) Patch Codes:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\51E60D152D9747647907C9BFB8404400\Patches\8E2BF302932C35B4AB349DA64D1DA594
DisplayName = AppSense Application Manager 8.9 SP1
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\51E60D152D9747647907C9BFB8404400\Patches\3E10CD669ADB33B4FA7481AB90648589
DisplayName = AppSense Application Manager 8.9 SP1 HF7

Each of these “Patches” subkeys, also has a corresponding key in the main patches key, ie:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Patches\3E10CD669ADB33B4FA7481AB90648589

Which tells you where the cache file is:

LocalPackage = C:\Windows\Installer\1c81a7.msp

Tip: If you want to find out what the cached files are in C:\Windows\Installer, you can add the “Title” column in windows explorer.

You can also use this if you have an error uninstalling an application, you can find the related MSI in either C:\Windows\Installer (using the Title column above), or the original installation MSI, and use Orca to find the product code from the MSI – from here you can subsequently work backwards to determine the related registry keys and any associated patches, and validate all cached .msi/.msp files are present and correct.