The Power of Exclusion
*This post originally appeared on the AppSense blog prior to the rebrand in January 2017, when AppSense, LANDESK, Shavlik, Wavelink, and HEAT Software merged under the new name Ivanti.
One thing that comes up often when talking to AppSense Support is whether customers have configured various exclusions. Additionally, the following articles, which address exclusions, are the two most viewed articles on the myAppSense.com Knowledgebase:
- TN-150517 - Recommended AppSense Excludes for Citrix Hooks
- TN-150728 - Recommended Anti-Virus/AppSense Exclusions
To understand why these are so commonly mentioned, it's important to understand the background behind them, why you should consider adding your own for performance reasons, and the stability issues that lead to the creation of these articles.
If we look at what products from AppSense, Citrix and most anti-virus vendors do, they all typically load one or more DLLs into other processes and often hook many of the same API calls:
- File & Registry – Read/Write
- Process launching (creation)/DLL (image) loading
- Network Access
Consider an application being personalized by Environment Manager, on a XenApp/XenDesktop client, with Citrix EdgeSight and a modern AntiVirus product.
If the application attempts to read or write data, the request must go through various hooks for each of these products. AppSense Environment Manager needs to check if the path is included/excluded and redirect the request if appropriate, EdgeSight monitors the response times and resource usage, AntiVirus scans the data, then there may also be DLP/Encryption and other systems monitoring software. At each stage, DLL’s are loaded into the process which, typically, will be communicating with an agent/service, which, in turn, may also pass the request to another back-end system.
For a process that is relatively “self contained” (one that does not interact with many other applications, and only reads/writes its own data) this isn't a particularly large problem. However, for processes that interact with many other processes, and potentially many sessions, this overhead increases significantly and compounds further when other systems also monitor and intervene.
Let's go back to our user launching an application. EmUser.exe will check for any process start actions, and check whether the process is personalized (managed), either directly as part of an Application Group or as a child process of a managed application. It will also check if “auto discover” or “data collection” is enabled.
The results of these checks may require EmUser.exe to connect to the Personalization Server, copy/edit file & registry data, run custom/execute actions and conditions, etc. Some of this will be done via PVC.dll within the newly launched process, some via EmUser.exe, but potentially also additional “external” processes - wscript.exe, for example (in the case of a custom (VBScript) action. Each of these associated actions may trigger code-paths within any monitoring or security product, including AppSense Performance Manager and Application Manager products, growing the footprint.
So what does this mean for the knowledgebase articles I mentioned at the start? Well, these should be taken as a good starting point, but really you need to think about what products “overlap” (and also determine which ones are relevant). The main technical reasons for these exclusions typically came about after lengthy, multi-vendor debugging sessions, both on live systems or from large memory dumps. While these investigations may lead to code-changes that ultimately prevent the issue from occurring, the saying “Prevention is better than cure” has its place here too, but with a handy side-effect improvements in logon times, application launch times and general resource usage as a whole.
Leading by Example
To shed some light on this, I ran a few simple tests on a server with the following spec/software (details on the exclusions are provided after the results):
Server 2008 R2 Enterprise SP1- 2GB Ram and 2 cores (running on VMware ESXi 5.5, HP BL465c G7 blades)
- AppSense Environment Manager 8.5.738.0 (EM 8.5 SP1)
- AppSense Application Manager 8.8.285.0 (AM 8.8 SP1)
- AppSense Performance Manager 8.2.320.0 (PM 8.2)
- Citrix XenApp 6.5, HFRP4
- Citrix EdgeSight Agent 220.127.116.11
- McAfee VirusScan Enterprise 8.8.04001
- Microsoft Office 2010
- Microsoft App-V 18.104.22.16870
Comparing logon times (averaged over 10 logons for the same user) with:
- No exclusions: 26 seconds
- All exclusions: 22 seconds
Launching a managed application (Firefox.exe), average over 10 launches:
- No exclusions: 7.25 seconds
- Citrix exclusions: 6.5 seconds
System Resource usage with 6 concurrent sessions (5 mandatory users / 1 admin, details taken with the same users and same apps / web pages open):
|Exclusions||CPU||Handles||Private Bytes||Virtual Bytes||Working Set||Paged pool||Non-paged pool|
All the above memory counters are Kb, and while the reductions may not seem monumental, bear in mind the relatively small number of users and apps featured. Each user had four managed applications open but all with blank documents and fairly static webpages. In a real world scenario, the savings would be much greater.
One aspect I had hoped would be better was the logon time, so I looked at what happens during logon and the majority of “work” that EmUser was doing, based on the logs I collected, was handling Process Start notifications – this is something I’ve previously discussed on the AppSense Exchange forum, but is worth mentioning here too.
The original post can be found here, and exclusions are addressed towards the end of this post.
To expand on this a little, we have to check every process we detect to see if it matches each condition and there’s code paths that are called regardless.
This includes checking against all personalized applications/application groups to see if the application is managed, obtaining and checking the parent process in case that is managed, taking a snapshot of any registry keys/values that are monitored for Desktop Refreshes, and comparing that snapshot once the application has started.
The first step here is to enable Environment Manager logging and capture a user logging on, then turn logging off again.
Check the EmUser-_Session-ID....log file and look for the point where we finish restoring Desktop Icons, this is the most realistic identifier of the point the desktop is “ready to use”
L4 T51720 84871548 [EmTaskRestoreIconsPMM] [EXIT]
Working back from this point, keep searching for “ProfileManager - Detected startup” –, which will give you a list of every process that EmUser.exe handled during this time, in my case:
L3 T57308 84871127 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process runonce.exe
L3 T57132 84869052 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process explorer.exe
L3 T56872 84867554 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process wfshell.exe
L3 T56780 84866681 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process acregl.exe
L3 T56704 84866384 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process cmd.exe
L3 T56660 84866244 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process persinfo.exe
L3 T56344 84864809 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process sftdcc.exe
L3 T52408 84864668 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process wscript.exe
L3 T55760 84864528 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process userinit.exe
L3 T55440 84851018 [ProfileManager::HandleApplicationStartMsg] ProfileManager - Detected startup of process rdpclip.exe
I also look at the very first logline:
L0 T55528 84847009 [CDbgEntry::LoadSystemDetails()] System Details Follow:
Looking at the tick counts from when the log started, to when the desktop icons were placed (84871548 – 84847009, the count is milliseconds since the system booted), this tells me it took 24 seconds, and the following processes all launched during this time (actually, explorer.exe and wfshell.exe also launched within this window, but as EmUser needs to know when these start, these should not be excluded):
runonce.exe / acregl.exe / cmd.exe / persinfo.exe / sftdcc.exe / userinit.exe / rdpclip.exe / wscript.exe
I added each of these to the AsVfxLdr “Exceptions” and EmDriver “ProcessWhiteList” as I have no need to run process start actions or personalize them, then timed logon again – taking just the two most significant loglines from earlier:
L0 T57704 87078057 [CDbgEntry::LoadSystemDetails()] System Details Follow:
L4 T57732 87097043 [EmTaskRestoreIconsPMM] [EXIT]
Subtracting 87078057 from 87097043 gives us 18,986 milliseconds – roughly 19 seconds. So with a simple registry tweak we’ve just reduced logon by a further 5 seconds – 9 seconds if you include the earlier exclusions – a reduction of around 30%!
Something else worth looking out for are processes that run repeatedly in the background, but also have no associated process start / stop actions, and have no interaction (i.e., a child process) of any applications that are personalized – such as SearchProtocolHost.exe / SearchFilterHost.exe (part of Indexing) – excluding these reduces the ongoing work that EmUser is doing mid-session (and also makes debug logs a lot less “chatty” – something support are always grateful for!)
For the CtxHook ExcludedImageNames, I used the following (I removed some of the items from the technotes as these relate to older versions of AppSense software):
AMAgent.exe, AMAgentAssist.exe, Cca.exe, WatchdogAgent.exe, WatchdogAgent64.exe, EmCoreService.exe, EmLoggedOnUser.exe, EmSystem.exe, EmUser.exe, EmUserLogoff.exe, PmAgent.exe, PmAgentAssist.exe
For the EdgeSight UviProcessExcludes:
AMAgent.exe; AMAgentAssist.exe; Cca.exe; WatchdogAgent.exe; WatchdogAgent64.exe; EmCoreService.exe; EmLoggedOnUser.exe; EmSystem.exe; EmUser.exe; EmUserLogoff.exe; PmAgent.exe; PmAgentAssist.exe
I also excluded some of the Citrix / McAfee processes from Application Manager’s UrmHookEx / AppHookEx “custom settings” (prior to Application Manager 8.8, these had to be set in the registry of the machine where the console was installed, however, Application Manager 8.8 exposes this within the Advanced Settings dialog under the Custom Settings tab)::
CdfSvc.exe / CtxSvcHost.exe / fbserver.exe / frameworkservice.exe / HCAService.exe / IMAAdvanceSrv.exe / imasrv.exe / mcshield.exe / rscorsvc.exe / rtclient.exe / xte.exe
Just to note, Environment Manager exclusions contain a number of defaults, which are often added to with subsequent releases – the list below is from an 8.5 SP1 install, and is used for both ProcessWhiteList and Exceptions, to which I’ve now added the items I mentioned above:
AmAgent.exe / AmAgentAssist.exe / AMDllInjectionAssist.exe / Blast.exe / ccmexec.exe / ccSvcHst.exe / checkdns.exe / CheckTermSrv.exe / cmstart.exe / concentr.exe / conhost.exe / CpsVcTest.exe / ctxhide.exe / DataNow_Service.exe / DataNow_Tray.exe / DataNowCA.exe / Dwm.exe / emcoreservice.exe / emexit.exe / EmLoggedOnUser.exe / EmMon.exe / EmMonSurrogate.exe / emsystem.exe / emuser.exe / emuserlogoff.exe / FixExtend.exe / FujiFldL.dll / FujiSynapseBridge.exe / gpscript.exe / ICAListener.exe / IMATest.exe / ISXAgent.exe / MigrateUserScans.exe / mmvdhost.exe / mpnotify.exe / mscorsvw.exe / pcoip_server_win32.exe / picaDispMgr.exe / PicaEuemRelay.exe / PicaSessionAgent.exe / picaSessionMgr.exe / PicaUserAgent.exe / PmAgent.exe / PmAgentAssist.exe / pnagent.exe / PrintIsolationHost.exe / RequestTicket.exe / smss.exe / spoolertest.exe / spoolsv.exe / SSOManHost.exe / ssonsvr.exe / startssonsvr.exe / svchost.exe / SynapseUpdateManager.exe / taskhost.exe / taskmgr.exe / TPAutoconnect.exe / TPAutoConnSvc.exe / TPVCGateway.exe / TSInstallSWUpdates.exe / TSManager.exe / tstheme.exe / verclsid.exe / ViewMPServer.exe / vmtoolsd.exe / VMWareViewClipboard.exe / VMwareview-RdeServer.exe / werfault.exe / wermgr.exe / wfcrun32.exe / WmiPrvSE.exe / wssm.exe
To summarise the above – excluding processes not only resolves a number of complex interoperability / stability issues, but can also make significant improvements in logon times, application launch times, and overall system-wide CPU & memory usage – in the case of a terminal services environment, this will ultimately lead to improved user load.
The published Knowledgebase articles on this subject give a good starting point, but paying close attention to what products are installed on your systems really can give you something for nothing!
Keep an eye out for future posts regarding performance, stability and troubleshooting…