Advanced Logging for Personalization Server
*This post originally appeared on the AppSense blog prior to the rebrand in January 2017, when AppSense, LANDESK, Shavlik, Wavelink, and HEAT Software merged under the new name Ivanti.
In this blog I’m going to discuss using advanced logging of Personalization HTTP traffic, with the goal of keeping life simple for those of us tasked with managing or troubleshooting Personalization systems.
The communication between a single Environment Manager Personalization server and endpoints is relatively simple and requires very little setup beyond the AppSense installer wizards. One of the major components added to the system, if you decide to scale out tier-2* of the standard architecture, will be a network load balancer. In just a couple of configuration steps, we can ensure that when adding a load balancer the troubleshooting and monitoring of your Personalization servers remains just as simple.
Put simply, a load balancer allows the load from the Environment Manager agents running on each endpoint to be balanced across multiple Internet Information Services (IIS) servers. However, a side effect of this is that the standard IIS logging facility can only log the client IP address (c-ip) field, which will now be the translated IP address from the load balancer, so information detailing which endpoint is communicating with which server is lost. If, for example, AppSense Support needs to troubleshoot Personalization data synchronization issues, it’s important to be able to ascertain which specific components were involved in each particular synchronization i.e. – endpoint X -> server Y –> database Z.
Custom HTTP Request Header
Load balancers generally provide functionality to allow you to insert custom header fields into HTTP traffic as it passes through the device. Traditionally this functionality is used to add the client IP address from each originating endpoint request, into a custom ‘x-forwarded-for’ header in the translated request. This practice has now been standardized in HTTP/1.1 by the IETF in RFC 7239.
The steps to configure the header insert vary depending on the appliance. For a Citrix NetScaler it’s merely a case of ticking the option ‘ClientIP_ Header’ in the advanced settings of a service and then entering the header name, but I would advise to refer to your particular vendor documentation for detailed guidance.
When setup, the HTTP requests through the load balancer reaching each Personalization server should look something like this.
Now the Personalization server itself doesn’t require this information, any client information required for Personalization to function will be added into the XML content of the request.
Also, if you have default IIS logging enabled on the server you won’t see this information in the usual IIS log files (although in IIS 8.5 the custom field can be added to the default IIS logging).
What you really need, in order to make full use of the new header information, is Microsoft’s Advanced Logging extension for IIS.
Installing this extension adds a new option in IIS Manager – ‘Advanced Logging’;
Note: Please also review the following Microsoft Hotfix - “Memory usage in IIS Advanced Logging increases when filtering is used for IIS 7.0 and IIS 7.5”
By default, advanced logging will be disabled, but will be configured with a log definition named %computername%-Server, with a default set of logging fields selected, so the first thing to configure is an additional logging field for the new ‘x-forwarded-for’ header using the ‘Edit Logging Fields…’ option;
Following this we need to create log definitions customized for Personalization requests; I suggest splitting these into two definitions (the option to ‘clone log definition…’ is very handy here), creating two base log files which separate client and load balancer monitoring activity:
- Environment Manager client activity
Base file name - %computername%-Personalization
- Filter – URI-Stem /PersonalizationServer/session.aspx
- OR – URI-Stem /PersonalizationServer/config.aspx
- OR – URI-Stem /PersonalizationServer/sync.aspx
- Load balancer monitoring activity
Base file name - %computername%-Personalization-Monitor
- Filter - URI-Stem /PersonalizationServer/dbmonitor.aspx
- OR - URI-Stem /PersonalizationServer/pingmonitor.aspx
Now configure the selected fields for each log definition, adding only the necessary will make the resulting log files easier to review, so I’d suggest the following;
We now have the following setup;
Log to network share
Using the ‘Edit Log Directory…’ option, enter a UNC path to a network file share. Configuring the logs to be generated this way will ensure they don’t take up space on the local file system, and will also allow analysis of the logs generated from multiple servers without requiring you to connect to each server. As all of the log definitions are set using the %computername% variable, the origin of each will be clear from the file name.
Analyse using Microsoft Log Parser 2.2
Now let’s explore a troubleshooting scenario – a user reports that their Internet Explorer settings were missing today; they were working on endpoint X. You do a quick check and today endpoint X had an IP address of 10.1.1.1. You have the Personalization IIS log share mapped to your admin machine as drive “L:” and you need to quickly find the Personalization server which serviced the synchronisation request…
- Install the Microsoft Log Parser 2.2:
- Run the following command:
LogParser.exe "SELECT date, time, c-protocol, cs-uri-stem, sc-status, sc-bytes, cs-bytes, c-ip, x-forwarded-for, s-computername INTO 'L:\Endpoint_X_Activity.CSV' FROM "L:\*.log" WHERE x-forwarded-for = `10.1.1.1’ ORDER BY date, time" -i:W3C -dQuotes ON
You now have a csv report file containing the activity from that endpoint IP address, ordered by date & time, which shows both the endpoint IP, the Load Balancer translated IP, and the Personalization server name. All components involved in the synchronization are now known, and so the troubleshooting Discovery phase (https://technet.microsoft.com/en-us/library/cc786052%28v=ws.10%29.aspx) is potentially (should an error have been logged on one of these components) already complete.