Ivanti Blog: Posts by

Environment Manager Configuration Part 2

Fri, 22 Apr 2016 07:31:34 Z

*This post originally appeared on the AppSense blog prior to the rebrand in January 2017, when AppSense, LANDESK, Shavlik, Wavelink, and HEAT Software merged under the new name Ivanti.

So in my last blog I introduced you to the Configuration API, I explained how to create the basic structure of a configuration with associated options and settings. In this blog I will cover creating actions and conditions on various triggers. Covering the whole conditions & actions set in this blog will be just too long and drawn out, so to keep things interesting I’ve compacted it down to just cover the most common we see in configuration files. Hopefully this will keep you interested and more importantly will allow you to learn as you go.

So to kick things off lets pick up where we left off last time.

Let’s recall our basic structure.

In a standard configuration the Parent-Child relationship needs to be maintained and is maintained with the use of GUID’s throughout the configuration. Therefore each condition or action is created using its Parents GUID.

Conditions

Getting to the nitty gritty of it all, let’s create a UserIsAdmin condition under the child node of the ‘User Pre-Desktop’ trigger

[code lang="powershell"]
# Add a node to the User Pre-Desktop trigger named 'Node Pre-Desktop'
$userpredesktopnodeid = $config.AddNodeToTrigger("UserPreDesktop", "Node Pre-Desktop")

# Add a child node to the node created above using its NodeID variable.
$predesktopchildnode1 = $config.AddNodeToParent($userpredesktopnodeid, "Child Node")

# Create as IsAdministrator Condition
$config.AddActionOrCondition($predesktopchildnode1, $isAdminCondition_true)
[/code]

To explain this code above lets break it down line by line. First things first we need to create the ‘UserIsAdministrator’ API constructor in which we declare what parameters are passed. In this case it simply is as follows:-

UserIsAdministrator(bool isAdministrator, bool evaluateOnce, bool stopIfFails)

**Remember PowerShell is ran line by line as opposed to compiled, so constructors have to be declared in code before they can be called. This seems obvious, however it is quite easy to miss structure your code and fall into this trap.

Translating to the following line of code:-

[code lang="powershell"]
# Create the Condition Object
$isAdminCondition_true = New-Object EMConfigAPI.Conditions.UserIsAdministrator($true, $true, $false)
[/code]

Next we have to add the Condition to the node we want:-

[code lang="powershell"]
# Add a child node to the node created above using its NodeID variable.
$predesktopchildnode1 = $config.AddNodeToParent($userpredesktopnodeid, "Child Node")

# Create as IsAdministrator Condition
$config.AddActionOrCondition($predesktopchildnode1, $isAdminCondition_true)
[/code]

So to look at this programmatically, we call the AddActionOrCondition API with the first parameter being the parent node of the node you wish to add the object to and the second is the actual variable of the condition or action.

Actions

The same logic is used for actions. Firstly using the API constructor we create the object variable

DriveMap(string driveLetter, string remotePath, bool unMapAtLogoff, ConnectAs connectAs, string userFriendlyName, bool stopIfFails)

DriveMap(string driveLetter, string remotePath, bool unMapAtLogoff)

Notice this API uses a ‘Connect As’ Enumerator. In instances like these only a pre-defined input can be used. In this case it is CurrentUser, System, AsUser or Default

[code lang="powershell"]
$driveMap_E = New-Object EMConfigAPI.Actions.DriveMap("E", "\\server\share", $true)
[/code]

Finally we add the action to the appropriate parent object. In this example I have added it to the ‘UserIsAdministrator’ condition created earlier.

[code lang="powershell"]
$childnode1_drivemap = $config.AddActionOrCondition($childnode1_isAdmin_true, $driveMap_E)
[/code]

Reuseable Nodes & Conditions

Reuseable’s work in much the same way, however utilize a slightly different API.

[code lang="powershell"]
$reusable_osIs81Condition = $config.InsertReusableConditionNode("OS is Windows 8.1")

$w81Condition = $config.AddActionOrCondition($reusable_osIs81Condition, $osIsW81)

$reusable_OS81Node = $config.InsertReusableNode("OS Check 8.1")

$osCondition = $config.AddReusableCondition($reusable_OS81Node, $reusable_osIs81Condition)
[/code]

This time the extra step is to Insert the reusable object in the library, using the InsertReusableConditionNode & InsertReusableNode API’s. Then we create the action object and then link it to the appropriate object in the configuration.

I hope everyone has followed up until this point, however I appreciate just reading isolated lines of code can be difficult to relate to real world examples. That is why I have bundled together a script which contains a lot more working examples for actions and conditions.

Once again this is not a fully optimized script, just an example of how objects are linked together and to show what is required to generate a configuration.

Again I hope that this has given you some food for thought and please feel free to comment if there is any need for clarification.

DesktopNow & Java Virtual Machines (JVM’s)

Tue, 26 May 2015 21:30:11 Z

*This post originally appeared on the AppSense blog prior to the rebrand in January 2017, when AppSense, LANDESK, Shavlik, Wavelink, and HEAT Software merged under the new name Ivanti.

Last time I started you off with a nice easy blog around troubleshooting. Now it’s time to venture deeper down the rabbit hole.

Let’s take a look at how our DesktopNow suite interacts with Java applications.

We have seen a trend of issues reporting that Java applications will no longer launch when either AM or EM has been upgraded. So I think it’s worth delving a little deeper into and explain what's occurring.

Before I start to explore the issues in technical detail, first I should explain at a high-level how Java environments are created. Most people who are admins have heard of the phrase Java Virtual Machine or JVM, but I bet most haven’t stopped to think, "Why on earth is it called that?" Well, It's because Java actually does run in the same principle as any other virtual machine, at least in memory reservation. Resources need to be assigned or reserved before the machine is started. This means that the memory ‘heap’ is first reserved, a small proportion is allocated, and then the ‘machine’ is started. Now those of you who regularly use Java applications in your environment will know that Java can be ran with a number of switches, particularly:

Command line options: -Xms:<min size> -Xmx:<max size>

The -Xmx switch is the one we need to focus on as it sets the maximum heap size for the JVM.

-Xmx
This option sets the maximum Java heap size. The Java heap (the “heap”) is the part of the memory where blocks of memory are allocated to objects and freed during garbage collection. Depending upon the kind of operating system you are running the maximum value you can set for the Java heap can vary.
-Xms
The -Xms option sets the initial and minimum Java heap size. The Java heap (the “heap”) is the part of the memory where blocks of memory are allocated to objects and freed during garbage collection.Note: -Xms does not limit the total amount of memory that the JVM can use.

Java will try to allocate a single contiguous block of memory upon each launch (although opinion seems to differ, seeming dependent on which JVM specification is used). When the –Xmx switch is used, you are effectively specifying the maximum amount of memory and therefore the size of the block that requires reserving. Ideally applications should be profiled to understand their memory requirements.

Generally these types of issue occur more frequently when using a 32-bit JVM. Primarily this is due to the limited amount of heap memory available within this architecture. The OS defaults to 2GB process or ‘User’ address space and the other 2GB for ‘Kernel’ or system processes, however this can be changed. 64-bit processes do not have such architectural limitation as the address spaces are chasmic in comparison at 8TB (Terabytes) for both User and Kernel address space. This is increased again to 128TB in Windows 8.1 and Server 2012 R2.

Memory Limits for Windows and Windows Server Releases

It’s easy to forget when talking about applications and their interactions that this is in the highest level of the OS stack. In this case, the issue emanates from a lot lower down in the stack. We are talking about how memory is allocated across the entire system, not just for applications. Therefore knowledge at this lower level is required for a true understanding of the issue.

With the issues we have seen here, the primary reason is that the AM, EM and related Microsoft libraries (DLL’s) are loading into an empty memory before Java is loaded. This fragments the free memory space available to Java when its starts.

Known Issues

Technically there are 3 distinct issues caused all in principle by the same root cause outlined above.

Application Manager - AMAppHook.dll & AmLdrAppInit.dll are loaded into a static shared memory address, 0x67C00000 & 0x67CF0000 respectively. Technically this is in no way incorrect, however it causes a fragmented memory region.

With the arrival EM 8.5 and AM 8.8 we introduced a new version of Microsoft Detours which forced allocations into an earlier memory address space causing more fragmentation of available free space than in previous versions.

The Detours library enables interception of function calls with user-defined calls. The first few instructions of the target function with an unconditional jump to the user-provided detour function. Instructions from the target function are preserved in a trampoline function. The trampoline consists of the instructions removed from the target function and an unconditional branch to the remainder of the target function.

In layman's terms, Detours allows you to insert your own functional calls to replace or extend the target function. The original target functions code is placed in a function called a ‘trampoline’, which can be called after the newly defined user-provided function.

The trampoline function is the key, as with any other object, as it requires a memory allocation. Detours use a complex algorithm to allocate this function's memory. The algorithm was seen to be allocating memory 1GB above or below the target library when the issue occurred. When the Java application failed, this allocation was in the middle of the memory block that the JVM was attempting to allocate and so in turn failed its memory allocation.

Detours uses a start address that needs to be below where the system DLL’s are located. Using a top-down allocation approach it works down through the memory ranges until a large enough free-block is found. In version 2.1 the address space is high up in the memory spectrum at 0x70000000. Beyond this range the ‘system DLL’s are loaded. As Operating Systems are becoming increasing more complex the number of system DLL’s is increasing, needing a larger proportion of reserved address space. In version 3.0 of Detours Microsoft changed the starting address space further down the spectrum at 0x50000000 which now coincidentally aligns with the ASLR (Address Space Layout Randomization) address range. Any trampoline functions are loaded in below this function, again increasing the likelihood of memory fragmentation problems under certain circumstances.

Applied Optimizations

Code optimization will reduce the occurrences for these types of issues. In Application Manager 8.9 Service Pack 1 the Application Manager modules have been ASLR enabled.

This will randomize the memory allocation, between 0x50000000 & 0x78000000 upon each system boot. Now for those of you paying attention you might be saying that ‘Hang on, 0x50000000 is lower than 0x67C00000! So the issue will still occur? ‘Correct, well half-way at least. The ASLR implementation means that each allocation is completed top-down, so the address ranges at the very top of the memory spectrum are more than likely ASLR enabled. Below is a representation of my workstations Memory heap, from this you can see that from 0x76EA0000 > 0x6E5E0000 each ‘Image’ has been tagged with an ASLR flag.

Further optimization in the form of a delay configurable via the Console > ‘Custom Settings > AppHookDelayLoad’ have been added into this release to delay the Application Manager Hook. This will to allow Java to allocate it's memory before Application Manager or Microsoft Detours reducing the likelihood of a fragmented memory region.

Working with Microsoft it was validated that this behaviour was due to a change in the Detours source code. A solution has been implemented to optimize the code base for 32-bit versions in
- Environment Manager 8.5 Service Pack 1 Agent Hotfix 5 - TN-151461
- Application Manager 8.8 Agent Hotfix 1 - TN-151205

There are currently no optimizations in place to reduce occurrences of these type of issues; however I would stress that there have been no reported cases attributed to it.

Summary

To sum up, under normal working conditions the Operating System can handle fragmented memory space well and without any performance impact. Java specifications can differ, but 32-bit applications created using a specification that architecturally requires a contiguous reservation will more than likely meet this issue one day or another. Unfortunately for application vendors such as AppSense, there is no quick fix or win to help our customers. All we can attempt to do is optimize our components as efficiently as possible to reduce occurrences and maximize the available memory for these applications.

Helping Us Help You: AppSense Troubleshooting Best Practices

Tue, 03 Feb 2015 17:05:56 Z

*This post originally appeared on the AppSense blog prior to the rebrand in January 2017, when AppSense, LANDESK, Shavlik, Wavelink, and HEAT Software merged under the new name Ivanti.

As a Senior Solutions Engineer, I see many high priority / high impact issues that require detailed analysis of a complex set of moving parts. This often includes memory dump analysis, in-depth log analysis, cross-platform interoperability issues, and generally finding creative solutions to tricky problems. The urgency and complexity of these situations requires me to work in an efficient environment. I like to be organized enough that if I need to reach for a resource, I know exactly where it is. If I can’t, quite frankly, it ‘drives me up the wall’ with frustration.

I’m sure that many reading this relate to this feeling and think, “Well, duh! Doesn’t everyone?” Well actually, no, not intentionally. When a high-pressure situation occurs, sometimes the simplest and easiest things that can make our lives easier can be overlooked. We all do it on a daily basis in one way or another in all aspects of life.

My goal is always to take a structured approach to troubleshooting. Where there’s structure, there’s efficiency. Microsoft provides an excellent article relating to ‘Troubleshooting Methodology’that outlines some clear and distinct phases of the troubleshooting process.

One of my pet hates is seeing or dealing with incidents where there is unnecessary “to’ing and fro’ing” back to customers asking for more information or gathering more logs. Unfortunately, some of the time it is a necessity, but other times the information could have been captured much earlier in the incident.

Logging a call with a software vendor is one of those times when you really need the process of to be as smooth and painless as possible. (Normally, because your boss is breathing down your neck, right? I remember those times well! ). It’s really important to ensure that the right information gets to the right people as soon as possible. Fortunately, we’ve developed a set of tools and tips over time that can make this process easier and much more efficient for our customers.

By the time I generally see issues, they have often reached Phase 4 of the methodology mentioned above. However, it’s common for more ‘Discovery’ to be required, which is time consuming from both the vendor’s point of view and the customer side as well. So reducing this amount of time is a must to create an efficient working environment and, more importantly, reduce incident time-to-resolution (TTR) for our customers.

Think about your past support interactions with AppSense and other software vendors and ask yourself a question: “After I have logged an incident, do I then get asked a lot of ‘non-specific’ questions?” If the answer is ‘yes,’ how much of that could have been provided at the point the first call or e-mail was made? The likely answer is a fair amount.

It’s in the best interests of both AppSense and you as a customer that we spend as much time as possible in troubleshooting (Phases 3, 4 & 5). This tends to go very smoothly with an efficient and collaborative approach, where all parties’ primary goal is to resolve the issues(s) reported by getting to root-cause.

Typically every support vendor has a minimum list of things that are required at the point of incident. Here at AppSense, we don’t impose a rigid list, but we do have a general list of information we will ask for, if they haven’t been provided at first point of contact:

· AppSense Support Toolkit – Data Collection Output (Formally the Support Script)^{1, 2}

· Product debug logs – (identifying problem sessions where applicable)

· Detailed reproduction steps or events building up to when the problem occurred

· Personalization Server Export (for Environment Manager Personalization incidents only)

¹Data Collection - This feature collects AppSense configurations, event, registry and file information details to aid support troubleshoot the issue.; ²We are planning on expanding the AppSense Support Toolkit to be the ‘go to’ place of all support related utilities, so watch this space for further details!;³ All of our products have detailed logging which can be toggled on/off.

If the AppSense Support Toolkit – Data Collection (AST-DC) or Personalization Server Export output has been provided recently (~3 months) and is still relevant, then clearly it is not required again. Simply reference any previous incident numbers and the information can be retrieved.

Below I’ve included links to common tools that support will most likely ask you to use (dependent on products installed) after an incident has been logged:

Common Support Tools
AppSense Support Toolkit

Environment Manager Agent	Application Manager Agent
8.5 EMLoggers85.zip 8.4 EMLoggers84.zip 8.3 EMLoggers83.zip Console Logging TN-150485	8.8 AMLoggers88.zip 8.7 AMLoggers87.zip 8.6 AMLoggers86.zip Rules Analyzer Logging TN-150988
Performance Manager Agent TN-151463
Client Communications Agent TN-150830

Ideally, the best sets of logs are the ones that are a combination of items that can be chronologically matched. They are even more useful when combined with third party utilities such as the Sysinternals suite from Microsoft: https://technet.microsoft.com/en-gb/sysinternals/bb545021.aspx

For example, the following are some sample support scenarios with corresponding example of the log data that is key to successful troubleshooting.

Example 1

Issue Symptom: EM Personalization Settings are not synchronizing on application stop.

Required Logs: AST-DC Output; EM Agent Logs; PS Export.

Example 2

Symptom: AM is blocking files through a proxy when I my configuration should allow it.

Required Logs: AST-DC Output; AM Agent Logs; AM Rules Analyzer Log

Example 3

Symptom: EM Console crashes when doing x,y,z.

Required Logs: AST-DC Output; EM Console Logs

Each situation is a little bit different, but providing a relevant set of logs will help up quickly and efficiently process your incident. While we hope you won’t ever need to contact us for help, if you do, taking that little extra time upfront goes a long way towards expediting the resolution of your incident. And that’s good for everyone.

Watch out for additional troubleshooting tips and tricks – coming soon to the AppSense blog!