Troubleshooting the RES Workspace Manager Relay Server

This post originally appeared on the RES blog before the company was acquired by Ivanti in July 2017.

From time to time customers turn to support with issues like “My agents aren’t getting a license assigned” or “changes I have made in the configuration are not coming through to the agents.”

An often forgotten detail is that one or more RES Workspace Manager Relay Servers are installed – which is understandable as it’s a component doing its job silently on the background and easy forgotten. However, it is key component in your RES Workspace Manager environment when installed and should not be left unnoticed.

In this article I will give you some insights on what an RES Workspace Manager Relay Server does and how to do some basic troubleshooting for this vital component.

RES Workspace Manager Relay Server Communication

To properly troubleshoot a Relay Server related issue it is important to have some basic understanding of how a client communicates to the Relay Server, how the Relay Server handles transactions, and how it communicates with the datastore.

The Relay Server polls for changes every x-times (default 5 seconds) and will download the changes to its local cache.

If changes are made in the RES ONE Workspace console, the “Update relay server cache on change” will trigger an update of the relay server(s) at the set interval.

To download the changes from (or process the transactions to) the datastore, the Relay Server can process up to 25 concurrent transactions to the datastore to speed up processing. The total amount of connections to the datastore is unlimited. (Figure 1)

When an agent connects to the Relay Server, it can have up to four connections to the RES Workspace Manager Relay server. When connected, the agent will compare the clients’ global update GUID with the global update GUID of the Relay Server. If they are different it will compare the update GUIDS for each component to see what has exactly changed and download this directly from the Relay Server.

During logon, a separate connection is created on which session information and licensing requests are handled. The Relay Server will handle this traffic with the highest priority.

Transactions consist of logging session information, usage tracking, and license information. These are uploaded to the Relay Server in different files. The transactions are uploaded to the Relay Server in %PROGRAMDATA%\RES\Relay Server\{environmentGUID}\Transactions.

When processing the files it will create a connection to the database to process the job. If this takes too long it will create a next connection to process the next job. This way it can create up to 25 simultaneous connections to process transactions.

Basic Troubleshooting

‘Eliminate all other factors, and the one which remains must be the truth’ – Sherlock Holmes

This is the main commandment to basic troubleshooting – eliminate factors and don’t jump to conclusions. Let’s start eliminating factors my dear Watson.

Agent Troubleshooting

Let’s start simple with the main suspect (the Client) and investigate if the agent is able to communicate with the Relay Server and if the datastore is reachable. To easily check if the issue could be Relay Server related you can connect one agent directly to the datastore again.

Type the command %respfdir%\svc\res.exe /config to open the RES Workspace Manager Agent connection configuration window.

1. Change the connection type to Database and click OK, now test if the issue persists.
2. If a full RES Workspace Manager installation is present on the agent, then you can also launch the RES Workspace Manager console. This will access the datastore directly, bypassing the Relay Server. This way you can determine if the datastore is accessible.
3. Check if the RES Workspace Manager Agent is able to connect and to which Relay Server it is connecting. This is more advanced and some assistance by RES Support is needed as this can only be determined from the RES Workspace Manager trace file.

Relay Server Troubleshooting

So, the client has a sound proof alibi, let’s give the Relay Server a thorough investigation.

The datastore is reachable and connecting directly to the datastore is solving the issue so there must be something wrong with the Relay Server.

There are several things to check on the Relay Server:

• Microsoft Windows Application event logs is one of our main witnesses. It’s helpful to see what it reports and if there are Relay Server related errors. A lot of the event error messages have a nice, related knowledgebase article.
• Check the available disk space on the Relay Server, which should have been reported on the Application event log. By default the Relay Server cache is stored on the system drive and the Relay Server will stop caching if there’s less then 500Mb disk space available. This store will open again when the available disk space increases by 10% (to 550mb). Also check that the registry key:

HKLM\SOFTWARE\RES\Workspace Manager\RelayServer\MinimumFreeDiskSpace has been set (DWORD, decimal value in MB).

• Investigate for clues of what the Relay Server is doing. Run the Relay Server in interactive mode by typing the command C:\Program Files\RES Software\Workspace Manager\Relay Server\RelayServer.exe /interactive. It never has been that easy to get through to someone I guess, but you’ll have to click Start to really have him talking.

1. Here you can see if the Relay Server is listening on a port and on which port (1943 in this example), this is 1942 by default.
2. The connected clients are shown. Unfortunately no client names are displayed.
3. How many agent and/or other relay server requests are being sent to this relay server.
4. Concurrent connections to the Relay Server. This is not the number of agents connected to the Relay Server but a nice rule of thumb is to divide this by two to get an estimate.
5. The different cache stages, of the different hosted environments, are displayed here. Normally only one or two stages for a hosted environment are displayed. More stages for a single environment may indicate an issue in processing the cache stages.
6. Transactions queued displays the amount of transactions in the Relay Servers’ transaction queue, check if this is high. When the Trans queued reaches 50,000 the Relay Server will stop accepting connections until it reaches 40,000 queued transactions.

• Is there a high CPU load on the Relay Server? If so, then there’s normally an issue on the Database server end. You could check if the RES Workspace Manager database is not full (is it set to fixed size or set to auto grow).
• If you want to see the mental health of your relay server then you can dump the Relay Server state to a RelayServer_State.xml file with the command C:\Program Files\RES Software\Workspace Manager\Relay Server\RelayServer.exe /dump

The file will be created in C:\Program Files\RES Software\Workspace Manager\Relay Server\RelayServer.exe and could become handy for RES Support when you raise a case.

• Some more in depth information over time can be gathered by using the Performance counters which are available for the Relay Server. Add to these counters the %Avg. Disk Read Queue Length and %Avg. Disk Write Length or the disk on which the Relay Server Cache resides, %Processor time and if needed some memory counters like Available Mbytes and Committed Bytes and you should have nice impression about what’s going on. For a detailed list of Relay Server Performance counters have a look at the RES Workspace Manager Administration Guide.

Relay Server Troubleshooting With Split Database

A common mistake when splitting the RES Workspace Manager datastore is to put the Secondary datastore on ‘cheap’ storage – it’s just logging right? Not entirely. If there’s any datastore that needs a good performance, it’s the secondary datastore which can contain logging and / or usage tracking. Depending on the RES Workspace Manager configuration, a lot of logging can be recorded and usage tracking can generate a lot of data which needs some decent iops on your storage.

When having Relay Server related issues and a split datastore there’s a way to troubleshoot to determine if there are issues on your primary or secondary datastore.

1. Stop the RES Workspace Manager Relay Server Service.
2. Move the transactions in the transactions folder to a safe location.
3. Move the “.sa” files back to the transactions folder for testing the primary datastore or the “.plla” files to test the secondary datastore.
4. Start the RES Workspace Manager Relay Server service again. A list will build and the transaction files will be processed. Keep a keen eye on the performance while processing the files. If there’s an issue on either the primary or secondary datastore you will notice a considerable difference in performance.
5. When finished troubleshooting don’t forget to perform steps one through four for the other transaction files to get them processed, otherwise incomplete logs may result.

Roundup

With this article I tried to give some insights on the Relay Server and how to troubleshoot this vital component in your RES Workspace Manager environment. After this advanced investigation it still may be necessary to have some CSI department – RES Support at the scene. To contact RES Support, you can reach us on Twitter @res_support or call/email the team. We are all happy to solve the case.