Troubleshoot EMC NetWorker KM

KM Behavior is Unchanged After Upgrade

Check the version of the KM from the main Infobox. If it has not changed, then the installation is not complete. Make sure both the PATROL Console and the PATROL Agent are uninstalled and installed correctly during the KM upgrade

EMC NetWorker Icon Missing After Loading

• Check that the KM is loaded using NSR_LOAD.kml and that NSR_MAIN is loaded.
• Check that there is no KM version mismatch between the PATROL Console and the PATROL Agent. Check the messages in the SOW to verify this.
• Check whether the PATROL Agent user has necessary privileges added in the Agent’s Access Control List (/AgentSetup/accessControlList) in order to read and write to the Agent Configuration Database.

Unable to Find NSR_LOAD.kml

• Check that the Load KM browser is looking for *.kml files under <PATROL_HOME>/lib/knowledge folder.
• EMC NetWorker KM files have not been installed correctly under the PATROL installation directory on the PATROL Console.

KM Application Instances Do Not Appear

• Check that the KM instance limits have not been exceeded. Look for error messages in the SOW, and increase the instance limits for affected objects using the menu Configuration > Instance Limits….
• Check whether the server icon is in offline state. None of the data collectors will be executed until the server is enabled and online.
• If the KM is configured for multi-node mode monitoring, some components are not monitored on the passive cluster node.

KM Configuration Menus are Disabled

By default, the EMC NetWorker KM runs in Central Monitoring Administration Mode (CMA Mode), allowing the configuration to be received from BMC Helix/TrueSight Operations Manager CMA policy. Check that the KM is setup to run in Classic Mode.

KM Objects Disappear from the Console

• Check whether there is a KM version mismatch between the PATROL Console and the PATROL Agent, possibly after an improper upgrade of the KM. Check the messages in the SOW to verify this.
• Check that the EMC NetWorker KM login details are still valid. Has the password changed on the system? Look for error messages in the SOW, and check for additional information in the last annotation point for parameter NSRLoginStatus.

Old Active Save Groups are not Removed

By default, all active save groups are monitored, and they are exempted from ageing. It is possible to change this behavior by unchecking the Keep Monitoring Active Save Groups Indefinitely box in the Save Groups Configuration window accessible from the Save Groups instance Configuration > Save Groups.

Old Acknowledged Save Groups Kept in pconfig

By default, the KM stores all acknowledged save groups. Use the following PSL through PATROL Console to keep only the last <Number> of jobs on <node-id>:

%PSL pconfig("REPLACE", "/Runtime/NSR/<node-id>/NSR_JOB/jobAcknowledgementCapacity", <number>);

Replace <node-id> with the appropriate node ID of the NetWorker server and restart the PATROL Agent. Leave the <node-id> empty for localhost monitoring.

CPU and Memory Usage is too High

CPU and memory usage will depend on the size and complexity of your environment and your EMC NetWorker KM configuration. As you increase data collection frequency, increase the number of servers and components monitored by the KM, your CPU and memory usage will increase.

When monitoring a standard local installation of EMC NetWorker using EMC NetWorker KM, the PATROL Agent will consume between 5MBytes and 10MBytes of additional system memory. An enterprise installation of EMC NetWorker Server with multiple Storage Nodes, Clients, Jukeboxes and Save Groups can consume more memory (as per other KMs used by the PATROL Agent). The memory usage of EMC NetWorker KM can be reduced by:

• disabling monitoring unnecessary component instances
• disabling unwanted components by setting their instance limits to 0 (zero)
• disabling unwanted collectors by using the PATROL Configuration Manager
• increasing the collector scheduling interval by using the PATROL Configuration Manager
• decreasing the instance limits to limit the number of instances created by the collectors

The data collectors in EMC NetWorker KM use EMC NetWorker command line interface to obtain EMC NetWorker information. Most of the performance degradation is associated with these command executions and the amount of data returned. It may improve the overall performance if the regular housekeeping is followed on all EMC NetWorker systems.

Network Traffic

When monitoring a NetWorker server through a local PATROL Agent, EMC NetWorker KM generates minimal network traffic. Most of the data is kept on the managed node. The amount of network traffic that it generates depends on the number of PATROL Consoles that are connected to the PATROL Agent and the frequency of data collection.

When monitoring remote NetWorker servers, some network traffic will be observed as it transfers the commands result over the network. The traffic depends on the amount of data polled during each command execution. When commands are expected to return large output, the KM is designed to use file transfers through SFTP (on UNIX/Linux) and Windows file shares (on Windows).

Parameters and Application Classes Refresh Takes too Long

Data collectors run according to their scheduling interval (polling cycle) defined in the KM. These intervals are defined for a standard environment with minimal resource impact; They can be customized from the PATROL Developer Console or PCM to suit your environment. Refer to the PATROL Console User Guide for more details.

Poor Performance of the Server/Storage Node

The performance of the Server/Storage Node may change after installing EMC NetWorker KM on a heavily used system. Depending on the complexity of your EMC NetWorker environment, the KM may consume more resources to interrogate the application and process the data. In such a complex environment, the EMC NetWorker KM may require some fine-tuning to optimize the available resources. Consider the following options:

• Disable the monitoring of unnecessary application instances. Refer to the section Filtering Elements to Monitor for more details.

• Increase the scheduling interval (polling cycle) for data collectors.

• Deactivate monitoring non-critical components by setting the Instance Limits to 0 (zero).

• Deactivate unnecessary data collectors during selected time intervals, where there is no EMC NetWorker activity. For example, if the save group monitoring can be disabled between 9 am and 4 pm every day, except weekends, then disable the save group data collector (NSRSaveGrpCollector) during this period by running the following PSL through the PATROL Console:

  %PSL pconfig("REPLACE","/Runtime/NSR/<node-id>/NSRSaveGrpCollectorMode","0|32400|57600|0|0|||||||||||0|0");

  Here the pconfig variable is named as: <collector name>Mode. Replace <node-id> with the appropriate node ID of the NetWorker server. Leave the <node-id> empty for localhost monitoring. The value contains the following details, delimited by a pipe (|):enabled (1)/disabled (0) data collection, default start/end times in a number of seconds since midnight, start/end times for non-default days starting from Sunday through to Saturday. Restart the PATROL Agent after the change.

• Defining a “no command execution window” for all collectors will pause running commands at peak times or during NetWorker maintenance windows. This can be set using the PSL below. Replace <node-id> with the appropriate node ID of the NetWorker server and restart the PATROL Agent:

  %PSL pconfig("REPLACE", "/Runtime/NSR/<node-id>/noExecuteWindow", "23:59:00|120");

  The value of this configuration variable is in format <start time in HH:MM:SS 24-hour clock>|<duration in seconds>. The above 23:59:00|120 sets all collectors to sleep between 23:59:00 and 00:01:00 (2 minutes) every day before executing commands. Also, this noExecutewindow supports multiple time windows:

  %PSL pconfig("REPLACE", "/Runtime/NSR/<node-id>/noExecuteWindow",["23:59:00|120","11:59:00|120"]);

• Purge unnecessary information in EMC NetWorker catalog databases and log files.

• If there are too many clients configured in EMC NetWorker, the NSRClientCollector and NSRGroupCollector may affect the overall performance. In such an environment, disable the NSRClientCollector, or set their instance limits to 0 (zero), using menu Configuration > Instance Limits.

• Refer to the Infinite Loop Errors section below for a possible PATROL internal scheduling delay that may impact the performance of the KM.

Infinite Loop Errors

If error messages in the SOW report that some EMC NetWorker KM data collectors may be in an infinite loop, check the setting of the tuning variable /AgentSetup/AgentTuning/pslInstructionMax.

PATROL Agent uses the pre-configured tuning variable (/AgentSetup/AgentTuning/pslInstructionMax) to stop running PSL functions in an infinite loop. When a PSL function reaches this maximum threshold, it reports this error and puts its execution to the back of the process queue. This will not only delay the data collector but also impact the performance of the system.

To resolve this situation, the maximum number of instructions should be increased to an optimum value depending on the complexity of your environment. It is required that the default value of 500,000 should be increased to at least 5,000,000 on a standard EMC NetWorker environment to enable the EMC NetWorker KM data collectors to execute without impacting your system.

If this still does not resolve the problem, you can disable this functionality by setting the value of the tuning variable to 0 (zero).

NSRLoginStatus in Suspicious (Warning) Status

This parameter will show a “suspicious” state if any command executed by EMC NetWorker KM fails.

• Check the annotation point on this parameter's first state change data point to look for failing commands. If an annotation point cannot be found or is not up-to-date, check the KM Status Report in the menu KM Status of the server icon. These errors are produced from the EMC NetWorker commands executed by the EMC NetWorker KM.
• Check that the operating system user configured in the menu Configuration > Login can execute all NetWorker commands and access the NetWorker files.

Enabling the Debug Mode

1. In the Console, right-click the server instance > KM Commands > Configuration > Debug… The NetWorker KM Debug Configuration dialog is displayed:

   

2. Select On for the debug switch(es) you want to enable. Refer to the table below to know the application classes and collector parameters included in the debug switch:

   Object Debug Switch	Application Classes	Collector Parameters
   Server	NSR_MAIN, NSR_SERVER	NSRCreateDistribution
   Database	NSR_DATABASE_CONTAINER, NSR_DATABASE	NSRDatabaseCollector
   Log	NSR_LOG_CONTAINER, NSR_LOG	NSRLogCollector
   Client	NSR_CLIENT_CONTAINER, NSR_CLIENT	NSRClientCollector
   Storage Node	NSR_MEDIASERVER_CONTAINER, NSR_MEDIASERVER	NSRStorageNodeCollector
   Jukebox/Drive	NSR_LIBRARY_CONTAINER, NSR_LIBRARY, NSR_LDEVICE, NSR_DEVICE_CONTAINER, NSR_DEVICE	NSRJukeboxCollector, NSRDeviceCollector
   Daemon	NSR_DAEMON_CONTAINER, NSR_DAEMON	NSRDaemonCollector
   Volume Pool	NSR_POOL_CONTAINER, NSR_POOL	NSRPoolCollector
   Group/Save Group/Pending Request	NSR_POLICY_CONTAINER, NSR_POLICY, NSR_PCLIENT, NSR_JOB_CONTAINER, NSR_JOB, NSR_JCLIENT, NSR_MOUNT_CONTAINER, NSR_MOUNT	NSRGroupCollector, NSRSaveGrpCollector, NSRRequestCollector

3. Set the Debug Options:

   • In the Debug End Time field, indicate the date and time after which debug information will no longer be logged.
   • In the Debug Path field, indicate where the debug file will be stored (by default: <PATROL_HOME>/log on UNIX/Linux systems; %PATROL_HOME%\log on Microsoft Windows systems). The debug folder must have at least 10MB of available disk space and read, write and execute permissions for both the PATROL Agent user and the EMC NetWorker KM login user
   • Select Send to File(s) to write all debug messages to files under the Debug Path.
   • Select Send to Task Window to view the debug messages as they occur in a task window on the PATROL Console, labeled NSR Debug Output for <node-id>. Before selecting this option, make sure the PATROL Console is connected to the PATROL Agent.
   • Select Both (Recommended) to send the debug messages both to files and task window.

4. Click OK first to start EMC NetWorker KM debugging

5. Click Yes to reload the KM and capture debug from the initial collection cycle.

If you cannot turn on the KM Debug following the method described above, you can do so by setting the appropriate PATROL Agent configuration variable with a timestamp value. This timestamp value determines when the debug should be turned off. For example, to turn on the debug for 60 minutes from now, run the following PSL through PATROL Console:

%PSL pconfig("REPLACE","/Runtime/NSR/<node-id>/<component>Debug",time()+3600);

Where <component> is either server for debugging the server discovery or name of the collector component in lower case followed by “Collector”, like daemonCollector for debugging daemon collector. Replace <node-id> with the appropriate node ID of the remote NetWorker server. For a local host, remove <node-id>.

Disabling the Debug Mode

The debug switches will be turned off automatically when the debug end time is reached. A tar/zip file is then automatically created under <PATROL_HOME>/lib/NSR/ and can be sent to the Sentry Software Support for help. It is also recommended to check the SOW or NSR_<port>.log file, stored in <PATROL_HOME>/log, for any error.

To manually stop debugging:

1. Access the NetWorker KM Debug Configuration dialog
2. Select Off for the debug switch(es) you want to disable
3. Click OK.

Preparing the Debug Files for Sending to Sentry Software

If you choose to only send the output to a task window, you can save the debug output as follows:

• Right-click in the NSR KM Debug Output for <node-id> window.
• Select Save As and specify a location to save data.

If you choose to send debug information to file(s), you will be asked to prepare the debug files for sending to Sentry Software:

• Click Yes to compress the files into a:

  • nsr_km_debug_<node-id>_<date>_<time>.tar file (UNIX/Linux systems). This file could later be found in $PATROL_HOME/log/
  • nsr_km_debug_<node-id>_<date>_<time>.zip file (Windows systems). This file could later be found in %PATROL_HOME%\log\

• Click No if you do not want to compress the files. You will then be asked if you want to retain or delete them.

If the compressed file is not created successfully, you can create it manually by including all files and sub-directories under $PATROL_HOME/log/ (on UNIX/Linux) or %PATROL_HOME%\log\ (on Microsoft Windows).

Appending Output Data to Existing Files

If you want to gather output from several debug sessions:

1. Disable the debug mode
2. Click No when asked to prepare the files for sending
3. Choose to Retain the debug files
4. Start the next debug session
5. Choose to append the output data to existing files.