Configure the Hardware Sentry Agent

Single host

Systems to monitor are defined under hosts with the below syntax:

hosts:

- host:
    hostname: <hostname>
    type: <host-type>
  <protocol-configuration>

where:

• <hostname> is the name of the host, or its IP address.

• <host-type> is the type of the host to be monitored. Possible values are:

  • win for Microsoft Windows systems
  • linux for Linux systems
  • network for network devices
  • oob for Out-of-band management cards
  • storage for storage systems
  • aix for IBM AIX systems
  • hpux for HP UX systems
  • solaris for Oracle Solaris systems
  • tru64 for HP Tru64 systems
  • vms for HP Open VMS systems Refer to Monitored Systems for more details.

• <protocol-configuration> is the protocol(s) Hardware Sentry will use to communicate with the hosts: http, ipmi, oscommand, ssh, snmp, wmi, wbem or winrm. Refer to Protocols and credentials for more details.

Multiple hosts

You can group hosts that share the same characteristics (device type, protocols, credentials, etc.) using one of the below syntax:

hosts:

- hostGroup:
    hostnames: [ <hostname1>,<hostname2>, etc.]
    type: <host-type>
  <protocol-configuration>

where:

• <hostname1>,<hostname2>, etc. is a comma-delimited list of hosts to be monitored. Provide their hostname or IP address.
• <host-type> is the type of the host to be monitored.
• <protocol-configuration> is the protocol(s) Hardware Sentry will use to communicate with the hosts: http, ipmi, oscommand, ssh, snmp, wmi, wbem or winrm. Refer to Protocols and credentials for more details.

or

hosts:
- hostGroup:
    type: <host-type>
    hostnames:
      <hostname>:
        extraLabels:
          host.name: server-01.local.net
          host.id: my-server-01-id
    # <hostname>:
    #   extraLabels:
  <protocol-configuration>

where:

• <hostname> is the name of the host, or its IP address. Using this format, you can provide extraLabels for each configured host.
• <host-type> is the type of the host to be monitored.
• <protocol-configuration> is the protocol(s) Hardware Sentry will use to communicate with the hosts: http, ipmi, oscommand, ssh, snmp, wmi, wbem or winrm. Refer to Protocols and credentials for more details.

HTTP

Use the parameters below to configure the HTTP protocol:

Example

hosts:

  - host:
      hostname: myhost-01
      type: storage
    http:
      https: true
      port: 443
      username: myusername
      password: mypwd
      timeout: 60

IPMI

Use the parameters below to configure the IPMI protocol:

Example

hosts:

- host:
    hostname: myhost-01
    type: oob
  ipmi:
    username: myusername
    password: mypwd

OS commands

Use the parameters below to configure OS Commands that are executed locally:

Example

hosts:
  - host:
      hostname: myhost-01
      type: linux
    osCommand:
      timeout: 120
      useSudo: true
      useSudoCommands: [ cmd1, cmd2 ]
      sudoCommand: sudo

SSH

Use the parameters below to configure the SSH protocol:

Example

hosts:
  - host:
      hostname: myhost-01
      type: linux
    ssh:
      timeout: 120
      useSudo: true
      useSudoCommands: [ cmd1, cmd2 ]
      sudoCommand: sudo
      username: myusername
      password: mypwd
      privateKey: /tmp/ssh-key.txt

SNMP

Use the parameters below to configure the SNMP protocol:

Example

hosts:

- host:
    hostname: myhost-01
    type: linux
  snmp:
    version: v1
    community: public
    port: 161
    timeout: 120s

- host:
    hostname: myhost-01
    type: linux
  snmp:
    version: v2c
    community: public
    port: 161
    timeout: 120s

- host:
    hostname: myhost-01
    type: linux
  snmp:
    version: v3-md5
    community: public
    port: 161
    timeout: 120s
    privacy: des
    privacyPassword: myprivacypwd
    username: myusername
    password: mypwd

WBEM

Use the parameters below to configure the WBEM protocol:

Example

hosts:

  - host:
      hostname: myhost-01
      type: storage
    wbem:
      protocol: https
      port: 5989
      timeout: 120s
      username: myusername
      password: mypwd

WMI

Use the parameters below to configure the WMI protocol:

Example

hosts:

  - host:
      hostname: myhost-01
      type: win
    wmi:
      timeout: 120s
      username: myusername
      password: mypwd

WinRM

Use the parameters below to configure the WinRM protocol:

Example

hosts:

  - host:
      hostname: server-11
      type: win
    winrm:
      protocol: http
      port: 5985
      username: myusername
      password: mypwd
      timeout: 120s
      authentications: [ntlm]

Disabling alerts (Not Recommended)

To disable Hardware Sentry's alerts:

• for all your hosts, set the disableAlerts parameter to true just before the hosts section:

  disableAlerts: true
  
  hosts: # ...
  

• for a specific host, set the disableAlerts parameter to true in the relevant host section:

  hosts:
  
  - host:
      hostname: myhost
      type: linux
    snmp:
      version: v1
      community: public
      port: 161
      timeout: 120s
    disableAlerts: true
  

Hardware problem template

When detecting a hardware problem, Hardware Sentry triggers an alert as OpenTelemetry log. The alert body is built from the following template:

Hardware problem on ${FQDN} with ${MONITOR_NAME}.${NEWLINE}${NEWLINE}${ALERT_DETAILS}${NEWLINE}${NEWLINE}${FULLREPORT}

To change this default hardware problem template:

• for all your hosts, configure the hardwareProblemTemplate parameter just before the hosts section:

  hardwareProblemTemplate: Custom hardware problem on ${FQDN} with ${MONITOR_NAME}.
  
  hosts: # ...
  

• for a specific host, configure the hardwareProblemTemplate parameter in the relevant host section:

  hosts:
  
  - host:
      hostname: myhost
      type: linux
    snmp:
      version: v1
      community: public
      port: 161
      timeout: 120s
    hardwareProblemTemplate: Custom hardware problem on myhost with ${MONITOR_NAME}.
  

and indicate the template to use when building alert messages.

For more information about the alert mechanism and the macros to use, refer to the Alerts page.

Basic authentication header

The Hardware Sentry Agent's internal OTLP Exporter authenticates itself with the OpenTelemetry Collector's OTLP gRPC Receiver by including the HTTP Authorization request header with the credentials. A predefined Basic Authentication Header value is stored internally and included in each request when sending telemetry data.

To override the default value of the Basic Authentication Header, add a new Authorization header under the exporter:otlp:headers section:

exporter:
  otlp:
    headers:
      Authorization: Basic <credentials>

hosts: # ...

where <credentials> are built by first joining your username and password with a colon (myUsername:myPassword) and then encoding the value in base64.

For more security, encrypt the Basic <credentials> value. See Encrypting Passwords for more details.

Warning: If you update the Basic Authentication Header, you must generate a new .htpasswd file for the OpenTelemetry Collector Basic Authenticator.

OTLP endpoint

The Hardware Sentry Agent's internal OTLP Exporter pushes telemetry signals to the OTLP Receiver through gRPC on port TCP/4317.

By default, the internal OTLP Exporter is configured to push data to the OTLP Receiver endpoint https://localhost:4317.

To override the OTLP endpoint, configure the endpoint property under the exporter:otlp section:

exporter:
  otlp:
    endpoint: https://my-host:4317

hosts: #...

Collect period

By default, Hardware Sentry collects metrics from the monitored hosts every minute. To change the default collect period:

• for all your hosts, add the collectPeriod parameter just before the hosts section:

  collectPeriod: 2m
  
  hosts: # ...
  

• for a specific host, add the collectPeriod parameter in the relevant host section:

  hosts:
  
  - host:
      hostname: myhost
      type: linux
    snmp:
      version: v1
      community: public
      port: 161
      timeout: 120s
    collectPeriod: 1m30s # Customized
  

Warning: Collecting metrics too frequently can cause CPU-intensive workloads.

Connectors

Hardware Sentry comes with the Hardware Connector Library, a library that consists of hundreds of hardware connectors that describe how to discover hardware components and detect failures. When running Hardware Sentry, the connectors are automatically selected based on the device type provided and the enabled protocols. You can however indicate to Hardware Sentry which connectors should be used or excluded.

Use the parameters below to select or exclude connectors:

Connector names must be comma-separated, as shown in the example below:

hosts:

  - host:
      hostname: myhost-01
      type: oob
    wmi:
      timeout: 120s
      username: myusername
      password: mypwd
    selectedConnectors: [ VMwareESX4i, VMwareESXi ]
    excludedConnectors: [ VMwareESXiDisksStorage ]

Note: Any mispelled connector will be ignored.

To know which connectors are available, refer to Monitored Systems or run the below command:

$ hws -l

For more information about the hws command, refer to Hardware Sentry CLI (hws)

Discovery cycle

Hardware Sentry periodically performs discoveries to detect new components in your monitored environment. By default, Hardware Sentry runs a discovery after 30 collects. To change this default discovery cycle:

• for all your hosts, add the discoveryCycle just before the hosts section:

  discoveryCycle: 15
  
  hosts: # ...
  

• for a specific host, add the discoveryCycle parameter in the relevant host section:

  hosts:
  
  - host:
      hostname: myhost
      type: linux
    snmp:
      version: v1
      community: public
      port: 161
      timeout: 120s
    discoveryCycle: 5 # Customized
  

and indicate the number of collects after which a discovery will be performed.

Warning: Running discoveries too frequently can cause CPU-intensive workloads.

Extra labels

Add labels in the extraLabels section to override the data collected by the Hardware Sentry Agent or add additional attributes to the Host Resource. These attributes are added to each metric of that Resource when exported to time series platforms like Prometheus.

In the example below, we override the host.name attribute resolved by Hardware Sentry with host01.internal.domain.net and indicate that it is the Jenkins app:

hosts:

- host:
    hostname: host01
    type: Linux
  snmp:
    version: v1
    port: 161
    timeout: 120
  extraLabels:
    host.name: host01.internal.domain.net
    app: Jenkins

In the example below, we configure several hosts which share the same characteristics (hostGroup) and override the host.name attributes resolved by Hardware Sentry with server-01.local.net and server-02.local.net and indicate that these hosts are Jenkins apps:

hosts:
- hostGroup:
    type: <host-type>
    hostnames:
      server-01:
        extraLabels:
          host.name: server-01.local.net
          host.id: my-server-01-id
          app: Jenkins
      server-02:
        extraLabels:
          host.name: server-02.local.net
          host.id: my-server-02-id
          app: Jenkins

Hostname resolution

By default, Hardware Sentry resolves the hostname of the host to a Fully Qualified Domain Name (FQDN) and displays this value in the Host Resource attribute host.name. To display the configured hostname instead, set resolveHostnameToFqdn to false:

resolveHostnameToFqdn: false

hosts:

- host:
    hostname: host01
    type: Linux

Job pool size

By default, Hardware Sentry runs up to 20 discovery and collect jobs in parallel. To increase or decrease the number of jobs Hardware Sentry can run simultaneously, add the jobPoolSize parameter just before the hosts section:

jobPoolSize: 20

hosts: # ...

and indicate a number of jobs.

Warning: Running too many jobs in parallel can lead to an OutOfMemory error.

Sequential mode

By default, Hardware Sentry sends the queries to the host in parallel. Although the parallel mode is faster than the sequential one, too many requests at the same time can lead to the failure of the targeted system.

To force all the network calls to be executed in sequential order:

• for all your hosts, enable the sequential option just before the hosts section (NOT RECOMMENDED):

  sequential: true
  
  hosts: # ...
  

• for a specific host, enable the sequential option in the relevant host section:

  hosts:
  
  - host:
      hostname: myhost
      type: linux
    snmp:
      version: v1
      community: public
      port: 161
      timeout: 120s
    sequential: true # Customized
  

Warning: Sending requests in sequential mode slows down the monitoring significantly. Instead of using the sequential mode, you could increase the maximum number of allowed concurrent requests in the monitored system, if the manufacturer allows it.

Timeout, duration and period format

Timeouts, durations and periods are specified with the below format:

Command line

By default, the Hardware Sentry Agent launches the OpenTelemetry Collector process using the following command line: otel/otelcol-contrib --config otel/otel-config.yaml --feature-gates=pkg.translator.prometheus.NormalizeName.

If you want to run your own distribution of the OpenTelemetry Collector or update the default program's arguments such as the --feature-gates flag, you need to override the OpenTelemetry Collector default command line by setting the commandLine property under the otelCollector section:

otelCollector:
  commandLine:
    - /opt/hws/otel/my-otelcol
    - --config
    - /opt/hws/otel/my-otel-config.yaml
    - --feature-gates=pkg.translator.prometheus.NormalizeName

hosts: # ...

Disabling the collector (Not recommended)

In some cases, you might want the Hardware Sentry Agent to send OpenTelemetry signals directly to an existing OpenTelemetry Collector running the gRPC OTLP Receiver, in which case, running a local OpenTelemetry Collector is unnecessary.

To disable the OpenTelemetry Collector, set the disabled property to true under the otelCollector section:

otelCollector:
  disabled: true

hosts: # ...

Environment

When Hardware Sentry is installed as a Windows service, the OpenTelemetry Collector may fail to start if it cannot connect to the Windows service controller. To address this issue, you can set the NO_WINDOWS_SERVICE environment variable to 1 to force the OpenTelemetry Collector to be started as if it were running in an interactive terminal.

You can set additional environment variables to be used by the OpenTelemetry Collector in the otelCollector:environment section (e.g.: HTTPS_PROXY):

otelCollector:
  environment:
    HTTPS_PROXY: https://my-proxy.domain.internal.net
    NO_WINDOWS_SERVICE: 1

hosts: # ...

Process output

By default, the Hardware Sentry Agent listens to the OpenTelemetry Collector standard output (STDOUT) and standard error (STDERR) and streams each output line to the logs/otelcol-<timestamp>.log file when the logger is enabled.

To print the OpenTelemetry Collector output to the console, set the output property to console under the otelCollector section:

otelCollector:
  output: console  # Default: log

hosts: # ...

To disable the OpenTelemetry Collector output processor, set the output property to silent under the otelCollector section:

otelCollector:
  output: silent   # Default: log

hosts: # ...

Working directory

By default, the OpenTelemetry Collector working directory is set to hws/otel. If your working directory is different (typically in heavily customized setups), add the workingDir attribute under the otelCollector section in config/hws-config.yaml:

otelCollector:
  workingDir: /opt/hws/otel

hosts: # ...

Important: The OpenTelemetry Collector might not start if the value set for the workingDir attribute is not correct, more especially if the otel/otel-config.yaml file uses relative paths.

Trusted certificates file

A TLS handshake takes place when the Hardware Sentry Agent's OTLP Exporter instantiates a communication with the OTLP gRPC Receiver. By default, the internal OTLP Exporter client is configured to trust the OTLP gRPC Receiver's certificate security/otel.crt.

If you generate a new server's certificate for the OTLP gRPC Receiver, you must configure the trustedCertificatesFile parameter under the exporter:otlp section:

exporter:
  otlp:
    trustedCertificatesFile: /opt/hws/security/new-server-cert.crt

hosts: # ...

The file should be stored in the security folder and should contain one or more X.509 certificates in PEM format.