Agents Administration - Tests
 

Default Parameters for HdpAppStatsTest

This test auto-discovers the queues spawned by the YARN scheduler, and for each queue, reports the count of jobs that are in different states of activity. In the process, the test points administrators to those queues that have one/more failed jobs, long-running jobs, and those with jobs that have not been assigned a container yet. This way, administrators can be proactively alerted to potential issues in resource allocations by the scheduler.

This page depicts the default parameters that need to be configured for the HdpAppStatsTest test.

  • The TEST PERIOD list box helps the user to decide how often this test needs to be executed.

  • The eG agent collects metrics using Hadoop's WebHDFS REST API. While some of these API calls pull metrics from the NameNode, some others get metrics from the resource manager. NameNode is the master node in the Apache Hadoop HDFS Architecture that maintains and manages the blocks present on the DataNodes (slave nodes). NameNode is a very highly available server that manages the File System Namespace and controls access to files by clients. To run API commands on the NameNode and pull metrics, the eG agent needs access to the NameNode's web port.

    To determine the correct web port of the NameNode, do the following:

    • Open the hdfs-default.xml file in the hadoop/conf/app directory.

    • Look for the dfs.namenode.http-address parameter in the file.

    • This parameter is configured with the IP address and base port where the DFS NameNode web user interface listens on. The format of this configuration is: <IP_Address>:<Port_Number>. Given below is a sample configuration:

      192.168.10.100:50070

    Configure the <Port_Number> in the specification as the Name Node Web Port. In the case of the above sample configuration, this will be 50070.

  • The eG agent collects metrics using Hadoop's WebHDFS REST API. While some of these API calls pull metrics from the NameNode, some others get metrics from the resource manager. NameNode is the master node in the Apache Hadoop HDFS Architecture that maintains and manages the blocks present on the DataNodes (slave nodes). NameNode is a very highly available server that manages the File System Namespace and controls access to files by clients.

    In some Hadoop configurations, a simple authentication User name may be required for running API commands and collecting metrics from the NameNode. When monitoring such Hadoop installations, specify the name of the simple authentication user here. If no such user is available/required, then do not disturb the default value none of this parameter.

  • The eG agent collects metrics using Hadoop's WebHDFS REST API. While some of these API calls pull metrics from the NameNode, some others get metrics from the resource manager. The YARN Resource Manager Service (RM) is the central controlling authority for resource management and makes resource allocation decisions.

    To pull metrics from the resource manager, the eG agents first needs to connect to the resource manager. For this, you need to configure this test with the IP address/host name of the resource manager and its web port. Use the Resource Manager IP and Resource Manager Web Port parameters to configure these details.

    To determine the IP/host name and web port of the resource manager, do the following:

    • Open the yarn-site.xml file in the /opt/mapr/hadoop/hadoop- 2.x.x/etc/hadoop directory.

    • Look for the yarn.resourcemanager.webapp.address parameter in the file.

    • This parameter is configured with the IP address/host name and web port of the resource manager. The format of this configuration is: <IP_Address_or_Host_Name>:<Port_Number>. Given below is a sample configuration:

      • 192.168.10.100:8080

    Configure the <IP_ Address_ or_ Host_ Name> in the specification as the Resource Manager IP, and the <Port_Number> as the Resource Manager Web Port. In the case of the above sample configuration, this will be 8080.

  • The eG agent collects metrics using Hadoop's WebHDFS REST API. While some of these API calls pull metrics from the NameNode, some others get metrics from the resource manager. The YARN Resource Manager Service (RM) is the central controlling authority for resource management and makes resource allocation decisions.

    In some Hadoop configurations, a simple authentication User name may be required for running API commands and collecting metrics from the Resource Manager. When monitoring such Hadoop installations, specify the name of the simple authentication user here. If no such user is available/required, then do not disturb the default value none of this parameter.

  • Once the above values are provided, click on the UPDATE button to register the changes made.

When changing default configurations of tests, the values with “$” indicate variables that will be replaced by the eG system according to the specific server being managed - for instance, $hostName is the host/nickname of the target host, $port is the port number of the server being monitored. E.g., for a server xyz:80, $hostName will be changed automatically by the eG manager to “xyz*” and $port will be changed to “80” when configuring a test.