eG Monitoring
 

Measures reported by HdpNNUptimeTest

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.

Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives.

The NameNode is a Single Point of Failure for the HDFS Cluster. When the NameNode goes down, the file system goes offline.

High uptime of the NameNode is therefore imperative for the high availability of the file system. To make sure that the file system is always accessible to users, administrators need to continuously measure the uptime of the NameNode, promptly detect unscheduled reboots of the NameNode, investigate the root-cause for such reboots, and fix it before users notice. Similarly, they also need to keep an eye on changes to operational status of the NameNode and file system, so they can instantly detect if the NameNode is not active or if the file system is in the safe mode. The HdpNNUptimeTest test helps administrators achieve all of the above!

This test continuously tracks and reports the uptime of the NameNode, thus revealing whether/not a scheduled reboot occurred. Likewise, the test also alerts administrators to unexpected reboots, so they can quickly investigate and determine the reason for the same and eliminate it. This way, the test helps administrators ensure the high uptime of the NameNode. Additionally, the test periodically checks the status of the NameNode and the file system and brings abnormalities to light.

Outputs of the test : One set of the results for the target Hadoop cluster

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
Is_rebooted Indicates whether the NameNode has been rebooted during the last measurement period or not.   If this measure shows 1, it means that the NameNode was rebooted during the last measurement period. By checking the time periods when this metric changes from 0 to 1, an administrator can determine the times when the NameNode was rebooted.

The detailed diagnosis of this measure, if enabled, displays when the NameNodewas shutdown, for how long it was shutdown, when it was restarted, and whether/not the NameNode is in maintenance.
Diff_uptime Indicates the time period that the NameNode has been up since the last time this test ran. Seconds If the NameNode has not been rebooted during the last measurement period and the agent has been running continuously, this value will be equal to the measurement period. If the NameNode was rebooted during the last measurement period, this value will be less than the measurement period of the test. For example, if the measurement period is 300 secs, and if the NameNode was rebooted 120 secs back, this metric will report a value of 120 seconds. The accuracy of this metric is dependent on the measurement period - the smaller the measurement period, greater the accuracy.
Uptime Indicates the total time that the NameNode has been up since its last reboot.   This measure displays the number of years, months, days, hours, minutes and seconds since the last reboot. Administrators may wish to be alerted if a server has been running without a reboot for a very long period. Setting a threshold for this metric allows administrators to determine such conditions.
Status Indicates the current status of the NameNode.   The values that this measure reports and their corresponding numeric values are listed in the table below:

Measure Value Numeric Value
Standby or Stopping 0
Initializing 1
Active 2

Note:

By default, this measure reports the Measure Values listed in the table above to indicate the status of the NameNode. The graph of this measure however, represents the same using the numeric equivalents only.
Fs_state Indicates the current state of the filesystem.   The values that this measure reports and their corresponding numeric values are listed in the table below:

Measure Value Numeric Value
Safemode 0
Operational 1

Note:

By default, this measure reports the Measure Values listed in the table above to indicate the status of the file system. The graph of this measure however, represents the same using the numeric equivalents only.