eG Monitoring
 

Measures reported by MongoRepStatusTest

A replica set is a group of mongod instances that maintain the same data set. A replica set contains several data bearing nodes and optionally one arbiter node. Of the data bearing nodes, one and only one member is deemed the primary node, while the other nodes are deemed secondary nodes. The primary node receives all write operations. A replica set can have only one primary capable of confirming writes with { w: “majority” } write concern. The primary records all changes to its data sets in its operation log, i.e. oplog. Secondary members replicate this log and apply the operations to their data sets.

When a primary does not communicate with the other members of the set for more than 10 seconds, an eligible secondary will hold an election to elect itself the new primary. The first secondary to hold an election and receive a majority of the members' votes becomes primary. When such a switch happens, it is only natural that administrators want to be notified of it, as they may then need to troubleshoot the failure of the primary and bring it back up.

Also, for the failover to be successful, the current primary should be able to access at least the majority of members in the replica set. If not, then the primary will step down and become a secondary, rendering the replica set unable to accept any further writes. To avoid this, administrators should be able to instantly detect the inaccessibility or unavailability of any member in the replica set, and quickly restore it to normalcy. The MongoRepStatusTest test enables administrators to achieve these goals! 

Using this test, administrators can keep track of the status of each member node and be promptly alerted if that node stops running or switches to an abnormal state. Furthermore, the test keeps tabs on heartbeats received from each member node, pinpoints the member node from which heartbeats were not received for a long time, and thus proactively alerts administrators to the potential non-availability of a node. This way, administrators will be able to ensure that quorum is maintained in the replica set and the primary node is able to communicate with each member node in the replica set. The test also notifies administrators if the primary of the replica set has switched. Detailed diagnostics reveals when the switch occurred and what is the current primary.

Outputs of the test : One set of results for each member node in a replica set.

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
Is_node_running Indicates whether this node is currently running or not.   The values that this measure can report and their corresponding numeric values are listed in the table below:

Measure Value Numeric Value
Yes 1
No 0

Note:

By default, the measure reports only the Measure Values provided in the table above to indicate whether/not a node is running. In the graph of this measure however, the same is indicated using the numeric equivalents only.

If any replicated node is down, then it will not be responding to requests from any available node in the replica set.

Current_status Indicates the current status of this node.   The values that this measure can report and their corresponding numeric values are listed in the table below:

Measure Value Description Numeric Value
STARTUP Not yet an active member of any set. All members start up in this state. The mongod parses the replica set configuration document while in STARTUP. 0
PRIMARY The member in state primary is the only member that can accept write operations. Eligible to vote. 1
SECONDARY A member in state secondary is replicating the data store. Eligible to vote. 2
RECOVERING Members either perform startup self-checks, or transition from completing a rollback or resync. Eligible to vote. 3
STARTUP2 The member has joined the set and is running an initial sync. 5
UNKNOWN The member's state, as seen from another member of the set, is not yet known. 6
ARBITER Arbiters do not replicate data and exist solely to participate in elections.

An arbiter does not have a copy of data set and cannot become a primary. Replica sets may have arbiters to add a vote in elections for primary. Arbiters always have exactly 1 election vote, and thus allow replica sets to have an uneven number of voting members without the overhead of an additional member that replicates data.		 7
DOWN The member, as seen from another member of the set, is unreachable. 8
ROLLBACK This member is actively performing a rollback. A rollback reverts write operations on a former primary when the member rejoins its replica set after a failover. 9
REMOVE TThis member was once in a replica set but was subsequently removed. 10

Note:

By default, the measure reports only the Measure Values provided in the table above to indicate whether/not a node is running. In the graph of this measure however, the same is indicated using the numeric equivalents only.

Is_primary_switched Indicates whether/not a failover occurred from primary to secondary.   The values that this measure can report and their corresponding numeric values are listed in the table below:

Measure Value Numeric Value
Yes 1
No 0

Note:

By default, the measure reports only the Measure Values provided in the table above to indicate whether/not a failover has occurred. In the graph of this measure however, the same is indicated using the numeric equivalents only.

The detailed diagnosis of this measure reveals the current primary, the previous primary, and when the failover occurred. With the help of these details, administrators will be able to rapidly identify which primary failed, and which secondary node has now been elected as the primary.

Is_primary Indicates whether/not this node is the primary node of the replica set.   The values that this measure can report and their corresponding numeric values are listed in the table below:

Measure Value Numeric Value
Yes 1
No 0

Note:

By default, the measure reports only the Measure Values provided in the table above to indicate whether/not a node is the primary node. In the graph of this measure however, the same is indicated using the numeric equivalents only.
Uptime Indicates the total uptime of this member node. Minutes A very low uptime could imply that the member node restarted recently.
Last_heartbeat_time Indicates the time that has elapsed since the last heartbeat was received from this node. Secs If the value of this measure increases consistently, it could imply a prolonged non-availability of the member node.