eG Help

Files in Hadoop's file system are broken into block-sized chunks called data blocks. Hadoop stores each file as a sequence of blocks. The blocks of a file are replicated across Data Nodes for fault tolerance.

The NameNode makes all decisions regarding replication of blocks. The NameNode periodically receives a Blockreport from each of the DataNodes in the cluster. A Blockreport contains a list of all blocks on a DataNode. The NameNode constantly tracks which blocks need to be replicated and initiates replication whenever necessary.

The timely and successful replication of blocks is important for minimizing data loss at the time of disaster recovery. It is hence the responsibility of the administrator to watch over replication operations and proactively detect abnormalities (if any) in it.

Other common causes for data loss are corrupted blocks and missing blocks. An administrator needs to be able to spot such anomalies rapidly, isolate their cause, and resolve them quickly, so that Hadoop is able to deliver on its promise of 'reliable storage'.

Towards this end, administrators can take the help of the HdpBlockStatTest test. This test monitors data blocks in the Hadoop file system and turns administrator attention to corrupt and missing blocks. This way, the test urges administrators to find the reasons for such problems and the resolution for the same. The test also monitors replication operations and alerts administrators to deviations in the replication process, so they can see if the replication policy can be tweaked to remove the deviations and improve storage reliability.

Measurement	Description	Measurement Unit	Interpretation
Num_corrupt_blocks	Indicates the current number of blocks that HDFS reports as corrupted.	Number	Ideally, the value of these measures should be 0. A non-zero value for the Corrupt blocks measure indicates that one/more blocks are with corrupt replicas. A block is “with corrupt replicas” in HDFS if it has at least one corrupt replica along with at least one live replica. As such, a block having corrupt replicas does not indicate unavailable data, but they do indicate an increased chance that data may become unavailable. A non- zero value for the Missing blocks measure indicates that one/more blocks are missing. If none of a block’s replicas are live, the block is called a missing block by HDFS. Here are lists of potential causes and actions that you may take to handle the missing or corrupted blocks: HDFS automatically fixes corrupt blocks in the background. A failure of this may indicate a problem with the underlying storage or filesystem of a DataNode. Use the HDFS fsck command to identify which files contain corrupt blocks. Delete the corrupt files and recover them from backup, if it exists. Some DataNodes are down and the replicas that are missing blocks are only on those DataNodes. Bring up the failed DataNodes with missing or corrupt blocks to resolve this issue. The corrupt/missing blocks are from files with a replication factor of 1. New replicas cannot be created because the only replica of the block is missing. You may want to increase the replication factor of critical data to 3 to address this issue.
Num_missing_blocks	Indicates the current number of missing blocks.	Number
Num_allocated_blocks	Indicates the current number of allocated blocks in the system.	Number
Replication_sched_blocks	Indicates the current number of blocks scheduled for replications.	Number	This value varies from datanodes being online or offline, and the number of replicas being changed in the hdfs-site.xml(dfs.replication).
Under_replication_blocks	Indicates the current number of blocks that are under-replicated.	Number	UnderReplicatedBlocks are the number of blocks with insufficient replication. Hadoop’s replication factor is configurable on a per-client or per- file basis. The default replication factor is three, meaning that each block will be stored on three DataNodes. If you see a large, sudden spike in the number of under- replicated blocks, it is likely that a DataNode has died.
Delete_pending_blocks	Indicates the current number of blocks that are waiting for deletion.	Number	Datanodes that are back online after being down reduces the number of blocks waiting for deletion.
Excess_blocks	Indicates the current number of excess blocks in the cluster.	Number	Excess blocks can be caused by a NameNode losing heartbeats from one or more datanodes, thus resulting in the scheduling of extra replicas.