eG Monitoring
 
Measures reported by VnxMoverStatTest

Data movers (also known as X-Blades) contain the defined file system. They access data from the back end and provide host access using the same UltraFlex I/O Technology that supports the NFS, CFS, MPFS, and pNFS protocols. These data movers are contained within a Data Mover Enclosure (DME). A DME can consist of upto 14 data movers. These provide network connectivity, storage connectivity and run the Data Access in Real Time (DART) operating system for high performance. Each data mover operates like an independent physical server, with its own CPU, memory, and ports. It is good practice to keep an eye on the CPU usage, RAM usage, and internal operations of each of the data movers in a DME, so that abnormal usage patterns and potential resource contentions in a data mover can be proactively detected and promptly resolved. The VnxMoverStatTest test helps in this regard.

The test monitors the CPU and RAM usage of and threads executed within each data mover in the DME, and proactively alerts administrators to the over/under-utilization of the CPU resources, the steady/rapid erosion of RAM, and operational bottlenecks such as blocked threads. This way, the test warns administrators of an impending slowdown in data mover performance, and enables them to initiate measures to prevent the anomaly well before it affects user experience with the storage system as a whole.

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
Runnable_threads Indicates the total number of threads that are currently running as well as the number of threads that are waiting for a CPU to be available in this Data Mover. Number  
Blocked_threads Indicates the number of threads that are currently blocked in this Data Mover. Number A blocked thread is the one that cannot be run because it is waiting for some specific event to occur or the threads waiting on a condition variable or a Sthread Mutex. Threads block for many reasons for e.g., they attempt to read data when no data is available, they execute a thread blocking method or they attempt to acquire a synchronization lock that another thread already holds etc.
Joined_threads Indicates the number of joined threads that are currently executing on this Data Mover. Number Sometimes, a currently executing thread would require the result of another thread for completing a process. In such case, both the threads are joined together so that the executing thread completes the process.
Available_RAM Indicates the amount of RAM that is currently available for use in this Data Mover. MB A high value is desired for this measure. A consistent decrease in this value is a cause for concern, as it indicates that the RAM usage is steadily increasing. In such situations, you can compare the value of this measure between data movers to identify which data mover has very little RAM available. You may want to consider allocating additional RAM to that data mover, so that the potential resource crisis can be avoided.
CPU_idle Indicates the percentage of time the CPU of this Data Mover was idle. Percent Comparing the value of this measure across all the Data Movers of the EMC VNX Storage system will reveal which data mover is not using its CPU effectively.
CPU_busy Indicates the percentage of time the CPU of this Data Mover was busy. Percent Comparing the value of this measure across all the Data Movers of the EMC VNX Storage system will reveal which data mover is making maximum use of its CPU resources. A value close to 100% is a cause for concern, as it indicates excessive CPU usage, and calls for allocation of more CPU resources to that data mover.