eG Monitoring
 

Measures reported by EsxMemoryTest

This test reports statistics pertaining to the machine memory of the VMware ESX server.

Measurement Description Measurement Unit Interpretation
Machine_memory The total amount of machine memory. MB  
Memory_consumed The amount of machine memory that is in use. MB Ideally, the value of this measure should be low. Using the detailed diagnosis of this measure, you can determine the amount of memory consumed by each VM.
Machine_free_memory The amount of machine memory that is free. MB A high value is typically desired for this measure.
Pct_free_memory The percentage of machine memory that is available for use. Percent A very low value for this measure indicates a shortage of memory resources. If more machine memory is not made available soon, then this could significantly degrade the performance of the guest OS'.
Memory_state The contention for memory. Number This measure takes one of the following values:
  • 0 - high (lot of memory available)
  • 1 - soft
  • 2 - hard
  • 3 - low (memory is overcommitted)

The higher the number, the memory state is more constrained.

Memory_granted The amount of memory that the VMkernel has allocated to all virtual machines running on the server. MB The detailed diagnosis of this measure reveals the amount of memory that has been granted to each virtual machine on the host. Using this information, you can quickly identify which VM has been granted the maximum memory.
Memory_unreserved The amount of unreserved swap space. MB/Sec  
Shared_memory The current amount of shared guest operating system memory. MB VMware ESX can share common memory pages across VMs. This includes pages from VMs running the same virtual machine OS and applications.
Memory_shared_common The amount of memory required for a single copy of shared pages in running VMs. MB  
Balloon_memory The total amount of physical memory currently reclaimed by the ESX server using the vmmemctl modules. MB The vmmectl driver that is installed on a virtual machine, emulates an increase or decrease in memory pressure on the guest operating system; this way, it forces the guest OS to placememory pages into its local swap file. This driver differs from theVMware swap file method as it forces the operating system to determinewhat memory it wishes to page. Once the memory is paged locally on the guest operating system, the free physical pages of memory may be reallocated to other guests. As the ESX hosts sees thatmemory demand has been reduced, it will instruct vmmemctl to “deflate” the balloon and reduce pressure on the guest OS to pagememory.

The maximum amount of memory that can be reclaimed from a guest may be configured by modifying the “sched.mem.maxmemctl” advanced option.

If the memory reclaimed from a guest (i.e., the value of this measure) is very low, it indicates excessive memory usage by the guest. Under such circumstances, you might want to consider allocating more memory to the guest.

Pct_balloon_memory The percentage of balloon memory. Percent
Swap_used The total amount of swap space used. MB This counter reflects the total amount of VMkernel swap usage on the host. In almost any scenario, this counter should be at or close to zero as VMkernel memory swapping is used as a last resort. Significant or consistent memory swapping indicates that ESX host memory is severely overcommitted and that performance degradation is imminent or actively occurring.
Zero_memory The amount of memory that is zeroed out. MB The "Memory Zero" amount will fluctuate as memory is over allocated. ESX will zero out the VM's memory to use with other VMs. The detailed diagnosis of this measure will reveal the "Memory Zero" amount for each VM.
Memory_reserved_capacity The amount of memory currently utilized to satisfy minimum memory values set for all VMs. MB  
Active_memory The amount of memory that is actively used. MB The detailed diagnosis of this measure reveals the active memory of each VM executing on the target host.
Kernel_memory The amount of machine memory being used by the ESX Server VMKernel. MB  
Swap_In_Rate The rate at which memory is swapped from disk into active memory. Mbps A high rate of swap ins and swap outs could be indicative of a memory contention on the host.
Swap_Out_Rate The rate at which memory is swapped from active memory to disk. Mbps
Memory_Overhead The total of all overhead metrics for powered-on virtual machines, plus the overhead of running vSphere services on the host. MB vSphere/ESXi virtual machines can incur two kinds of memory overhead:
  • The additional time to access memory within a virtual machine.
  • The extra space needed by the ESX/ESXi host for its own code and data structures, beyond the memory allocated to each virtual machine.

vSphere/ESXi memory virtualization adds little time overhead to memory accesses. Because the processor's paging hardware uses page tables (shadow page tables for software-based approach or nested page tables for hardware-assisted approach) directly, most memory accesses in the virtual machine can execute without address translation overhead.

The memory space overhead has two components:

  • A fixed, system-wide overhead for the VMkernel
  • Additional overhead for each virtual machine

Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures, such as shadow page tables. Overhead memory depends on the number of virtual CPUs and the configured memory for the guest operating system. vSphere/ESXi also provides optimizations such as memory sharing to reduce the amount of physical memory used on the underlying server. These optimizations can save more memory than is taken up by the overhead.

Over_Committed Indicates whether memory is over-committed or not.   Host memory is over-committed when the total memory space configured for powered-on VMs, is greater than the amount of total physical memory available to the host less the service console memory. However, note that it is unwise to run a virtual machine with a working set that is larger than the host memory. If this is the case, the hypervisor has to reclaim the virtual machine’s active memory through ballooning or hypervisor swapping, which will lead to potentially serious virtual machine performance degradation.

If the host memory is overcommited, then this measure will report the value Yes. If not, then this measure will report No.

The numeric values that correspond to the measure values discussed above are listed in the table below:

Numeric Value Measure Value
1 Yes
0 No

Note:

By default, this measure reports the values Yes or No only to indicate whether the host memory is overcommitted or not. The graph of this measure however, represents the host memory state using the numeric equivalents - 0 or 1.

Pct_over_commit Indicates how much percentage of memory is over-committed. Percent Host memory is over-committed when the total memory space configured for powered-on VMs, is greater than the amount of total physical memory available to the host less the service console memory.

A very high value for this measure indicates a shortage of memory resources in the host.

Usage The memory usage as a percentage of the total configured or available memory. Percent A consistent increase in this value could be indicative of a slow, but steady erosion of the host physical memory. If the trend continues, it could significantly degrade the performance of the host and the guest OS'.
Service_Console The amount of memory that is currently reserved for the service console. MB  
Machine_mem_saving The amount of memory saved due to sharing of memory by this system resource which includes kernels, VMs and console. MB The saved memory value indicated in this measure is obtained by subtracting the Memory_shared_common measure value from the Shared_memory measure value.
host_Swap_Used Indicates the space used for caching swapped pages in the host cache. MB Datastores that are created on solid state drives (SSD) can be used to allocate space for host cache. The host reserves a certain amount of space for swapping to host cache.

The host cache is made up of files on a low-latency disk that ESXi uses as a write back cache for virtual machine swap files. The cache is shared by all virtual machines running on the host.

When there is severe memory pressure and the hypervisor needs to swap memory pages to disk it will swap to the host cache on the SSD drive instead. If the value of this measure rises consistently, it indicates that memory pages are constantly been swapped to the host cache. This in turn is indicative of a serious memory crunch on the hypervisor. You may want to throw more memory on your hypervisor to avoid this.

Also, If the value of this measure becomes equal to the space allocated to the host cache, it means that the swapped memory pages have completely filled the host cache. Under such circumstances, these memory pages will need to be copied to the regular .vswp file. This is not a recommended practice as it will decrease performance for your VMs as these pages more than likely at some point will need to be swapped in. To avoid this therefore, you want to resize the host cache.

host_Swap_OutRate Indicates the rate at which memory is being swapped from active memory to host cache. Mbps When there is severe memory pressure and the hypervisor needs to swap memory pages to disk it will swap out to the host cache on the SSD drive instead.

If the memory pages are swapped out to the host cache at a high rate - i.e., if the value of this measure is consistently high - check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host requires more memory resources.

host_Swap_InRate Indicates the rate at which memory is being swapped from host cache into active memory. Mbps Ideally, the value of this measure should be high.
latency Indicates the percentage of time the virtual machines on the host were blocked waiting to access swapped, compressed memory, or ballooned memory. Percent The higher the value of this measure, the more adverse will be the impact on VM performance.  
active_Write Indicates the amount of memory actively being written to by the virtual machines on the host. MB  
low_Free_Thresh Indicates the threshold of free host physical memory below which vSphere will begin reclaiming memory from virtual machines through ballooning and swapping. MB If the value of the Free physical memory measure has fallen below the value of this measure, it is a clear indicator of a memory contention on the host.
compresion_Rate Indicates the rate of memory compression for the virtual machines on the host. Mbps If there is a danger of host level swapping, then ESXi will use memory compression to reduce the number of pages that it needs to swap out. If the value of this measure is high, it could be indicative of a memory contention on the host.
com_Pressed Indicates the amount of memory compressed. MB Higher the value of this measure, more will be the capacity of the host.
de_Compression Indicates the rate of memory decompression for the virtual machines on the host. Mbps  
Over_commitment Indicates whether amount of memory that is actively used by VMs is higher than machine memory available.   If the machine memory is overcommited, then this measure will report the value Yes. If not, then this measure will report No. The numeric values that correspond to the measure values discussed above are listed in the table below:

Numeric Value Measure Value
1 Yes
0 No

Note:

By default, this measure reports the values Yes or No only to indicate whether the machine memory is overcommitted or not. The graph of this measure however, represents the machine memory state using the numeric equivalents - 0 or 1.

host_SwapIn Indicates the sum of swapin values for all powered-on virtual machines on the host. MB  
host_SwapOut Indicates the sum of swapout values for all powered-on virtual machines on the host. MB A high value for this measure is indicative of a severe memory contention on the host.
Heap Indicates the VMkernel virtual address space dedicated to VMkernel main heap and related data. MB The main consumer of VMFS heap are the pointer blocks which are used to address file blocks in very large files/VMDKs on a VMFS filesystem. Therefore, the larger your VMDKs, the more VMFS heap you can consume.

As a rule of thumb, a single ESXi host should have enough default heap space to address around 10TB of open files/VMDKs on a VMFS-5 volume.

HeapFree Indicates the free address space in the VMkernel main heap. MB The value of this measure varies based on number of physical devices and configuration options.

A high value is desired for this measure. If there is no free heap left, then you will be unable to perform any VM operations (power-on, power-on, VMotion) and may also be denied access to your VMs. To avoid this, it would be good practice to allocate adequate heap.

VmfsMissRatio Indicates the trailing average of the ratio of capacity misses to compulsory misses for the VMFS PB Cache. Percent Typically, if a block in memory is accessed for the first time, the block is brought into the cache. This is called a compulsory miss. If a cache miss occurs because the block requested was discarded from cache owing to lack of memory space, then such a miss is called a capacity miss. Ideally, the value of this measure should be 0. A high value is indicative of frequent capacity cache misses. To avoid this, make sure that your cache is sized adequately.
VmfsOverHead Indicates the amount of VMFS heap used by the VMFS PB Cache. MB  
VmfsSize Indicates the space used for holding VMFS Pointer Blocks in memory. MB If the value of the VMFS PB cache capacity miss ratio measure is very high, you may want to compare the value of this measure with that of the Maximum VMFS PB cache size measure to determine whether/not the cache is getting filled up rapidly. If so, then you may want to increase the maximum cache size, so that the cache can grow freely without having to evict any pointer block entries.
VmfsMaxSize Indicates the maximum size the VMFS pointer block cache can grow to. MB You can configure the minimum and maximum sizes of the pointer block cache on each ESXi host. When the size of the pointer block cache approaches the configured maximum size, an eviction mechanism removes some pointer block entries from the cache.

Base the maximum size of the pointer block cache on the working size of all open virtual disk files that reside on VMFS datastores. All VMFS datastores on the host use a single pointer block cache.