eG Monitoring
 

Measures reported by VmGuestTest

This test monitors the amount of the physical server's resources that each guest on an ESX server is taking up. Using the metrics reported by this test, administrators can determine which virtual guest is taking up most CPU, which guest is generating the most network traffic, which guest is taking up the maximum memory utilization, which guest has the maximum disk activity, etc. Note that the amount of resources taken up by a virtual guest will be limited by the resource allocations that have been made by administrators. For example, an administrator could cap the amount of memory that a specific guest may take. Also, virtual guests can be organized into resource pools, and allocation of resources can be made at the resource-pool level. In this case, virtual guests allocated to the same resource pool contend for the resources allocated to the resource pool.

Measurement Description Measurement Unit Interpretation
Current_sessions This measure is relevant only for monitoring of virtual desktops (i.e., for Vmware_vdi_servers). When reporting metrics for specific users, this metric indicates the number of sessions that each user has currently logged into; this measure will be available only if the test reports measures per currently logged in user. Number This is a good indicator of how busy the user is. The detailed diagnosis of this measure, if enabled, reveals the guests to which the user is currently logged on to.

Powered_on Whether the virtual machine is currently running on the ESX server host or not.   While the test reports a wide variety of other metrics too for virtual machines that are alive, only the Powered_on status is indicated for virtual machines that are currently not available.

If this measure reports the value Yes, it indicates that the guest is up and running. The value No could indicate that the guest has been powered-off; it could also indicate that VMware VMotionŽ has moved the guest to a different ESX server.

The numeric values that correspond to each of the powered-on states discussed above are listed in the table below:

State Value
Yes 1
No 0

Note:

By default, this measure reports the values Yes or No to indicate the status of a VM. The graph of this measure however, represents the status of a VM using the numeric equivalents - 0 or 1.

Cpu_used Indicates the percentage of physical CPU used by the guest. Percent A high value for this measure indicates a virtual machine that is using a lot of the processor - possibly because one or more processes on this VM are taking a lot of CPU. Correlation of the Cpu_used with the Ready metric will indicate if the VM is getting sufficient processor time or not. If the Cpu_used is high and Ready value is low, this is indicative of a VM that is getting enough processor time.
Cpu_usage_Mhz Indicates the Cpu usage of of the VM in Mhz Mhz The percent CPU usage measure serves as an effective indicator of how resource-intensive a particular VM is on a specific ESX server. However, for performing capacity planning or what-if analysis, the CPU usage of a VM measured in absolute terms would be more useful. For instance, the physical CPU usage of a VM could be 30% on a particular ESX server - this means that the VM is consuming 30% of the total physical CPU capacity of that ESX server. If you are planning to migrate the VM to another ESX server, then it would be unwise to assume that the VM will only consume 30% of CPU on the other ESX server as well, as the percentage will vary depending upon the physical CPU resources that are available to the other ESX server. The absolute measure however, will remain unchanged across ESX servers. Therefore, to decide which ESX server the VM is to be moved to, and to analyze the impact of this movement on the CPU resources of the new ESX host, you would require an absolute measure of CPU usage.

System_cpu Indicates the percentage of time this guest spent in the ESX server VMKernel to process interrupts and to perform other system activities. Percent An unusually high value indicates a problem and may be due to too many system-level tasks executing simultaneously.
Busy_wait Indicates the percentage of time the guest spent in the wait state - i.e., IDLE, waiting for interrupt, etc. Percent While the "Ready" metric denotes the time when the VM is waiting for CPU to execute, the busy wait state represents the time when the VM is waiting for some event to happen before it is ready to execute.
Max_limited Indicates the percentage of time the ESX Server VMKernel deliberately did not run the guest, because that would violate the limit setting for the guest. Percent A Limit refers to the maximum CPU the host can make available to the virtual guest or its resource pool.

Even though the resource pool is ready to run, if it is prevented from running owing to a probable limit violation, then the value of this measure will not be included in the value of the Ready measure.

A high value for this metric indicates that the virtual guest has been trying to get additional CPU resources but has not been able to do so because the CPU usage for this VM or its resource pool (if it is a part of a resource pool) has reached the allocated limit.
Ready Indicates the percentage of time the guest was ready to run (i.e., it had instructions to execute) but was not able to because of processor contention. Percentage This metric should typically be low - generally 5% or less. The more time a VM spends waiting to run, the more lag time there is in responsiveness within the VM.
Memory_size Indicates the amount of physical memory allocated to the guest. MB An ESX Server host determines allocations for each virtual machine based on the number of shares allocated to it and an estimate of its recent working set size.

  • Shares: ESX Server hosts use a modified proportional-share memory allocation policy. Memory shares entitle a virtual machine to a fraction of available physical memory.

  • Working set size: ESX Server hosts estimate the working set for a virtual machine by monitoring memory activity over successive periods of virtual machine execution time. Estimates are smoothed over several time periods using techniques that respond rapidly to increases in working set size and more slowly to decreases in working set size.

The maximum memory that can be allocated to a virtual machine is governed by the Limit parameter. However, the ESX host will disregard this Limit, if it is higher than the physical memory size that was specified for the virtual machine during its creation.

Target_memory Indicates the amount of physical memory the ESX Server VMKernel wants to allocate to the guest. MB
Touched_memory Indicates the working set estimate for the guest. MB
Active_memory Indicates the percentage of memory allocated to the guest that is currently in use. Percent High memory consumption over long periods can deplete the free memory on the guest, causing prolonged delays in the execution of the application(s) hosted by the guests. Comparison of the memory usage across guests indicates the guest(s) that could be causing a memory bottleneck on the host.
Swap_usage Indicates the current swap usage by the guest. MB ESX Server hosts use swapping to forcibly reclaim memory from a virtual machine when no vmmemctl driver is available because the vmmemctl driver:

  • Was never installed
  • Has been explicitly disabled
  • Is not running (for example, while the guest operating system is booting)
  • Is temporarily unable to reclaim memory quickly enough to satisfy current system demands.

Standard demand-paging techniques swap pages back in when the virtual machine needs them.

Swap space must be reserved on disk for any unreserved virtual machine memory. This swap reservation is required to ensure the system is able to preserve virtual machine memory under any circumstances. In practice, only a small fraction of the swap space may actually be used.

Typically, swap space usage for each VM should be low. Since access from RAM is much faster than access from physical disk, excessive usage of swap memory will slow down the performance of a VM. Watch for VMs that are seeing higher swap usage and more swap reads and writes.
Swap_reads Indicates the rate at which memory is being swapped in by the ESX Server from the disk for the guest. MB/Sec
Swap_writes Indicates the rate at which memory is being swapped to disk by the ESX Server. MB/Sec
Packets_transmitted Indicates the number of packets transmitted per second by the guest. Packets/Sec  
Data_transmitted Indicates the amount of data the guest transmitted per second. Mbps Comparing the data transmitted across all the virtual guests provides an indicator of the guest that is generating most out-bound network traffic.
Packets_received Indicates the number of packets received per second by the guest. Packets/Sec  
Data_received Indicates the rate at which data was received by the guest. Mbps Comparing the data received across all the virtual guests provides an indicator of the guest that has the most in-bound network traffic.
Outbound_packet_drops Indicates the percentage of transmit packets dropped by the guest. Percent Packet loss is often caused by network buffer overflows at a network router or by packet corruptions over the network.
Inbound_packet_drops Indicates the percentage of receive packets dropped by the guest. Percent
Active_disk_commands Indicates the number of commands in the ESX server VMKernel that are currently active for the guest. Number  
Queued_disk_commands Indicates the number of commands in the ESX Server VMKernel that are currently queued for the guest. Number A consistent increase in the value of this measure could imply that the VMKernel is unable to execute commands on the guest, resulting in a queue that keeps growing. You might want to investigate the reasons behind such an occurrence.
Disk_command_rate Indicates the number of commands issued per second to the storage adapters of the guest. Commands/Sec  
Disk_reads Indicates the rate at which read commands were issued to the storage adapters of the guest. Reads/Sec  
Disk_writes Indicates the rate at which write commands were issued to the storage adapters of the guest. Writes/Sec  
Balloon_mem_ status Indicates whether the memory balloon driver is installed or not. Boolean The ESX Server employs two distinct techniques for dynamically expanding or contracting the amount of memory allocated to virtual machines:
  • A memory balloon driver (vmmemctl), loaded into the guest operating system running in a virtual machine, part of the VMware Tools package
  • Paging from a virtual machine to a server swap file, without any involvement by the guest operating system
    • .

The balloon driver, also known as the vmmemctl driver, is installed on a guest operating system and communicates with the VMKernel. If the value of this measure is 0, it indicates that the vmmectl driver has not been installed on the guest. In such a case, the ESX server uses swapping to reclaim memory from the guest. The value 1 for this measure, indicates that the balloon driver is installed on the guest.
Balloon_mem_ current Indicates the total amount of physical memory currently reclaimed from this guest using the vmmemctl modules. MB The vmmectl driver that is installed on a virtual machine, emulates an increase or decrease inmemory pressure on the guest operating system; this way, it forces the guest OS to placememory pages into its local swap file. This driver differs from theVMware swap file method as it forces the operating system to determinewhat memory it wishes to page. Once the memory is pagedlocally on the guest operating system, the free physical pages of memorymay be reallocated to other guests. As the ESX hosts sees thatmemory demand has been reduced, it will instruct vmmemctl to"deflate" the balloon and reduce pressure on the guest OS to pagememory. The maximum amount of memory that can be reclaimed from a guest may be configured by modifying the "sched.mem.maxmemctl" advanced option.If the memory reclaimed from a guest (i.e., the Balloon_mem_current) is very low, it indicates excessive memory usage by the guest. Under such circumstances, you might want to consider allocating more memory to the guest. Similarly, if the value of the Balloon_mem_current measure is dangerously close to the Balloon_mem_max measure, it indicates that the guest operating system is left with very limited reclaimable memory. This, once again, is indicative of high memory pressure on the guest. A re-look at the memory allocations to the guest is hence recommended.
Balloon_mem_ target Indicates the total amount of physical memory the ESX server attempts to reclaim from this guest using the vmmemctl modules. MB
Balloon_mem_ max Indicates the maximum amount of physical memory the ESX server can reclaim from this guest using the vmmemctl modules. MB
Shared_pages Indicates the amount of physical memory pages allocated to the guest that are shared. MB Many ESX Server workloads present opportunities for sharing memory across virtualmachines. For example, several virtual machines may be running instances of the same guest operating system, have the same applications or components loaded, or contain common data. In such cases, ESX Server systems use a proprietary transparent page-sharing technique to securely eliminate redundant copies of memory pages. With memory sharing, a workload running in virtual machines often consumes less memory than it would when running on physical machines. As a result, higher levels of overcommitment can be supported efficiently.
Avg_guest_ latency Indicates the average guest operating system latency, per command. Msecs/cmd  

Note:

The VmGuestTest will report the following measures only if the procedure discussed in Section 9 of the Monitoring VMware Infrastructures is followed:

  • Swap_usage
  • Swap_reads
  • Swap_writes
  • Balloon_mem_status
  • Balloon_mem_current
  • Balloon_mem_target
  • Balloon_mem_max
  • Shared_pages
  • Avg_guest_latency