eG Monitoring
 

Measures reported by XenPoolStatusTest

A Resource Pool comprises of multiple XenServer Host installations, bound together into a single managed entity which can host Virtual Machines. When combined with shared storage, a Resource Pool enables VMs to be started on any XenServer Host which has sufficient memory and then dynamically moved between XenServer Hosts while running with minimal downtime (XenMotion).

A pool always has at least one physical host, known as the “pool master”, that provides a single point of contact for all of the servers in the pool, known as “slaves”, managing communication to other members of the pool as necessary. If the pool master is shut down or unavailable, you will not able to connect to the pool until the master is online again or until you nominate one of the other members as the new pool master for the pool. However, if a pool is High Availability-enabled, then, upon the failure of the master, another host in the pool is automatically selected as the master. VMs in the pool then automatically restart on the new master.

Likewise, you can also enable the Workload balancing component on a pool. Workload Balancing is a XenServer component, packaged as a virtual appliance, that:

  • Creates reports about VM performance in your XenServer environment
  • Evaluates resource utilization and locates virtual machines on the best possible hosts in the pool for their workload's needs

Using this test, you can determine whether/not the XenServer being monitored is the pool master, and if so, understand the composition of the pool and know the status of the hosts in the pool. In addition, for the pool master, this test reports whether/not the HA and Workload balancing features are enabled for the pool.

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
Is_pool_master Indicates whether/not the monitored XenServer is the master in this pool.   If the monitored XenServer is the pool master, then this measure will report the value Yes. If not, then, this measure will report the value No.

The numeric values that correspond to the above-mentioned measure values are as follows:

Measure Value Numeric Value
Yes 1
No 0

Note:

By default, this test reports the Measure Values listed in the table above to indicate whether/not a server is the pool master. In the graph of this measure however, the same will be represented using the numeric equivalents.
Is_ha_enabled Indicates whether this pool is high-availability (HA) enabled or not?   This measure is reported only if the XenServer being monitored is the pool master - i.e., only if the ‘Is_pool_master’ measure reports the value ‘Yes’.

If the pool for which the target XenServer is the master is HA-enabled, then this measure will report the value Yes. If not, then, this measure will report the value No.

The numeric values that correspond to the above-mentioned measure values are as follows:

Measure Value Numeric Value
Yes 1
No 0

Note:

By default, this test reports the Measure Values listed in the table above to indicate whether/not the pool is HA-enabled. In the graph of this measure however, the same will be represented using the numeric equivalents.

Ha_host_failure Indicates the number of failures that this host can tolerate before the pool is declared to be overcommitted. Number This measure is reported only if the XenServer being monitored is the pool master - i.e., only if the ‘Is_pool_master’ measure reports the value ‘Yes’.

High Availability works by creating a failure plan (that is, by calculating how many hosts can be restarted based on the priorities you set). The number of hosts that can be restarted is based on the available resources (CPU, memory) in the pool. As you specify the restart priority for VMs, XenServer evaluates the resources required to start each VM. When there are not enough resources to restart all the VMs set to be restarted, the pool reaches its Maximum failure capacity and is considered overcommitted. The pool can also be overcommitted for reasons such as not enough free memory or changes to virtual disks and networks that affect which VMs can be restarted on which servers.

To increase the maximum failure capacity for a pool, you need to do one or more of the following:

  • Reduce the number of VMs set to Restart as their restart priority.
  • Increase the amount of RAM on your servers or add more servers to the pool to increase its capacity.
  • Reduce the amount of memory configured on some VMs.
  • Shut down non-essential VMs.
Is_wlb_enabled Indicates whether/not this pool is enabled for workload balancing.   This measure is reported only if the XenServer being monitored is the pool master - i.e., only if the ‘Is_pool_master’ measure reports the value ‘Yes’.

If the pool for which the target XenServer is the master is workload balancing-enabled, then this measure will report the value Yes. If not, then, this measure will report the value No.

The numeric values that correspond to the above-mentioned measure values are as follows:

Measure Value Numeric Value
Yes 1
No 0

Note:

By default, this test reports the Measure Values listed in the table above to indicate whether/not the pool is workload balancing-enabled. In the graph of this measure however, the same will be represented using the numeric equivalents.
Total_hosts Indicates the number of XenServer hosts in this pool. Number This measure is reported only if the XenServer being monitored is the pool master - i.e., only if the ‘Is_pool_master’ measure reports the value ‘Yes’.
Online_hosts Indicates the number of XenServer hosts in this pool that are currently online. Number This measure is reported only if the XenServer being monitored is the pool master - i.e., only if the ‘Is_pool_master’ measure reports the value ‘Yes’.
Offline_hosts Indicates the number of XenServer hosts in this pool that are currently offline. Number This measure is reported only if the XenServer being monitored is the pool master - i.e., only if the ‘Is_pool_master’ measure reports the value ‘Yes’.

If the value of this measure is equal to the value of the Total_hosts measure, it indicates that none of the hosts in the pool are currently available. In this situation, users will neither be able to access the pool or its VMs.

If the pool master of a pool that is not HA-enabled goes offline, the slaves realize that communication has been lost and each retry for sixty seconds. Each slave then puts itself into emergency mode, whereby the slave hosts will now only accept the pool emergency commands. If the master comes back up at this point, it will reestablish communication with its slaves, they will leave emergency mode, and operation will return to normal. If the master remains offline, you should choose a slave and promote it to master. Once a slave becomes the master, you need to inform the other slaves who the new master is. Until this process is complete, you will not be able to access the pool.

Now, if the slaves in a pool that is not HA-enabled go offline, they will stop sending heartbeat messages to the master. If no heartbeat has been received for 30 seconds then the master assumes the slave is dead. To recover from this problem, you can repair the slave or instruct the master to forget about the slave node. In the case of the latter, all VMs running on the slave will be marked as ‘offline’ and can be restarted on other hosts.

In case of HA-enabled pools, if any host (be it the master or a slave) in the pool goes offline, the HA mechanism automatically moves protected VMs to a healthy host. Additionally, if the host that fails is the master, HA selects another host to take over the master role automatically, meaning that you can continue to manage the XenServer pool.

Disabled_hosts Indicates the number of XenServer hosts in this pool that are currently disabled. Number This measure is reported only if the XenServer being monitored is the pool master - i.e., only if the ‘Is_pool_master’ measure reports the value ‘Yes’.

If the server in a resource pool is placed in the Maintenance mode, then all running VMs will be automatically migrated from it to another server in the same pool. If the server is the pool master, a new master will also be selected for the pool. When all running VMs have been successfully migrated off the server, the server's status is changed to and set to Disabled.