eG Monitoring
 

Measures reported by RabMQVHostTest

A Virtual host provides a way to segregate applications using the same RabbitMQ instance. Different users can have different access privileges to different vhosts, and queues and exchanges can be created so they only exist in one vhost.

In fact, RabbitMQ employs a more tangible realization of virtual hosts by effectively making them “virtual clusters” on top of the broker. This means that much like RabbitMQ clusters, the workload of a virtual host is defined by the messages it receives from publishers and delivers to consumers and the message read-write activity on its disk. Understanding the current workload of a virtual host and measuring how well the virtual host handles the workload is important to figure out if the virtual host has been sized and configured right to perform well at peak load. The RabMQVHostTest test enables administrators to gain this understanding.

This test tracks the messages published to and delivered by each of the virtual hosts the configured user (i.e., the user configured for this test)has access to. Furthermore, the test measures the rate at which each virtual host processes the messages, and thus reveals bottlenecks (if any) in message processing. Additionally, the test also reports the level of I/O activity on the cluster disks, and reveals the count of unacknowledged and redelivered messages. This will enable administrators to assess the impact of IOPS and messages on the performance of the virtual host. This way, the test helps administrators gauge the current workload on each virtual host, understand the present processing power of every virtual host, and identify those virtual hosts that could be experiencing processing bottlenecks. Pointers provided by the test can also be used to figure out how the virtual host configuration can be fine-tuned to ensure optimal performance.

Outputs of the test : One set of results for the each virtual host to which the configured user has access.

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
currReady Indicates the number of messages on this virtual host that are available to be delivered now. Number  
currUnacknowledged Indicates the number of messages for which this virtual host is waiting for acknowledgement. Number A low value is desired for this measure. This is because, all unacknowledged messages have to reside in RAM on the servers. If you have too many unacknowledged messages you will run out of memory. An efficient way to limit unacknowledged messages is to limit how many messages your clients prefetch.
currTotal Indicates the total number of messages on this virtual host currently. Number This is the sum total of the value of the currReady and currUnacknowledged measures.

The value of this measure is a good indicator of the current message load on the virtual host.
diffPublish Indicates the rate at which publishers are publishing messages on this virtual host. Messages/Sec  
diffConfirm Indicates the rate at which this virtual host confirms the receipt of a message to a publisher. Messages/Sec A ‘Publish Confirm’ is nothing but a acknowledgement sent to a publisher confirming the receipt of a message from that publisher. Publish Confirms have a performance impact. This means, the lower the value of this measure, the better. However, one should keep in mind that a Publish Confirm is required if the publisher needs at-least-once processing of messages.
diffDeliverNoAck Indicates the rate at which messages on this virtual host are delivered to consumers that use manual acknowledgements. Messages/Sec Messages in transit might get lost in an event of a connection failure, and such a message might need to be retransmitted. Acknowledgements let the server and clients know when to retransmit messages.

A manual acknowledgement is an ‘explicit’ acknowledgement that is received from the consumer. Manually sent acknowledgements can be positive or negative. Positive acknowledgements simply instruct RabbitMQ to record a message as delivered and can be discarded. Negative acknowledgements with basic.reject have the same effect. The difference is primarily in the semantics: positive acknowledgements assume a message was successfully processed while their negative counterpart suggests that a delivery wasn't processed but still should be deleted.

Whether positive or negative, manual acknowledgements deliver low throughput and hence, should be avoided. A low value is therefore desired for this measure.
diffDeliver Indicates the rate at which messages on this virtual host are delivered to consumers that use automatic acknowledgements. Messages/Sec  
diffConsumerAck Indicates the rate at which messages on this virtual host are being acknowledged by consumers. Messages/Sec If the diffGetNoAck measure registers an abnormally low value, then, you may want to check the value of this measure at around the same time to determine whether the a delay by consumers in acknowledging the messages was what caused the delivery delay.
diffRedeliver Indicates the rate at which messages with the ‘redelivered’ flag set are being delivered by this virtual host. Messages/Sec  
diffGetNoAck Indicates the rate at which messages requiring acknowledgement are being delivered by this virtual host in response to basic.get. Messages/Sec Compare the value of this measure with that of the diffGet measure to figure out what type of messages are being delivered much slower than the rest.
diffGet Indicates the rate at which messages not requiring acknowledgement are being delivered by this virtual host in response to basic.get. Messages/Sec Compare the value of this measure with that of the diffGetNoAck measure to figure out what type of messages are being delivered much slower than the rest.
diffReturnValue Indicates the rate at which this virtual host sent unrouteable messages with ‘mandatory’ flag set to ‘true’, to publishers. Messages/Sec An unroutable message is a message without a destination. For example, a message sent to an exchange without any bound queue.

If the ‘mandatory’ flag is set to ‘true’, then an unroutable message is returned to the producer with a ‘basic.return’ AMQP method.
diffDiskReads Indicates the rate at which queues in this virtual host read messages from disk. Messages/Sec A high value could indicate that messages are frequently read from the disk and not from the RAM. This could be owing to high memory pressure, which may have forced RabbitMQ to move messages from RAM to disk.
diffDiskWrites Indicates the rate at which queues in this virtual host wrote messages to disk. Messages/Sec A high value for this measure could indicate any of the following:

  • Many messages have been published in such a way that they must be written to disk;

  • A very high memory pressure on RabbitMQ has caused the cluster to move majority of messages from RAM to disk;

publish Indicates the total number of messages entering this virtual host. Number  
confirm Indicates the total number of messages that this virtual host is confirming to publish. Number A ‘Publish Confirm’ is nothing but a acknowledgement sent by the cluster to a publisher confirming the receipt of a message from that publisher. Publish Confirms have a performance impact. This means, the lower the value of this measure, the better. However, one should keep in mind that a Publish Confirm is required if the publisher needs at-least-once processing of messages.
deliverNoAck Indicates the total number of messages are delivered to consumers that use manual acknowledgements. Number Messages in transit might get lost in an event of a connection failure, and such a message might need to be retransmitted. Acknowledgements let the server and clients know when to retransmit messages.

A manual acknowledgement is an ‘explicit’ acknowledgement that is received from the consumer. Manually sent acknowledgements can be positive or negative. Positive acknowledgements simply instruct RabbitMQ to record a message as delivered and can be discarded. Negative acknowledgements with basic.reject have the same effect. The difference is primarily in the semantics: positive acknowledgements assume a message was successfully processed while their negative counterpart suggests that a delivery wasn't processed but still should be deleted.
deliver Indicates the total number of messages that are delivered to consumers that use automatic acknowledgements. Number In automatic acknowledgement mode, a message is considered to be successfully delivered immediately after it is sent. This mode trades off higher throughput (as long as the consumers can keep up) for reduced safety of delivery and consumer processing. This mode is often referred to as “fire-and-forget”. Unlike with manual acknowledgement model, if consumers's TCP connection or channel is closed before successful delivery, the message sent by the server will be lost. Therefore, automatic message acknowledgement should be considered unsafe and not suitable for all workloads.
consumerAck Indicates the total number of messages that are being acknowledged by consumers. Number  
redeliver Indicates the total number of messages that are being delivered with the ‘redelivered’ flag set. Number If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the ‘redelivered’ flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before.
getNoAck Indicates the total number of messages requiring acknowledgement that are being delivered in response to basic.get. Number For best performance and high throughput, the value of this measure should be low.
get Indicates the total number of messages not requiring acknowledgement that are being delivered in response to basic.get. Number  
returnValue Indicates the rate at which unrouteable messages with ‘mandatory’ flag set to ‘true’, were sent to publishers. Messages/Sec An unroutable message is a message without a destination. For example, a message sent to an exchange without any bound queue.

If the ‘mandatory’ flag is set to ‘true’, then the cluster return an unroutable message to the producer with a ‘basic.return’ AMQP method.

To know which nodes in the cluster returned the maximum number of messages to publishers, use the detailed diagnosis of this measure.
diskReads Indicates the total number of messages read from disk. Number A high value could indicate that many messages are read from the disk and not from the RAM. This could be owing to high memory pressure, which may have forced RabbitMQ to move messages from RAM to disk.

If this measure reports an abnormally high value, then use the detailed diagnosis of this measure to know which nodes in the cluster are performing the maximum reads from the disk. Such nodes could be running out of memory.
diskWrites Indicates the total number of messages written to disk. Number A high value for this measure could indicate any of the following:

  • Many messages have been published in such a way that they must be written to disk;

  • A very high memory pressure on RabbitMQ has caused the cluster to move majority of messages from RAM to disk;

If this measure reports an abnormally high value, then use the detailed diagnosis of this measure to know which nodes in the cluster are performing the maximum reads from the disk. Such nodes could be running out of memory.