eG Monitoring
 

Measures reported by RedisClientTest

Redis accepts clients connections on the configured listening TCP port and on the Unix socket if enabled. When a new client connection is accepted the following operations are performed:

  • The client socket is put in non-blocking state since Redis uses multiplexing and non-blocking I/O

  • The TCP_NODELAY option is set in order to ensure that there are no delays in connections;

  • A readable file event is created so that Redis is able to collect the client queries as soon as new data is available to be read on the socket.

Sometimes, by issuing a blocking command, users may block client connections. Usually the reason to block is that we expect some external event to happen: this can be some change in the Redis data structures like in the BLPOP case, a long computation happening in a thread, to receive some data from the network, and so forth. If many clients are unable to connect to the server, administrators must be able to determine if it is because of blocking commands.

Also, Redis allocates an input buffer for each client, and its role is to temporarily save the commands sent by the client. Redis requires that the size of this input buffer does not exceed 1GB, as otherwise,the client may be shutdown. Administrators therefore must be proactively alerted if any client has an input buffer that is of a size greater than 1GB.

Moreover, Redis needs to handle a variable-length output buffer for every client, since a command can produce a big amount of data that needs to be transferred to the client. If this buffer is not correctly sized, then it can fill up quickly, causing subsequent responses to clients to be enqueued in an output list. If this output list is very long for any client, it is a sign that the output buffer needs resizing. Administrators must monitor the length of the output list of clients, so that they can be instantly alerted if any client's output list is abnormally long.

Using the RedisClientTest, administrators can do all of the above! This test monitors client connections to the Redis server and captures blocked connections. This way, administrators can confirm if connections being blocked (probably by blocking commands) is the reason why clients are unable to the connect to the server. The test also monitors the size of the input buffers of clients and the output list of clients and reports the size of the largest buffer and the length of longest output list. In the process, the test reveals whether/not input and output buffers are properly sized, thereby prompting administrators to fine-tune them.

Outputs of the test : One set of results for the target Redis server

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
Connected_clients Indicates the total number of client connections to the target server (excluding connections from slaves). Number Use the detailed diagnosis of this measure to know which clients are connected to the server.
Blocked_clients Indicates the number of clients pending on a blocking call (BLPOP, BRPOP, BRPOPLPUSH). Number A low value is desired for this measure.
Clients_biggest_input_buf Indicates the largest input buffer across all clients. MB If the value of this measure is 1GB, you may want to know which client has an input buffer this big. For this purpose, sort the detailed diagnosis of the Connected_clients measure in the descending order of the values of the Input buffer size column. Then, you may want to clear this input buffer to reduce its size, as otherwise, that client may be shut down.
Clients_longest_output_list Indicates the longest output list across all clients. Number If the value of this measure is very high, it indicates that some client has a very long output list. This in turn could indicate that the output buffer is overflowing. You may want to resize the output buffer to avoid this.

To identify the client with the longest output list, sort the detailed diagnosis of theConnected_clients measure in the descending order of the values of the Output list column.