eG Monitoring
 

Measures reported by CassCommitTest

Cassandra processes data at several stages on the write path, starting with the immediate logging of a write and ending in with a write of data to disk:

  • Logging data in the commit log
  • Writing data to the memtable
  • Flushing data from the memtable
  • Storing data on disk in SSTables

When a write occurs, Cassandra database node stores the data in a memory structure called memtable, and to provide configurable durability, it also appends writes to the commit log on disk. Memtables and SSTables are maintained per table. The commit log is shared among tables. The commit log receives every write made to a Cassandra node, and these durable writes survive permanently even if power fails on a node.

The memtable is a write-back cache of data partitions that Cassandra looks up by key. The memtable stores writes in sorted order until reaching a configurable limit, and then is flushed. To flush the data, Cassandra writes the data to disk, in the memtable-sorted order.

When the memtable content exceeds the configurable threshold or the commit log space exceeds the commitlog_total_space_in_mb, the memtable is put in a queue that is flushed to disk. If the data to be flushed exceeds the memtable_cleanup_threshold, Cassandra blocks writes until the next flush succeeds. To reduce the commit log replay time, the recommended best practice is to flush the memtable before you restart the nodes. If a node stops working, replaying the commit log restores to the memtable the writes that were there before it stopped. Data in the commit log is purged after its corresponding data in the memtable is flushed to an SSTable on disk.

If the size of the commit log is inadequately configured against the commitlog_total_space_in_mb parameter, then, the data should be frequently flushed which in turn may increase the disk I/O manifold. This may cause performance bottlenecks on the target database node. Sometimes the growth of the commit log may also increase steeply. This may be primarily due to the write load on the node exceeding the ability to keep up with flushing the memtables. In the process, the older commit logs may not be deleted on time to accommodate additional data. To address these problems, it is necessary to monitor the commit logs of the target Cassandra Database node periodically. The CassCommitTest test helps administrators in this regard!

This test monitors the commit log on the target database node and reports the current size and the growth rate of the commit log. In addition, this test reports how well the messages were written to the commit log and the rate at which messages were pending to be written to the disk from the commit log. The time spent to sync the data from the commit log to the disk is also monitored and reported. Using this test, administrators may be able to plan the capacity of the commit log based on the amount of data written to the commit log.

Ouputs of the test: One set of results for the target Cassandra Database node that is being monitored.

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
Completed Indicates the rate at which the messages were written to the commit log since the start/restart of the database node. Commits/sec A high value is desired for this measure.
Growth_rate Indicates the amount of data written to the commit log per second during the last measurement period. MB/sec A sudden/gradual increase in the value of this measure is a cause of concern. Sometimes the commit log can grow in an "out of control" fashion. Essentially, this can happen because the write load on the node exceeds Cassandra's ability to keep up with flushing the memtables (and thus, removing old commit log files). If administrators find a node with dozens of commit log files, and the number seems to keep growing, this might be a serious issue which needs to be addressed immediately.
Log_size Indicates the current size of the commit log. MB Ensure that the size of the commit log is adequately configured against the commitlog_total_space_in_mb parameter.
Pending Indicates the rate at which commit log messages were written but are yet to be fsynced. Commits/sec A low value is desired for this measure.
Wait_time_on_commit Indicates the time spent for the data to be fsynced in the commit log. Seconds  
Wait_time_seg_alloc Indicates the time spent waiting for a segment to be allocated on the commit log. Seconds Ideally, the value of this measure should be zero.