eG Monitoring
 

Measures reported by FSIndexerTest

Indexers manage the content to be included in an index, and propagate that content to query servers where they are stored in index files.

Each item that a connector submits for indexing passes through the indexing pipeline, in which a predefined set of modules manipulates the items one by one. At the end of this, the FIXML needed for indexing is produced and handed over to the indexer by the Indexing Dispatcher. The Indexer receives the item and creates binary indexes in the data folder on disk. These binary indexes are then sent to the query matching component and are used during query evaluation during searching.

If too many feeding operations/documents are sent to an indexer for processing, that indexer may be forced to push many of these operations into the API queue, where they will be held until they are taken up for processing. Administrators need to monitor the length of this queue at frequent intervals to determine the processing power of an indexer.

Using this test, administrators can keep an eye on the number and nature of documents that each indexer processes, track the growth of the API queues, monitor the FiXML fill rate of every indexer, and thus promptly capture potential overloads, processing delays, processing failures, and other inefficiencies in the functioning of every indexer.

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
FSApiOpers Indicates the total number of feed operations processed by this indexer. Number This measure indicates the total number of items that are indexed and updated.
FSApiOpersPerSec Indicates the rate at which the feed operations are processed by this indexer. Operations/sec A consistent decrease in the value of this measure could indicate a processing bottleneck.
FSApiParUpde Indicates the number of feed operations processed by this indexer that are partial update operations. Number  
FSApiQueLod Indicates the current load on the API queue of this indexer. Bytes A low value is desired for this measure.

The API queue is used by the indexer to hold pending operations waiting to be persisted.

The value of this measure is obtained by the ratio of the current size of the API queue to the maximum size of the API queue.
FSApiQueSiz Indicates the number of batches that are currently in the API queue of this indexer. Number Indexers queue incoming work under high load. This is common, especially for partial updates. If API queues never (intermittently) reach zero, the indexer is the bottleneck. The crawler will pause when the queue reaches 256MB in one of the indexers. This can occur if the storage subsystem is not sufficiently powerful. It will also occur during large redistribution of content between partitions, which temporarily blocks more content from being indexed.
FSApiRemOpers Indicates the number of feed operations processed by this indexer that are remove opreations. Number  
FSApiStstUpdat Indicates the number of feed operations processed by this indexer that are status updates. Number  
FSCollsRemvd Indicates the number of feed operations processed by this indexer that are remove collection operations. Number  
FSDocsInInd Indicates the total number of valid items that are currently contained in the docindex of this Indexer. Number The docindex is a mapping to the persisted items on disk, so this translates to the number of valid items persisted on disk (FiXML).
FSDocsInd Indicates the number of feed operations processed by this indexer that are add/update operations. Number  
FSFaiOpers Indicates the number of feed operations that were not successfully processed by this indexer. Number  
FSFixFilRat Indicates the ratio of the number of valid items in a persisted file to the total number of persisted items. Percent FiXML files (internal item storage in the indexers) are compacted regularly, by default between 3am and 5am every night. Low FiXML fill rate (<70%) will lead to inefficient operation.
FSLstBatSiz Indicates the size of the previous batch that was processed by this indexer. Bytes  
FSMaxBatSiz Indicates the size of the largest batch that was processed by this indexer during the last measurement period. Bytes  
FSMaxDocsPerSec Indicates the maximum number of items that were indexed by this indexer in an index job per second. Indexed/Sec By closely observing variations to these measures over time, you can proactively detect potential processing latencies.
FSMinDocsPerSec Indicates the minimum number of items that were indexed by this indexer in an index job per second. Number
FSRegsFilRecvs Indicates the total number of file receivers that are currently registered on this indexer. Number The file receivers can either be backup indexers or search controllers.
FSRegsSrchConts Indicates the total number of search controllers that are currently registered on this indexer. Number The registered search controllers get new indices from over the indexer.
FSSCtlCurAloctd Indicates the total number of search controller stubs that are currently allocated by this indexer. Number The search controller stub is used to communicate with a remote search controller.

This measure is a good indicator of the number of search controllers that are communicating with the indexer.
FSSCtlTotAloctd Indicates the number of times a search controller has been allocated on this indexer. Number  
FSSizOfDocs Indicates the size of all the valid items that are persisted on the disk. Bytes