eG Monitoring
 

Measures reported by KuberDaemonsTest

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Some typical uses of a DaemonSet are:

  • running a cluster storage daemon, such as glusterd, ceph, on each node.

  • running a logs collection daemon on every node, such as fluentd or logstash.

  • running a node monitoring daemon on every node

Daemon pods are typically scheduled using one of the following:

  • Daemonset Controller: Normally, the machine that a Pod runs on is selected by the Kubernetes scheduler. However, Pods created by the DaemonSet controller have the machine already selected .

  • Default scheduler: You can also schedule DaemonSets using the default scheduler instead of the DaemonSet controller, by adding the NodeAffinity term to the DaemonSet pods, instead of the .spec.nodeName term. The default scheduler is then used to bind the pod to the target host. Regardless of which scheduler (Daemonset Controller or default scheduler) schedules Daemon Pods, taints and tolerations are used to ensure that Daemon pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.

Sometimes, a Daemon Pod may be ‘ misscheduled ’ on to a node where it does not belong. In other words, a Daemon Pod could be scheduled on to a node without “matching taints”. This can cause certain cluster operations to run on nodes they should not run on, hampering cluster performance in the process. At some other times, a Daemon Pod may not run on the desired set of nodes. For instance, an anti-virus daemon, which should typically run on all nodes in a cluster/namespace, may run only on a few nodes. This is also detrimental to cluster performance. To ensure peak cluster performance, administrators should rapidly identify misscheduled DaemonSets and those that are not running on the desired nodes, and figure out what could have triggerred these anomalies. This is where the DaemonSet by Namespace test helps!

This test auto-discovers the DaemonSets in each namespace, and for each DaemonSet, reports the count of nodes scheduled to run that DaemonSet, the count of nodes on which it should run, and the count of nodes on which it should not. This way, the test promptly alerts administrators to incorrect scheduling of DaemonSets. Detailed diagnostics reveal which Daemon Pods are running on which node, thereby enabling administrators to quickly identify those nodes running Daemon Pods they should not be running. Additionally, the test also alerts administrators if a DaemonSet is updated.

Outputs of the test:One set of results for each DaemonSet in every namespace configured in the Kubernetes cluster being monitored

First-level Descriptor:Namespace

Second-level Descriptor:DaemonSet

The measures made by this test are as follows:

Measurement Description Measurement Unit Interpretation
Daemons_age Indicates how old this DaemonSet is.   The value of this measure is expressed in number of days, hours, and minutes.

You can use the detailed diagnosis of this measure to know the labels and images used by the daemons run by the DaemonSet.

Current_count Indicates the number of nodes (in this namespace) that are currently running this DaemonSet and are supposed to run this DaemonSet. Number Use the detailed diagnosis of this measure to know which Daemon Pods are running on which nodes in the namespace.
Misscheduled_count Indicates the number of nodes in this namespace, that are running this DaemonSet, but are not supposed to run this DaemonSet. Number Ideally, the value of this measure should be 0.
Desired_count Indicates the number of nodes (in this namespace) that should be running this DaemonSet. Number The value of this measure also includes the count of nodes that are already running the DaemonSet.

Ideally therefore, this value of this measure should be the same as the value of the Current_count measure. Any mismatch implies issues in scheduling, which in turn may impact cluster performance.
Ready_count Indicates the number of nodes (in this namespace) that should be running this DaemonSet and have one or more of the Daemon Pods already running and ready. Number  
Update_count Indicates the number of nodes (in this namespace) that run the updated daemon pod spec. Number Updating a DaemonSet may involve:

  • Changing node labels: If node labels are changed, the DaemonSet will promptly add Pods to newly matching nodes and delete Pods from newly not-matching nodes.

  • Changing a Daemon Pod: You can modify the Pods that a DaemonSet creates. However, Pods do not allow all fields to be updated. Also, the DaemonSet controller will use the original template the next time a node (even with the same name) is created.

  • Deleting a DaemonSet: When deleting a DaemonSet, you can choose to leave the Daemon Pods on the nodes. In this case, if you subsequently create a new DaemonSet with the same selector, the new DaemonSet adopts the existing Pods. If any Pods need replacing the DaemonSet replaces them according to its updateStrategy.

  • Performing a rolling update on a DaemonSet: With RollingUpdate update strategy, after you update a DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods will be created automatically, in a controlled fashion.
Available_count Indicates the number of nodes (in this namespace) that should be running this DaemonSet and have one or more of the Daemon Pods running and available. Number A Daemon Pod is considered to be ‘available’ if it is ready without any of its containers crashing for at least the duration specified against spec.minReadySeconds in the DaemonSet configuration (YAML) file.
Unavailable_count Indicates the number of nodes (in this namespace) that should be running this DaemonSet, but does not have it running and available. Number A Daemon Pod is considered to be ‘unavailable’ if it is not ready without any of its containers crashing for even the minimum duration specified against spec.minReadySeconds in the DaemonSet configuration (YAML) file.

Ideally, the value of this measure should be 0.