3.2.7 - Ensure that the --eventRecordQPS argument is set to 0 or a level which ensures appropriate event capture (Automated)

Views:

Profile applicability: Level 2

Security relevant information should be captured. The --eventRecordQPS flag on the Kubelet can be used to limit the rate at which events are gathered. Setting this too low could result in relevant events not being logged, however the unlimited setting of 0 could result in a denial of service on the kubelet.

It is important to capture all events and not restrict event creation. Events are an important source of security information and analytics that ensure that your environment is consistently monitored using the event data.

Note

See the AKS documentation for the default value.

Impact

Setting this parameter to 0 could result in a denial of service condition due to excessive events being created. The cluster event processing and storage systems should be scaled to handle expected event loads.

Audit

Audit Method 1

SSH to each node.
Run the following command on each node to find the Kubelet process:
```
ps -ef | grep kubelet
```
Review the value set for the --eventRecordQPS argument and determine whether this has been set to an appropriate level for the cluster. The value of 0 can be used to ensure that all events are captured.
If the --eventRecordQPS argument does not exist, check that there is a Kubelet config file specified by --config and review the value in this location. The output of the above command should return something similar to --config /etc/kubernetes/kubelet/kubelet-config.json which is the location of the Kubelet config file.

Open the Kubelet config file:

cat /etc/kubernetes/kubelet/kubelet-config.json

If there is an entry for eventRecordQPS, check that it is set to 0 or an appropriate level for the cluster.

Audit Method 2

If using the api configz endpoint, search for the status of eventRecordQPS by extracting the live configuration from the nodes running kubelet.

Set the local proxy port and the following variables and provide proxy port number and node name:

HOSTNAME_PORT="localhost-and-port-number" NODE_NAME="The- Name-Of-Node-To-Extract-Configuration"
                     from the output of "kubectl get nodes"

kubectl proxy --port=8001 &

export HOSTNAME_PORT=localhost:8001
export NODE_NAME=ip-192.168.31.226.aks.internal

curl -sSL "http://${HOSTNAME_PORT}/api/v1/nodes/${NODE_NAME}/proxy/configz"

Remediation

Remediation Method 1

If modifying the Kubelet config file, edit the kubelet-config.json file /etc/kubernetes/kubelet/kubelet-config.json and set the below parameter to 5 or a value greater or equal to 0:

"eventRecordQPS": 5

Check that /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf does not define an executable argument for eventRecordQPS because this would override your Kubelet config.

Remediation Method 2

If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf on each worker node and add the below parameter at the end of the KUBELET_ARGS variable string:

--eventRecordQPS=5

Remediation Method 3

If using the api configz endpoint, search for the status of "eventRecordQPS" by extracting the live configuration from the nodes running kubelet.

See detailed step-by-step configmap procedures in Reconfigure a Node's Kubelet in a Live Cluster, and then rerun the curl statement from audit process to check for kubelet configuration changes:

kubectl proxy --port=8001 &

export HOSTNAME_PORT=localhost:8001
export NODE_NAME=ip-192.168.31.226.aks.internal

curl -sSL "http://${HOSTNAME_PORT}/api/v1/nodes/${NODE_NAME}/proxy/configz"

For all three remediations: Based on your system, restart the kubelet service and check status:

systemctl daemon-reload
systemctl restart kubelet.service
systemctl status kubelet -l