in a Kubernetes cluster, audit logs are very useful for tracing and tracking activities and changes to different cluster resources. By enabling auditing (since it’s not enabled by default), you will be able to know who did what, and when.
In this tutorial, you will learn how to enable auditing, and configure an auditing policy, An auditing policy is a configuration file that instructs the kube-apiserver on what exactly to audit. Maybe you don’t need to audit everything.
Watch the implementation steps in the following video:
Let’s start first with the theory
The lifecycle of an “Audit Event” in Kubernetes has four stages:(“Panic” stage is reached should an error occur)
What should be recorded?
You can specify the level of logging based on the following criteria. This is defined in the audit policy configuration file.
|none||don’t log events that match this rule|
|metadata||log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.|
|Request||log event metadata and request body but not response body. This does not apply for non-resource requests|
|RequestResponse||log event metadata, request and response bodies. This does not apply for non-resource requests.|
According to Kubernetes documentation, there are two Audit Backend types:
- Log Backend: in this type, events are recorded and stored locally in JSON format.
- Webhook Backend: events are sent to a remote HTTP API server, like FluentD, and ElasticSearch.
In this tutorial we will be focusing on the first type, the “log backend”.
Before we proceed with the implementation steps, let’s have a look at the structure of audit policy file. The following one is copied from Kubernetes documentation. You don’t need to define everything. In our tutorial, I will be only defining it according to my needs to avoid flooding the server with unneeded events.
apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: # Log pod changes at RequestResponse level - level: RequestResponse resources: - group: "" # Resource "pods" doesn't match requests to any subresource of pods, # which is consistent with the RBAC policy. resources: ["pods"] # Log "pods/log", "pods/status" at Metadata level - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] # Don't log requests to a configmap called "controller-leader" - level: None resources: - group: "" resources: ["configmaps"] resourceNames: ["controller-leader"] # Don't log watch requests by the "system:kube-proxy" on endpoints or services - level: None users: ["system:kube-proxy"] verbs: ["watch"] resources: - group: "" # core API group resources: ["endpoints", "services"] # Don't log authenticated requests to certain non-resource URL paths. - level: None userGroups: ["system:authenticated"] nonResourceURLs: - "/api*" # Wildcard matching. - "/version" # Log the request body of configmap changes in kube-system. - level: Request resources: - group: "" # core API group resources: ["configmaps"] # This rule only applies to resources in the "kube-system" namespace. # The empty string "" can be used to select non-namespaced resources. namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets", "configmaps"] # Log all other resources in core and extensions at the Request level. - level: Request resources: - group: "" # core API group - group: "extensions" # Version of group should NOT be included. # A catch-all rule to log all other requests at the Metadata level. - level: Metadata # Long-running requests like watches that fall under this rule will not # generate an audit event in RequestReceived. omitStages: - "RequestReceived"
In this tutorial, I will be only auditing “Pods” to keep things simple. Now, let’s get started.
Enabling Auditing – steps
1- SSH into the master node.
2- I will now create a simple audit policy config file to audit only certain events about pods, like pod creation, and deletion. the policy file will be saved at “/kube/audit/policy.yaml”. This path is stored locally on the master node filesystem. You can save it in a different path if you’d like to, but it must be saved locally on the master node(s). I have only a single master node in my environment, I guess a path backed by NFS mount could be used in multi-master node environment to have a single central copy of the config file, but never tried that approach. The policy file will look like this sample below:
apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: # Log pod changes at Request level - level: Request resources: - group: "" resources: ["pods"] # Log pod changes at RequestResponse level - level: RequestResponse resources: - group: "" resources: ["pods"]
3- Create a directory to store the audit log files. In this example, I’m creating it at the same path as the policy file. “/kube/audit/”, but you can set is according to your requirements.
4- Configure the kube-apiserver to load this audit policy file. edit the manifest file of the kube-apiserver pod located at “/etc/kubernetes/manifests/kube-apiserver.yaml”.
We should add multiple arguments to the main command of the container, and then mount volumes pointing to both the policy.yaml file we created, and another path for storing the filesystem of the master node.
Let’s first have a look at the available arguments that we could use:
|–audit-policy-file||define the path of the audit policy file||Yes|
|— audit-log-path||define the path where the server will be keeping the logs||Yes|
|–audit-log-maxage||the max age in days for keeping audit logs||Optional|
|–audit-log-maxbackup||the max number of log files to be kept||Optional|
|–audit-log-maxsize||the max size of the log file in MB||Optional|
4.1- add the arguments to the manifest file of kube-apiserver as following. In this example, I will be using all arguments.
4.2- In the same manifest file, let’s mount a volume pointing to the local paths in our filesystem where the policy file is located, and the path of the log files, to grant the pod of the apiserver access to these paths.
I will add a single volume, since we are having the policy file, and the audit log directory saved at the same path.
#mountPath under the Container section - name: audit mountPath: /kube/audit
And then, add it to the “Volumes” section:
- name: audit hostPath: path: /etc/kube/audit type: DirectoryOrCreate
Save the file and exit.
5- restart the kube-apiserver pod. there are many methods available to achieve this, use one according to your preferences. In this example, I will move the manifest file outside the “manifests” directory, and then bring it back.
mv /etc/kubernetes/manifests/kube-apiserver.yaml . # move it back after a couple of seconds mv kube-apiserver.yaml /etc/kubernetes/manifests/
Give it few moments to reload, and then check if’s back online. Run any command to check, for example:
kubectl get pods
Mine came back after a minute. if it never comes back, check the logs at /var/logs/pods/<kube-apiserver pod directory>
6- lets inspect the path to check if a “logs” directory was created by the pod in our filesystem.
Examine the created log file at “/kube/audit/kube-audit.log”
7- Now, let’s make some actions, and then check against the generated audit logs file. In this example, I will create, get, and then delete a pod.
kubectl run audit-nginx --image=nginx
Make sure that it’s in a good state, then let’s delete it afterwards.
8- Check the audit file for the recorded events for that pod.
cat /kube/audit/logs/kube-audit.log | grep audit-nginx
9- it’s not easy to understand the output, so, use a JSON formatter of your choice. I will use an online JSON formatter at: http://jsonviewer.stack.hu/
Click “Format” after pasting the text. The output should look like this.
I hope this has been informative. Thank you,