in a Kubernetes cluster, audit logs are very useful for tracing and tracking activities and changes to different cluster resources. By enabling auditing (since it’s not enabled by default), you will be able to know who did what, and when.

In this tutorial, you will learn how to enable auditing, and configure an auditing policy, An auditing policy is a configuration file that instructs the kube-apiserver on what exactly to audit. Maybe you don’t need to audit everything.

Watch the implementation steps in the following video:

Let’s start first with the theory

The lifecycle of an “Audit Event” in Kubernetes has four stages:(“Panic” stage is reached should an error occur)

What should be recorded?

You can specify the level of logging based on the following criteria. This is defined in the audit policy configuration file.

nonedon’t log events that match this rule
metadatalog request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
Requestlog event metadata and request body but not response body. This does not apply for non-resource requests
RequestResponselog event metadata, request and response bodies. This does not apply for non-resource requests.

Audit Backends

According to Kubernetes documentation, there are two Audit Backend types:

  • Log Backend: in this type, events are recorded and stored locally in JSON format.
  • Webhook Backend: events are sent to a remote HTTP API server, like FluentD, and ElasticSearch.

In this tutorial we will be focusing on the first type, the “log backend”.

Before we proceed with the implementation steps, let’s have a look at the structure of audit policy file. The following one is copied from Kubernetes documentation. You don’t need to define everything. In our tutorial, I will be only defining it according to my needs to avoid flooding the server with unneeded events.

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

In this tutorial, I will be only auditing “Pods” to keep things simple. Now, let’s get started.

Enabling Auditing – steps

1- SSH into the master node.

2- I will now create a simple audit policy config file to audit only certain events about pods, like pod creation, and deletion. the policy file will be saved at “/kube/audit/policy.yaml”. This path is stored locally on the master node filesystem. You can save it in a different path if you’d like to, but it must be saved locally on the master node(s). I have only a single master node in my environment, I guess a path backed by NFS mount could be used in multi-master node environment to have a single central copy of the config file, but never tried that approach. The policy file will look like this sample below:

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at Request level
  - level: Request
    resources:
    - group: ""
      resources: ["pods"]
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]

3- Create a directory to store the audit log files. In this example, I’m creating it at the same path as the policy file. “/kube/audit/”, but you can set is according to your requirements.

4- Configure the kube-apiserver to load this audit policy file. edit the manifest file of the kube-apiserver pod located at “/etc/kubernetes/manifests/kube-apiserver.yaml”.

We should add multiple arguments to the main command of the container, and then mount volumes pointing to both the policy.yaml file we created, and another path for storing the filesystem of the master node.

Let’s first have a look at the available arguments that we could use:

FlagDescriptionRequired?
–audit-policy-filedefine the path of the audit policy fileYes
— audit-log-pathdefine the path where the server will be keeping the logsYes
–audit-log-maxagethe max age in days for keeping audit logsOptional
–audit-log-maxbackupthe max number of log files to be keptOptional
–audit-log-maxsizethe max size of the log file in MBOptional

4.1- add the arguments to the manifest file of kube-apiserver as following. In this example, I will be using all arguments.

4.2- In the same manifest file, let’s mount a volume pointing to the local paths in our filesystem where the policy file is located, and the path of the log files, to grant the pod of the apiserver access to these paths.

I will add a single volume, since we are having the policy file, and the audit log directory saved at the same path.

#mountPath under the Container section
 - name: audit
   mountPath: /kube/audit

And then, add it to the “Volumes” section:

 - name: audit
   hostPath:
     path: /etc/kube/audit
     type: DirectoryOrCreate

Save the file and exit.

5- restart the kube-apiserver pod. there are many methods available to achieve this, use one according to your preferences. In this example, I will move the manifest file outside the “manifests” directory, and then bring it back.

mv /etc/kubernetes/manifests/kube-apiserver.yaml .

# move it back after a couple of seconds

mv kube-apiserver.yaml /etc/kubernetes/manifests/

Give it few moments to reload, and then check if’s back online. Run any command to check, for example:

kubectl get pods

Mine came back after a minute. if it never comes back, check the logs at /var/logs/pods/<kube-apiserver pod directory>

6- lets inspect the path to check if a “logs” directory was created by the pod in our filesystem.

Examine the created log file at “/kube/audit/kube-audit.log”

7- Now, let’s make some actions, and then check against the generated audit logs file. In this example, I will create, get, and then delete a pod.

kubectl run audit-nginx --image=nginx

Make sure that it’s in a good state, then let’s delete it afterwards.

8- Check the audit file for the recorded events for that pod.

cat /kube/audit/logs/kube-audit.log | grep audit-nginx

9- it’s not easy to understand the output, so, use a JSON formatter of your choice. I will use an online JSON formatter at: http://jsonviewer.stack.hu/

Click “Format” after pasting the text. The output should look like this.

Resources: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

I hope this has been informative. Thank you,