Mutate Resources
A mutate rule can be used to modify matching resources and is written as either a RFC 6902 JSON Patch or a strategic merge patch.
By using a patch in the JSONPatch - RFC 6902 format, you can make precise changes to the resource being created. A strategic merge patch is useful for controlling merge behaviors on elements with lists. Regardless of the method, a mutate rule is used when an object needs to be modified in a given way.
Resource mutation occurs before validation, so the validation rules should not contradict the changes performed by the mutation section. To mutate existing resources in addition to those subject to AdmissionReview requests, use mutateExisting policies.
This policy sets the imagePullPolicy to IfNotPresent if the image tag is latest:
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: set-image-pull-policy
5spec:
6 rules:
7 - name: set-image-pull-policy
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 mutate:
14 patchStrategicMerge:
15 spec:
16 containers:
17 # match images which end with :latest
18 - (image): "*:latest"
19 # set the imagePullPolicy to "IfNotPresent"
20 imagePullPolicy: "IfNotPresent"
RFC 6902 JSONPatch
A JSON Patch, implemented as a mutation method called patchesJson6902, provides a precise way to mutate resources and supports the following operations (in the op field):
addreplaceremove
With Kyverno, the add and replace have the same behavior (i.e., both operations will add or replace the target element).
The patchesJson6902 method can be useful when a specific mutation is needed which cannot be performed by patchesStrategicMerge. For example, when needing to mutate a specific object within an array, the index can be specified as part of a patchesJson6902 mutation rule.
One distinction between this and other mutation methods is that patchesJson6902 does not support the use of conditional anchors. Use preconditions instead. Also, mutations using patchesJson6902 to Pods directly are not converted to higher-level controllers such as Deployments and StatefulSets through the use of the auto-gen feature. Therefore, when writing such mutation rules for Pods, it may be necessary to create multiple rules to cover all relevant Pod controllers.
This patch policy adds, or replaces, entries in a ConfigMap with the name config-game in any Namespace.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: policy-patch-cm
5spec:
6 rules:
7 - name: pCM1
8 match:
9 any:
10 - resources:
11 names:
12 - config-game
13 kinds:
14 - ConfigMap
15 mutate:
16 patchesJson6902: |-
17 - path: "/data/ship.properties"
18 op: add
19 value: |
20 type=starship
21 owner=utany.corp
22 - path: "/data/newKey1"
23 op: add
24 value: newValue1
If your ConfigMap has empty data, the following policy adds an entry to config-game.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: policy-add-cm
5spec:
6 rules:
7 - name: pCM1
8 match:
9 any:
10 - resources:
11 names:
12 - config-game
13 kinds:
14 - ConfigMap
15 mutate:
16 patchesJson6902: |-
17 - path: "/data"
18 op: add
19 value: {"ship.properties": "{\"type\": \"starship\", \"owner\": \"utany.corp\"}"}
This is an example of a patch that removes a label from a Secret:
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: policy-remove-label
5spec:
6 rules:
7 - name: remove-unwanted-label
8 match:
9 any:
10 - resources:
11 kinds:
12 - Secret
13 mutate:
14 patchesJson6902: |-
15 - path: "/metadata/labels/purpose"
16 op: remove
This policy rule adds elements to a list. In this case, it adds a new busybox container and a command. Note that because the path statement is a precise schema element, this will only work on a direct Pod and not higher-level objects such as Deployments.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: insert-container
5spec:
6 rules:
7 - name: insert-container
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 mutate:
14 patchesJson6902: |-
15 - op: add
16 path: "/spec/containers/1"
17 value: {"name":"busybox","image":"busybox:latest"}
18 - op: add
19 path: "/spec/containers/1/command"
20 value:
21 - ls
Note
Mutations usingpatchesJson6902 which match on Pods are not translated to higher-level Pod controllers as noted above.When needing to append an object to an array of objects, for example in pod.spec.tolerations, use a dash (-) at the end of the path.
1mutate:
2 patchesJson6902: |-
3 - op: add
4 path: "/spec/tolerations/-"
5 value: {"key":"networkzone","operator":"Equal","value":"dmz","effect":"NoSchedule"}
JSON Patch uses JSON Pointer to reference keys, and keys with tilde (~) and forward slash (/) characters need to be escaped with ~0 and ~1, respectively. For example, the following adds an annotation with the key of config.linkerd.io/skip-outbound-ports with the value of "8200".
1- op: add
2 path: /spec/template/metadata/annotations/config.linkerd.io~1skip-outbound-ports
3 value: "8200"
Some other capabilities of the patchesJson6902 method include:
- Adding non-existent paths
- Adding non-existent arrays
- Adding an element to the end of an array (use negative index
-1)
Strategic Merge Patch
The kubectl command uses a strategic merge patch with special directives to control element merge behaviors. Kyverno supports this style of patch to mutate resources. The patchStrategicMerge overlay resolves to a partial resource definition.
This policy adds a new container to the Pod, sets the imagePullPolicy, adds a command, and sets a label with the key of name and value set to the name of the Pod from AdmissionReview data. Once again, the overlay in this case names a specific schema path which is relevant only to a Pod and not higher-level resources like a Deployment.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: strategic-merge-patch
5spec:
6 rules:
7 - name: set-image-pull-policy-add-command
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 mutate:
14 patchStrategicMerge:
15 metadata:
16 labels:
17 name: "{{request.object.metadata.name}}"
18 spec:
19 containers:
20 - name: nginx
21 image: nginx:latest
22 imagePullPolicy: Never
23 command:
24 - ls
Note that when using patchStrategicMerge to mutate the pod.spec.containers[] array, the name key must be specified as a conditional anchor (i.e., (name): "*") in order for the merge to occur on other fields.
Conditional logic using anchors
Like with validate rules, conditional anchors are supported on mutate rules. Refer to the anchors section for more general information on conditionals.
An anchor field, marked by parentheses and an optional preceding character, allows conditional processing for mutations.
The mutate overlay rules support three types of anchors:
| Anchor | Tag | Behavior |
|---|---|---|
| Conditional | () | Use the tag and value as an “if” condition |
| Add if not present | +() | Add the tag value if the tag is not already present |
| Global | <() | Add the pattern when the global anchor is true |
The anchors values support wildcards:
*- matches zero or more alphanumeric characters?- matches a single alphanumeric character
Conditional anchors are only supported with the patchStrategicMerge mutation method.
Conditional anchor
A conditional anchor evaluates to true if the anchor tag exists and if the value matches the specified value. Processing stops if a tag does not exist or when the value does not match. Once processing stops, any child elements or any remaining siblings in a list will not be processed.
For example, this overlay will add or replace the value 6443 for the port field, for all ports with a name value that starts with “secure”:
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: policy-set-port
5spec:
6 rules:
7 - name: set-port
8 match:
9 any:
10 - resources:
11 kinds :
12 - Endpoints
13 mutate:
14 patchStrategicMerge:
15 subsets:
16 - ports:
17 - (name): "secure*"
18 port: 6443
If the anchor tag value is an object or array, the entire object or array must match. In other words, the entire object or array becomes part of the “if” clause. Nested conditional anchor tags are not supported.
Add if not present anchor
A variation of an anchor is to add a field value if it is not already defined. This is done by using the add anchor (short for “add if not present” anchor) with the notation +(...) for the tag.
An add anchor is processed as part of applying the mutation. Typically, every non-anchor tag-value is applied as part of the mutation. If the add anchor is set on a tag, the tag and value are only applied if they do not exist in the resource.
For example, this policy matches and mutates pods with an emptyDir volume to add the safe-to-evict annotation if it is not specified.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: add-safe-to-evict
5 annotations:
6 pod-policies.kyverno.io/autogen-controllers: none
7spec:
8 rules:
9 - name: "annotate-empty-dir"
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 mutate:
16 patchStrategicMerge:
17 metadata:
18 annotations:
19 +(cluster-autoscaler.kubernetes.io/safe-to-evict): true
20 spec:
21 volumes:
22 - <(emptyDir): {}
Global Anchor
Similar to validate rules, mutate rules can use the global anchor. When a global anchor is used, the condition inside the anchor, when true, means the rest of the pattern will be applied regardless of how it may relate to the global anchor.
For example, the below policy will add an imagePullSecret called my-secret to any Pod if it has a container image beginning with corp.reg.com.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: add-imagepullsecrets
5spec:
6 rules:
7 - name: add-imagepullsecret
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 mutate:
14 patchStrategicMerge:
15 spec:
16 containers:
17 - <(image): "corp.reg.com/*"
18 imagePullSecrets:
19 - name: my-secret
The below Pod meets this criteria and so the imagePullSecret called my-secret is added.
1apiVersion: v1
2kind: Pod
3metadata:
4 name: static-web
5 labels:
6 role: myrole
7spec:
8 containers:
9 - name: web
10 image: corp.reg.com/nginx
11 imagePullSecrets:
12 - name: my-secret
13 ports:
14 - name: web
15 containerPort: 80
16 protocol: TCP
Anchor processing flow
The anchor processing behavior for mutate conditions is as follows:
First, all conditional anchors are processed. Processing stops when the first conditional anchor returns a
false. Mutation proceeds only of all conditional anchors return atrue. Note that for conditional anchor tags with complex (object or array) values, the entire value (child) object is treated as part of the condition as explained above.Next, all tag-values without anchors and all add anchor tags are processed to apply the mutation.
Mutate Existing resources
In addition to mutation of “incoming” or “new” resources, Kyverno also supports mutation on existing resources with patchesStrategicMerge and patchesJson6902. Unlike standard mutate policies that are applied through the AdmissionReview process, mutate existing policies are applied in the background which update existing resources in the cluster. These mutate existing policies, like traditional mutate policies, are still triggered via the AdmissionReview process but apply to existing–and even different–resources. They may also optionally be configured to apply upon updates to the policy itself.
Because these mutations occur on existing resources, Kyverno may need additional permissions which it does not have by default. See the section on customizing permissions on how to grant additional permission to the Kyverno ServiceAccount to determine, prior to installing mutate existing rules, if additional permissions are required.
To define such a policy, trigger resources need to be specified in the match block. The target resources–resources that are mutated in the background–are specified in each mutate rule under mutate.targets. Note that all target resources within a single rule must share the same definition schema. For example, a mutate existing rule fails if this rule mutates both Pod and Deployment as they do not share the same OpenAPI V3 schema (except metadata).
Note
Mutation of existing Pods is limited to mutable fields only. See the Kubernetes documentation here for more details.This policy, which matches when the trigger resource named dictionary-1 in the staging Namespace changes, writes a label foo=bar to the target resource named secret-1 also in the staging Namespace.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: mutate-existing-secret
5spec:
6 rules:
7 - name: mutate-secret-on-configmap-event
8 match:
9 any:
10 - resources:
11 kinds:
12 - ConfigMap
13 names:
14 - dictionary-1
15 namespaces:
16 - staging
17 mutate:
18 targets:
19 - apiVersion: v1
20 kind: Secret
21 name: secret-1
22 namespace: "{{ request.object.metadata.namespace }}"
23 patchStrategicMerge:
24 metadata:
25 labels:
26 foo: bar
By default, the above policy will not be applied when it is installed. This behavior can be configured via mutateExistingOnPolicyUpdate attribute. If you set mutateExistingOnPolicyUpdate to true, Kyverno will mutate the existing secret on policy CREATE and UPDATE AdmissionReview events.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: mutate-existing-secret
5spec:
6 mutateExistingOnPolicyUpdate: true
7 rules:
8 - name: mutate-secret-on-configmap-event
9 match:
10 any:
11 - resources:
12 kinds:
13 - ConfigMap
14 names:
15 - dictionary-1
16 namespaces:
17 - staging
18...
Note
Installation of a mutate existing policy affects theValidatingWebhookConfiguration Kyverno manages as opposed to traditional mutate rules affecting the MutatingWebhookConfiguration.Variables Referencing Target Resources
To reference data in target resources, you can define the variable target followed by the path to the desired attribute. For example, using target.metadata.labels.env references the label env in the target resource.
This policy copies the ConfigMaps’ value target.data.key to their label with the key env.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: sync-cms
5spec:
6 mutateExistingOnPolicyUpdate: false
7 rules:
8 - name: concat-cm
9 match:
10 any:
11 - resources:
12 kinds:
13 - ConfigMap
14 names:
15 - cmone
16 namespaces:
17 - foo
18 mutate:
19 targets:
20 - apiVersion: v1
21 kind: ConfigMap
22 name: cmtwo
23 namespace: bar
24 - apiVersion: v1
25 kind: ConfigMap
26 name: cmthree
27 namespace: bar
28 patchesJson6902: |-
29 - op: add
30 path: "/metadata/labels/env"
31 value: "{{ target.data.key }}"
The {{ @ }} special variable is added to reference the in-line value of the target resource.
This policy adds the value of keyone from the trigger ConfigMap named cmone in the foo Namespace as the prefix to target ConfigMaps in their data with keynew.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: sync-cms
5spec:
6 mutateExistingOnPolicyUpdate: false
7 rules:
8 - name: concat-cm
9 match:
10 any:
11 - resources:
12 kinds:
13 - ConfigMap
14 names:
15 - cmone
16 namespaces:
17 - foo
18 mutate:
19 targets:
20 - apiVersion: v1
21 kind: ConfigMap
22 name: cmtwo
23 namespace: bar
24 - apiVersion: v1
25 kind: ConfigMap
26 name: cmthree
27 namespace: bar
28 patchStrategicMerge:
29 data:
30 keynew: "{{request.object.data.keyone}}-{{@}}"
Once a mutate existing policy is applied successfully, there will be an event and an annotation added to the target resource:
1$ kubectl describe deploy foobar
2...
3Events:
4 Type Reason Age From Message
5 ---- ------ ---- ---- -------
6 Normal PolicyApplied 29s (x2 over 31s) kyverno-mutate policy add-sec/add-sec-rule applied
7
8$ kubectl get deploy foobar -o yaml
9apiVersion: apps/v1
10kind: Deployment
11metadata:
12 annotations:
13 ...
14 policies.kyverno.io/last-applied-patches: |
15 add-sec-rule.add-sec.kyverno.io: added /spec/template/spec/containers/0/securityContext
To troubleshoot policy application failure, inspect the UpdateRequest Custom Resource to get details. Successful UpdateRequests may be automatically cleaned up by Kyverno.
For example, if the corresponding permission is not granted to Kyverno, you should see a value of Failed in the updaterequest.status field:
$ kubectl get ur -n kyverno
NAME POLICY RULETYPE RESOURCEKIND RESOURCENAME RESOURCENAMESPACE STATUS AGE
ur-swsdg add-sec mutate Deployment foobar default Failed 84s
$ kubectl describe ur ur-swsdg -n kyverno
Name: ur-swsdg
Namespace: kyverno
...
Status:
Message: deployments.apps "foobar" is forbidden: User "system:serviceaccount:kyverno:kyverno-service-account" cannot update resource "deployments" in API group "apps" in the namespace "default"
State: Failed
Mutate Rule Ordering (Cascading)
In some cases, it might be desired to have multiple levels of mutation rules apply to incoming resources. The match statement in rule A would apply a mutation to the resource, and the result of that mutation would trigger a match statement in rule B that would apply a second mutation. In such cases, Kyverno can accommodate more complex mutation rules, however rule ordering matters to guarantee consistent results.
For example, assume you wished to assign a label to each incoming Pod describing the type of application it contained. For those with an image having the string either cassandra or mongo you wished to apply the label type=database. This could be done with the following sample policy.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: database-type-labeling
5spec:
6 rules:
7 - name: assign-type-database
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 mutate:
14 patchStrategicMerge:
15 metadata:
16 labels:
17 type: database
18 spec:
19 (containers):
20 - (image): "*cassandra* | *mongo*"
Also, assume that for certain application types a backup strategy needs to be defined. For those applications where type=database, this would be designated with an additional label with the key name of backup-needed and value of either yes or no. The label would only be added if not already specified since operators can choose if they want protection or not. This policy would be defined like the following.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: database-backup-labeling
5spec:
6 rules:
7 - name: assign-backup-database
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 selector:
14 matchLabels:
15 type: database
16 mutate:
17 patchStrategicMerge:
18 metadata:
19 labels:
20 +(backup-needed): "yes"
In such a case, Kyverno is able to perform cascading mutations whereby an incoming Pod that matched in the first rule and was mutated would potentially be further mutated by the second rule. In these cases, the rules must be ordered from top to bottom in the order of their dependencies and stored within the same policy. The resulting policy definition would look like the following:
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: database-protection
5spec:
6 rules:
7 - name: assign-type-database
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 mutate:
14 patchStrategicMerge:
15 metadata:
16 labels:
17 type: database
18 spec:
19 (containers):
20 - (image): "*cassandra* | *mongo*"
21 - name: assign-backup-database
22 match:
23 any:
24 - resources:
25 kinds:
26 - Pod
27 selector:
28 matchLabels:
29 type: database
30 mutate:
31 patchStrategicMerge:
32 metadata:
33 labels:
34 +(backup-needed): "yes"
Test the cascading mutation policy by creating a Pod using the Cassandra image.
1$ kubectl run cassandra --image=cassandra:latest
2pod/cassandra created
Perform a get or describe on the Pod to see the result of the metadata.
1$ kubectl describe po cassandra
2Name: cassandra
3Namespace: default
4<snip>
5Labels: backup-needed=yes
6 run=cassandra
7 type=database
8<snip>
As can be seen, both type=database and backup-needed=yes were applied according to the mutation rules.
Verify that applying your own backup-needed label with the value of no triggers the first mutation rule but not the second.
1$ kubectl run cassandra --image=cassandra:latest --labels backup-needed=no
Perform another get or describe to verify the backup-needed label was not altered by the mutation rule.
1$ kubectl describe po cassandra
2Name: cassandra
3Namespace: default
4<snip>
5Labels: backup-needed=no
6 type=database
7<snip>
foreach
A foreach declaration can contain multiple entries to process different sub-elements e.g. one to process a list of containers and another to process the list of initContainers in a Pod.
A foreach must contain a list attribute that defines the list of elements it processes and either a patchStrategicMerge or patchesJson6902 declaration. For example, iterating over the list of containers in a Pod is performed using this list declaration:
1list: request.object.spec.containers
2patchStrategicMerge:
3 spec:
4 containers:
5 ...
When a foreach is processed, the Kyverno engine will evaluate list as a JMESPath expression to retrieve zero or more sub-elements for further processing. The value of the list field may also resolve to a simple array of strings, for example as defined in a context variable. The value of the list field should not be enclosed in braces even though it is a JMESPath expression.
A variable element is added to the processing context on each iteration. This allows referencing data in the element using element.<name> where name is the attribute name. For example, using the list request.object.spec.containers when the request.object is a Pod allows referencing the container image as element.image within a foreach.
Each foreach declaration can optionally contain the following declarations:
- Context: to add additional external data only available per loop iteration.
- Preconditions: to control when a loop iteration is skipped.
foreach: a nestedforeachdeclaration described below.
For a patchesJson6902 type of foreach declaration, an additional variable called elementIndex is made available which allows the current index number to be referenced in a loop.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: foreach-json-patch
5spec:
6 rules:
7 - name: add-security-context
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 preconditions:
14 any:
15 - key: "{{ request.operation }}"
16 operator: Equals
17 value: CREATE
18 mutate:
19 foreach:
20 - list: "request.object.spec.containers"
21 patchesJson6902: |-
22 - path: /spec/containers/{{elementIndex}}/securityContext
23 op: add
24 value: {"runAsNonRoot" : true}
For a complete example of the patchStrategicMerge method that mutates the image to prepend the address of a trusted registry, see below.
1apiVersion : kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: prepend-registry
5spec:
6 background: false
7 rules:
8 - name: prepend-registry-containers
9 match:
10 any:
11 - resources:
12 kinds:
13 - Pod
14 preconditions:
15 all:
16 - key: "{{request.operation}}"
17 operator: AnyIn
18 value:
19 - CREATE
20 - UPDATE
21 mutate:
22 foreach:
23 - list: "request.object.spec.containers"
24 patchStrategicMerge:
25 spec:
26 containers:
27 - name: "{{ element.name }}"
28 image: registry.io/{{ images.containers."{{element.name}}".name}}:{{images.containers."{{element.name}}".tag}}
Note that the patchStrategicMerge is applied to the request.object. Hence, the patch needs to begin with spec. Since container names may have dashes in them (which must be escaped), the {{element.name}} variable is specified in double quotes.
Nested foreach
The foreach object also supports nesting multiple foreach declarations to form loops within loops. This is especially useful when the mutations you need to perform are either replacements or removals as these require the use of JSON patches (patchesJson6902). When using nested loops, the special variable {{elementIndex}} requires a loop number to identify which element to process. Preconditions are supported only at the top-level loop and not per inner loop.
For example, consider a scenario in which you must replace all host names which end in old.com with new.com in an Ingress resource under the spec.tls[].hosts[] list. Because spec.tls[] is an array of objects, and hosts[] is an array of strings within each object, a foreach declaration must iterate over each object in the tls[] array and then internally loop over each host in the hosts[] array.
1apiVersion: networking.k8s.io/v1
2kind: Ingress
3metadata:
4 name: myingress
5 labels:
6 app: myingress
7spec:
8 rules:
9 - host: myhost.corp.com
10 http:
11 paths:
12 - backend:
13 service:
14 name: myservice
15 port:
16 number: 8080
17 path: /
18 pathType: ImplementationSpecific
19 tls:
20 - hosts:
21 - foo.old.com
22 - bar.old.com
23 secretName: mytlscertsecret
This type of advanced mutation can be performed with nested foreach loops as shown below. Notice that in the JSON patch, the path value references the current index of tls[] as {{elementIndex0}} and the current index of hosts[] as {{elementIndex1}}. In the value field, the {{element}} variable still references the current value of the hosts[] array being processed.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: replace-image-registry
5spec:
6 background: false
7 rules:
8 - name: replace-dns-suffix
9 match:
10 any:
11 - resources:
12 kinds:
13 - Ingress
14 mutate:
15 foreach:
16 - list: request.object.spec.tls[]
17 foreach:
18 - list: "element.hosts"
19 patchesJson6902: |-
20 - path: /spec/tls/{{elementIndex0}}/hosts/{{elementIndex1}}
21 op: replace
22 value: "{{ replace_all('{{element}}', '.old.com', '.new.com') }}"