OpenTelemetry
OpenTelemetry Setup
Setting up OpenTelemetry requires configuration of a few YAML files. The required configurations are listed below.
Install Cert-Manager
Install Cert-Manager by following the documentation.
Config file for OpenTelemetry Collector
Create a configmap.yaml file in the kyverno Namespace with the following content:
1apiVersion: v1
2kind: ConfigMap
3metadata:
4 name: collector-config
5 namespace: kyverno
6data:
7 collector.yaml: |
8 receivers:
9 otlp:
10 protocols:
11 grpc:
12 endpoint: ":8000"
13 processors:
14 batch:
15 send_batch_size: 10000
16 timeout: 5s
17 extensions:
18 health_check: {}
19 exporters:
20 jaeger:
21 endpoint: "jaeger-collector.observability.svc.cluster.local:14250"
22 tls:
23 insecure: true
24 prometheus:
25 endpoint: ":9090"
26 logging:
27 loglevel: debug
28 service:
29 extensions: [health_check]
30 pipelines:
31 traces:
32 receivers: [otlp]
33 processors: []
34 exporters: [jaeger, logging]
35 metrics:
36 receivers: [otlp]
37 processors: [batch]
38 exporters: [prometheus, logging]
- Here the Prometheus exporter endpoint is set as 9090 which means Prometheus will be able to scrape this service on the given endpoint to collect metrics.
- Similarly, the Jaeger endpoint references a Jaeger collector at the default Jaeger endpoint 14250.
The Collector Deployment
Create a deployment.yaml file in the kyverno Namespace with the following content:
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: opentelemetrycollector
5 namespace: kyverno
6spec:
7 replicas: 1
8 selector:
9 matchLabels:
10 app.kubernetes.io/name: opentelemetrycollector
11 template:
12 metadata:
13 labels:
14 app.kubernetes.io/name: opentelemetrycollector
15 spec:
16 containers:
17 - name: otelcol
18 args:
19 - --config=/conf/collector.yaml
20 image: otel/opentelemetry-collector:0.50.0
21 volumeMounts:
22 - name: collector-config
23 mountPath: /conf
24 volumes:
25 - configMap:
26 name: collector-config
27 items:
28 - key: collector.yaml
29 path: collector.yaml
30 name: collector-config
This references the collector defined in the configmap.yaml above. Here we are using a Deployemtn with just a single replica. Ideally, a DaemonSet is preferred. Check the OpenTelemetry documentation for more deployment strategies.
The Collector Service
Finally, create a service.yaml file in the kyverno Namespace with the following content:
1apiVersion: v1
2kind: Service
3metadata:
4 name: opentelemetrycollector
5 namespace: kyverno
6spec:
7 ports:
8 - name: otlp-grpc
9 port: 8000
10 protocol: TCP
11 targetPort: 8000
12 - name: metrics
13 port: 9090
14 protocol: TCP
15 targetPort: 9090
16 selector:
17 app.kubernetes.io/name: opentelemetrycollector
18 type: ClusterIP
This defines a Service for the discovery of the collector Deployment.
Setting up Kyverno and passing required flags
See the installation instructions for Kyverno. Depending on the method used, the following flags must be passed.
- Pass the flag
metricsPortto defined the OpenTelemetry Collector endpoint for collecting metrics. - Pass the flag
otelConfig=grpcto export the metrics and traces to an OpenTelemetry collector on the metrics port endpoint
Setting up a secure connection between Kyverno and the collector
Kyverno also supports setting up a secure connection with the OpenTelemetry exporter using TLS on the server-side (on the collector). This will require you to create a certificate-key pair for the OpenTelemetry collector from some private CA and then saving the certificate as a Secret in your Kyverno Namespace with key named ca.pem.
Considering you already have the server.pem and server-key.pem files along with the ca.pem file (you can configure these using a tool such as OepnSSL or cfssl). Your OpenTelemetry configmap.yaml and deployment.yaml files will also change accordingly:
configmap.yaml
1apiVersion: v1
2kind: ConfigMap
3metadata:
4 name: collector-config
5 namespace: kyverno
6data:
7 collector.yaml: |
8 receivers:
9 otlp:
10 protocols:
11 grpc:
12 endpoint: ":8000"
13 tls:
14 cert_file: /etc/ssl/certs/server/server.pem
15 key_file: /etc/ssl/certs/server/server-key.pem
16 ca_file: /etc/ssl/certs/ca/ca.pem
17 processors:
18 batch:
19 send_batch_size: 10000
20 timeout: 5s
21 extensions:
22 health_check: {}
23 exporters:
24 jaeger:
25 endpoint: "jaeger-collector.observability.svc.cluster.local:14250"
26 tls:
27 insecure: true
28 prometheus:
29 endpoint: ":9090"
30 logging:
31 loglevel: debug
32 service:
33 extensions: [health_check]
34 pipelines:
35 traces:
36 receivers: [otlp]
37 processors: []
38 exporters: [jaeger, logging]
39 metrics:
40 receivers: [otlp]
41 processors: [batch]
42 exporters: [prometheus, logging]
deployment.yaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: opentelemetrycollector
5 namespace: kyverno
6spec:
7 replicas: 1
8 selector:
9 matchLabels:
10 app.kubernetes.io/name: opentelemetrycollector
11 template:
12 metadata:
13 labels:
14 app.kubernetes.io/name: opentelemetrycollector
15 spec:
16 containers:
17 - name: otelcol
18 args:
19 - --config=/conf/collector.yaml
20 image: otel/opentelemetry-collector:0.50.0
21 volumeMounts:
22 - name: collector-config
23 mountPath: /conf
24 - name: otel-collector-secrets
25 mountPath: /etc/ssl/certs/server
26 - name: root-ca
27 mountPath: /etc/ssl/certs/ca
28 volumes:
29 - configMap:
30 name: collector-config
31 items:
32 - key: collector.yaml
33 path: collector.yaml
34 name: collector-config
35 - secret:
36 secretName: otel-collector-secrets
37 name: otel-collector-secrets
38 - secret:
39 secretName: root-ca
40 name: root-ca
This will ensure that the OpenTelemetry collector can only accept encrypted data on the receiver endpoint.
Pass the flag transportCreds as the Secret name containing the ca.pem file (Empty string means insecure connection will be used).
Setting up Prometheus
- For the metrics backend, you can install Prometheus on you cluster. For a general example, we have a ready-made configuration for you. Install Prometheus by running:
1kubectl apply -k github.com/kyverno/grafana-dashboard/examples/prometheus
- Port-forward the Prometheus service to view the metrics on localhost.
1kubectl port-forward svc/prometheus-server 9090:9090 -n kyverno
Setting up Jaeger
The traces are pushed to the Jaeger backend on port 14250. To install Jaeger:
First install the Jaeger Operator. Replace the version as needed.
1kubectl create namespace observability
2kubectl create -n observability -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.33.0/jaeger-operator.yaml
3kubectl wait --for=condition=Available deployment --timeout=2m -n observability --all
Create a Jaeger resource configuration as shown below
jaeger.yaml
1apiVersion: jaegertracing.io/v1
2kind: Jaeger
3metadata:
4 name: jaeger
5 namespace: observability
Install the Jaeger backend
1kubectl create -f jaeger.yaml
Port-forward the Jaeger Service on 16686 to view the traces.
1kubectl port-forward svc/jaeger-query 16686:16686 -n observability