How to implement Distributed Traces using OpenTelemetry on your Kubernetes cluster

In the dynamic world of microservices and cloud-native applications, maintaining visibility and understanding the intricate interactions between services can be a daunting task. The need for robust observability tools is more pressing than ever to ensure reliability, performance, and the overall health of the system. This is where distributed tracing comes into play, offering a powerful lens through which developers and operators can observe the flow of requests across service boundaries.

To setup distributed tracing, you’ll need to:

• install and configure OpenTelemetry
• install and configure Tempo to visualize traces in Grafana
• modify your nginx-ingress implementation to start sending traces to OpenTelemetry

This documentation will assume that you are using :

• kubernetes >= 1.28
• ingress-nginx >= 4.10.0
• helm >= v3.13.2

Install and configure OpenTelemetry Collector

The OpenTelemetry Collector processes and export telemetry data. This works with improved scalability and supports open source observability data formats (e.g. Jaeger, Prometheus, Fluent Bit, etc.) sending to one or more open source or commercial backends. In our case, Tempo from Grafana.

Add the chart :

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

Configuration :

The OpenTelemetry Collector Chart requires that mode is set. mode can be either daemonset, deployment, or statefulset depending on which kind of Kubernetes deployment your use case requires. In our example we will be using deployment.

values.yaml

mode: deployment

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false
  opencensus:
    enabled: true
    containerPort: 55678
    servicePort: 55678
    hostPort: 55678
    protocol: TCP

config:
  receivers:
    jaeger: null
    prometheus: null
    zipkin: null
    opencensus: {}
  exporters:
    otlp:
      endpoint: tempo.tempo:4317
      tls:
        insecure: true
  service:
    pipelines:
      traces:
        receivers:
          - otlp
          - opencensus
        exporters:
          - otlp
      metrics: null
      logs: null

This very simple configuration will let you receive otlp (enabled by default) and opencensus traces and export them to tempo (tempo.tempo:4317 will be installed later in this article). More configuration options can be found in the official helm chart repository.

Install the chart :

helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector -f values.yaml

You should now have the Opentelemetry Collector installed in the otel namespace (by default).

Now let’s configure Tempo to visualize traces in Grafana’s UI.