Skip to content

Lab: Application Performance Monitoring

Goals

  1. Understand GDI path for APM for common tech stacks (Docker, K8s).
  2. Be able to instrument an app from scratch (traces, custom metadata).
  3. Understand how distributed tracing works across tech stacks (header propagation, …)
  4. Understand positioning vs. trad. APM vendors

Task 10: Microservices Auto-instrumentation

The development team has broken up the monolithic service into microservices baesd on the docker-compose setup. Switch to the provided milestone 10microservices with the instructions from "Getting Started".

Test the service with:

curl -X POST http://127.0.0.1:8000/api -F text=@hamlet.txt

Add auto-instrumentation to the public_api microservice using the Splunk distribution of OpenTelemetry Python. Review the documentation and the getting Started steps and apply it to Dockerfile.

Take into account the trace exporter settings and add the required environment variables to the .env file for docker-compose. Use the configuration to send traces directly to Splunk Observability Cloud.

The milestone for this task is 10microservices-autoi. It has auto-instrumentation applied for all microservices.

Task 11: Infrastructure Correlation

There is no task 11 (yet).

Task 12: Instrumentation in Kubernetes

The development team has started using Kubernetes for container orchestration. Switch to the provided milestone 12microservices-k8s with the instructions from "Getting Started".

The Kubernetes manifests are located in the k8s folder. Add auto-instrumentation to the public_api microservice deployment by configuring the Splunk distribution of OpenTelemetry Python. The Dockerfile has already been prepared.

Install the OpenTelemetry Collector to the environment using Splunk's helm chart and use the provided values.yaml:

helm install my-splunk-otel-collector --set="splunkObservability.realm=${SPLUNK_REALM},splunkObservability.accessToken=${SPLUNK_ACCESS_TOKEN},clusterName=${CLUSTER_NAME}" splunk-otel-collector-chart/splunk-otel-collector -f values.yaml

Rebuild the container images for the private registry:

docker-compose build

Push the images to the private registry:

docker-compose push

Deploy to the cluster with

kubectl apply -f k8s

Test the service with

ENDPOINT=$(kubectl get service/public-api -o jsonpath='{.spec.clusterIP}')
curl http://$ENDPOINT:8000/api -F text=@hamlet.txt

The milestone for this task is 12microservices-k8s-autoi. It has auto-instrumentation applied for all microservices.

Task 13: Using OpenTelemetry instrumentation

Future Tasks

TODO YOUR Idea here? Let us know!

TODO metrics method being traced - how to disable?

from opentelemetry.context import attach, detach, set_value
token = attach(set_value("suppress_instrumentation", True))

TODO autodetect metrics with k8s labels: prometheus.io/scrape: true - run prometheus on separate port 9090.

TODO tracing examples

Last update: January 17, 2022
Back to top