Skip to content
Muhammet Şafak
tr
Asked by: Ece Answered:

How do I set up distributed tracing (OpenTelemetry) across microservices?


Question

A request travels API Gateway → Auth → Order → Stock. We're struggling to find the source of an 800ms delay in one of the services in between. With OpenTelemetry we plan to carry a `trace_id` across all services (Laravel and Go) and visualize it in a central APM (Jaeger/Grafana Tempo). How do I design context propagation over HTTP headers and queue messages?

Answer

Short answer: don’t invent a custom scheme; standardize on W3C Trace Context (traceparent/tracestate) so PHP and Go interoperate without friction.

Your real problem is “where’s the 800ms” — and to see it the trace has to travel across services and queues without breaking. Here’s the mechanism:

  1. Make W3C Trace Context the standard. Start the root span at the gateway; every service extracts context from the incoming HTTP header and injects it into outgoing calls. Because it’s a standard header, Laravel and Go read the same traceparent in common — don’t invent your own trace_id header.
  2. Carry the same traceparent through queues. When you publish a message, add traceparent as a message header/attribute and re-extract it in the worker. This is exactly the piece teams forget; skip it and async hops show up as orphan traces in the APM and the chain breaks.
  3. Put the OTel SDK in each service, ship via OTLP. Service → OTLP → a Collector → Jaeger/Tempo. A Collector in between decouples export from the app and makes swapping backends easy.
  4. Choose sampling smartly. Use parent-based so all spans of a trace share the same fate; add tail sampling on top so you definitely keep the 800ms outliers. With head-only sampling you drop the very slow trace you wanted to see.
  5. Don’t hand-roll trace_id. Don’t write your own header read/write code; the SDK’s context propagation already carries it. Do it by hand and you’ll forget it somewhere and the trace snaps there.

Bottom line: I’d set up W3C Trace Context + the OTel SDK + a Collector + Tempo/Jaeger, leave propagation to the SDK, and never skip putting traceparent into the message header on the queue hop. I explain the architectural reasoning for why I build it this way on sade.dev; you’ll find the detail of the Go-side HTTP mechanics in the hub post. Only an unbroken trace shows you the 800ms.

Related Reading

Tags: #architecture#observability#microservices
Share:

Comments

Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.

More Questions

All questions

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind