Skip to content
Muhammet Şafak
tr
Asked by: Ece Answered:

How do I set up distributed tracing (OpenTelemetry) across microservices?


Question

A request travels API Gateway → Auth → Order → Stock. We're struggling to find the source of an 800ms delay in one of the services in between. With OpenTelemetry we plan to carry a `trace_id` across all services (Laravel and Go) and visualize it in a central APM (Jaeger/Grafana Tempo). How do I design context propagation over HTTP headers and queue messages?

Answer

Short answer: don’t invent a custom scheme; standardize on W3C Trace Context (traceparent/tracestate) so PHP and Go interoperate without friction.

Short answer

Your real problem is “where’s the 800ms” — and to see it the trace has to travel across services and queues without breaking. I covered the Go-side HTTP mechanics, i.e. how context travels with the request, in writing HTTP services in Go.

Why

  1. A standard header is the common ground between two languages. Because W3C Trace Context is a standard, Laravel and Go read the same traceparent. Invent your own trace_id header and you owe a translation layer for every new service.

  2. The chain breaks most often at the queue hop. This is the piece teams forget: get the HTTP side right but fail to carry context when publishing a message, and async hops show up as orphan traces in the APM. I discussed the durability side of that same queue separately in the question about buffering the log pipeline.

  3. Head-only sampling drops exactly the trace you’re after. If you want to see the 800ms outlier, the start of the request is the wrong place to decide.

What to do

  1. Make W3C Trace Context the standard. Start the root span at the gateway; every service extracts context from the incoming HTTP header and injects it into outgoing calls.

  2. Carry the same traceparent through queues. When you publish a message, add traceparent as a message header/attribute and re-extract it in the worker.

  3. Put the OTel SDK in each service, ship via OTLP. Service → OTLP → a Collector → Jaeger/Tempo. A Collector in between decouples export from the app and makes swapping backends easy.

  4. Choose sampling smartly. Use parent-based so all spans of a trace share the same fate; add tail sampling on top so slow traces are definitely kept.

  5. Don’t hand-roll trace_id. Don’t write your own header read/write code; the SDK’s context propagation already carries it. Do it by hand and you’ll forget it somewhere and the trace snaps there.

Bottom line: I’d set up W3C Trace Context + the OTel SDK + a Collector + Tempo/Jaeger, leave propagation to the SDK, and never skip putting traceparent into the message header on the queue hop. I explain the architectural reasoning for why I build it this way on sade.dev. Only an unbroken trace shows you the 800ms.

Related Reading

Share:

Comments

Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.

More Questions

All questions

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind