Case study · Observability
Distributed tracing for Pluto TV's ad revenue services
One trace per ad request across Lambdas, containers, queues and an external transcoding platform, extended to 20+ services and real-user monitoring on mobile, web and TV apps.
- Product
- pluto.tv ↗
- Client
- Pluto TV
- Industry
- Free ad-supported streaming (FAST)
- Role
- Senior Software Engineer & Tech Leader
- Period
- 2020 – 2024
- 20+
- services traced end to end
- 3
- app platforms with RUM: mobile, web, TV
- 1
- trace id per ad request
Scroll sideways to see the full diagram →
Context
Pluto TV is a free, ad-supported streaming service. Its revenue services decide, fetch and prepare the ads shown in every ad break, across AWS Lambdas, containerized services, SQS and SNS, third-party VAST ad servers and Hybrik, an external transcoding platform. When something broke, revenue was on the line.
The challenge
Troubleshooting relied on logs spread across Loggly and CloudWatch and on manual correlation: engineers reconstructed each request by hand, service by service. Asynchronous hops (SQS messages, SNS notifications, an external platform in the middle) broke any simple correlation.
The solution
Distributed tracing with Datadog APM across the revenue services: a proof of concept on the ad request path, then rolled out to more than 20 services, plus Datadog RUM on the mobile, web and TV apps. Trace context is propagated over HTTP, SQS and SNS, even through Hybrik; spans carry business tags; logs live inside their parent spans.
Context propagation
The trace context travels over HTTP (adProxy → adRequester), SQS messages (adProxy → adCollector) and SNS notifications (adProxy → Hybrik → SNS → Lambda).
Through an external platform
The trace context is injected into the payload sent to Hybrik and extracted by the Lambda that receives its SNS notification, so the transcoding job stays inside the same trace.
Tags as automatic metrics
Business tags (app, device, partner, endpoint, status) on spans turn into metrics and filters without extra instrumentation.
Logs inside spans
Log entries are attached to their parent span, and request data such as query strings is written once as a tag on the root span instead of repeated in every log.
Real-user monitoring
Datadog RUM on the mobile, web and TV apps connects what viewers experience with the backend traces behind it.
Key decisions
- 01
Tracing, not more logging
Logs show what happened in one place; traces show one request end to end. The goal was a single trace id holding the complete history of a request, with errors highlighted at the trace level.
Alternatives considered Richer logs and correlation ids searched across tools.
- 02
Carry the trace through a third party in its own payload
Hybrik does not propagate trace headers. Injecting the trace context into the job payload sent to Hybrik, and extracting it in the Lambda that handles the SNS callback, kept the external transcoding step inside the same trace instead of starting a new one.
Alternatives considered Accepting a broken trace at the external hop; correlating by job id afterwards.
- 03
Prove it on the riskiest path first
The proof of concept covered three services and a transcoding Lambda that exercised every propagation mechanism (HTTP, SQS, SNS and an external platform), so the approach was validated where correlation was hardest.
- 04
Business context as tags, once
Putting device, partner and request data on spans makes every trace searchable by what matters to revenue, and writing heavy fields once on the root span cuts log redundancy.
- 05
From proof of concept to platform standard
Once the hardest path was traced end to end, the same conventions rolled out to more than 20 services and were extended to the client apps with RUM, so a viewer-side problem can be followed into the backend.
What I did
- Proposed and led the distributed tracing initiative for the revenue services.
- Built the proof of concept across adProxy, adCollector, adRequester and the transcoding status Lambda.
- Implemented trace propagation over HTTP, SQS and SNS, and through Hybrik by injecting the context into its payload.
- Rolled tracing out to more than 20 services and added Datadog RUM to the mobile, web and TV apps.
- Defined tagging and log-in-span conventions for the services.
Outcomes
- Distributed tracing across more than 20 services.
- Real-user monitoring on the mobile, web and TV apps, linked to backend traces.
- The complete history of an ad request in one trace id, with errors highlighted at the trace level.
- A direct route to the cause instead of manual, service-by-service reconstruction.