About the client
A media business that publishes video for several projects and publishers and needs every upload ready for adaptive streaming.
The client needed a service that takes a raw video dropped into S3 and produces everything a streaming player expects: several video resolutions, separate audio tracks per language, thumbnails and a short preview.
Uploads arrive unevenly and each conversion is heavy on CPU and memory. Paying for large machines around the clock made no sense, so the service had to grow with demand and disappear when idle.
What had to be solved
A single source file fans out into hundreds of outputs, and the pipeline had to stay predictable under load.
- Organize and upload hundreds of files per input, covering five video resolutions, five audio bitrates for each track, plus thumbnails and previews.
- Detect every audio stream, tag its language and list it correctly in the master playlist.
- Handle SQS duplicates, visibility timeouts and retries for failed jobs without double processing.
- Give each heavy job its own resources on Kubernetes and clean up after it.
- Keep output for each project or publisher in its own S3 location, and pick the best available encoder on any hardware.
How the team built it
A job manager long-polls SQS and reads the bucket and key from S3 event notifications. If the same file is already in progress, the message goes back to the queue with a delay. Each job then gets a dedicated pod, created from a template that sets resource limits, ephemeral volumes and node affinity for ARM instances.
FFmpeg produces five HLS resolutions from 1080p down to 240p in 4-second segments. The encoder is chosen automatically across NVIDIA, Intel Quick Sync and AMD, with a software fallback.
Every audio track is encoded to five bitrate tiers between 32 and 192 kbps, while commentary tracks get two. A master m3u8 links all video and audio variants, so players can switch quality and language.
Segments, playlists, thumbnails and a low-resolution preview go to S3 in parallel, in a separate folder per project or publisher. The autoscaler adds nodes as the queue grows and removes them all when it is empty.
What changed
The service turns uploads into stream-ready packages and only uses compute while there is work to do.
- Five adaptive video tiers, from 240p at 300 kbps to 1080p at 5000 kbps.
- Five audio tiers per track with language metadata kept, and two for commentary.
- Up to 5 concurrent jobs on up to 3 dedicated autoscaled nodes.
- Up to 500 parallel upload threads to S3, and $0 compute cost while idle.
