The first version of the renderer did the obvious thing: accept a job, start FFmpeg, return when it finished. It worked for one user. It worked for ten. At about forty concurrent jobs the machine stopped responding to health checks, the orchestrator declared it dead, and every job on it restarted — onto the machines that were already struggling. The failure was not capacity; it was that nothing in the system was allowed to say no.

Saying no is a feature

A queue without a bound is not a queue, it is a memory leak with a nice API. The fix was unglamorous: a semaphore per worker sized to real CPU count, a bounded channel in front of it, and an explicit 429 when the channel was full.

// admit returns false instead of queueing forever.
func (w *Worker) admit(job Job) bool {
    select {
    case w.slots <- job:
        return true
    default:
        metrics.Rejected.Inc()
        return false
    }
}

What the numbers did

Throughput went down. Completion rate went up. Those are different things, and only one of them is on the dashboard by default:

Metric Before After
Jobs accepted per minute 62 48
Jobs actually completed 31 47
p99 latency 14 min 2m 40s

A system that cannot reject work will eventually reject all of it.

— something my ops lead said twice before I listened

Three things I would now do on day one:

  1. Bound every queue, and make the bound a config value someone will read.
  2. Return a real rejection, with a retry-after, before you return a timeout.
  3. Graph rejections next to throughput, or nobody will believe the trade.

Notes on execution:

  • Unordered lists use a subtle marker in muted text color.
  • Nested lists step in and maintain readable rhythm.
  • Rejection metrics should always be treated as a first-class health signal.