Batching, Queueing and Backpressure

Description

Queue depth, batch size and tail latency pull against each other. What we learned running a shared inference tier, and the backpressure design that stopped one team's spike from taking down everyone.