Hippocratic AI · HR, India
Skills: Python, AWS, Docker, CI/CD
About the Role ------------------
We're looking for a Staff Site Reliability Engineer who is equally at home writing production software and running the infrastructure it lives on — and who wants to take ownership of one of the hardest, highest-leverage problems on our platform: intelligently managing a large fleet of GPU-backed models.
We run nearly 30 models across heterogeneous hardware, and keeping that fleet fast, reliable, and cost-effective is a serious engineering challenge. You'll build the GPU management and scheduling platform that sits at the center of it — collecting utilization and load metrics, interpreting what they actually mean, and using them to make real-time decisions about admission control and scaling. The goal: route and schedule inference calls so we use our capacity efficiently without exceeding it, and scale model replicas up and down automatically as demand shifts.
This is a senior role for someone with a decade in the field who can move fluidly between systems engineering and software development, and who is excited to own a complex, evolving system end to end.
What You'll Do ------------------
Design and build our GPU management and scheduling platform — the system that decides when, where, and how inference calls run across a fleet of ~30 models on heterogeneous hardware Build the metrics pipeline that collects GPU load and utilization data, and the logic that turns those signals into decisions Implement admission control to protect capacity — deciding when to accept, queue, or shed inference requests so we operate within fleet limits Build autoscaling that adjusts the number of model replicas in response to real-time demand and utilization Develop cloud orchestration systems and operators in Python and Go to manage the model fleet Architect and operate scalable, fault-tolerant, secure production systems on AWS, GCP, or Azure Design and build infrastructure automation and deployment pipelines (Terraform, CI/CD) as first-class software Stan…
Sign in to see the original posting
Browse more developer jobs on RoleBurst