AI Infrastructure Engineer, Model Serving Platform

The shape of it

In the posting’s own words

As a Software Engineer on the ML Infrastructure team, you will design and build platforms for scalable, reliable, and efficient serving of LLMs. Our platform powers cutting-edge research and production systems, supporting both internal and external use cases across various environments.

What it asks for · 7

• 4+ years of experience building large-scale, high-performance backend systems.
• Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.)
• Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc.
• Experience with containers and orchestration tools (e.g., Docker, Kubernetes).
• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).
• Proven ability to solve complex problems and work independently in fast-moving environments.
• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

Also a plus

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

Tools and skills named

Words the posting leans on

• experience6×
• llm6×
• system6×
• design4×
• e.g4×
• serving4×
• build3×
• infrastructure3×
• platform3×
• research3×
• backend2×
• cases2×
• engineer2×
• fundamentals2×
• high-performance2×
• internal2×

Counted from the posting after the mission statement and the legal notices are set aside. The ones near the top are the ones a screener is looking for.

More open at Scale AI

• AI Advisory ConsultantSan Francisco, CA; New York, NY
• AI Advisory PrincipalSan Francisco, CA; New York, NY
• AI Applications Ops Manager, GPSDoha, Qatar
• AI Builder InternSan Francisco, CA
• AI Infrastructure Engineer, Sandbox PlatformLondon, UK
• AI Infrastructure Engineer, Sandbox PlatformSan Francisco, CA; Seattle, WA; New York, NY