Research Scientist, Agent Robustness

The shape of it

In the posting’s own words

As the leading data and evaluation partner for frontier AI companies, Scale plays an integral role in understanding the capabilities and safeguarding AI models and systems. Building on this expertise, Scale Labs has launched a new team focused on policy research, to bridge the gap between AI research and global policymakers to make informed, scientific decisions about AI risks and capabilities.

What it asks for · 5

• Experience with post-training and RL techniques such as RLHF, DPO, GRPO, and similar approaches.
• A track record of published research in machine learning, particularly in generative AI.
• At least three years of experience addressing sophisticated ML problems, whether in a research setting or in product development.
• Hands-on experience with agent evaluation frameworks such as SWE-bench, WebArena, OSWorld, Inspect, or similar tools.
• Experience with red-teaming, prompt injection, or adversarial testing of AI systems.

Also a plus

• Hands-on experience with agent evaluation frameworks such as SWE-bench, WebArena, OSWorld, Inspect, or similar tools.
• Experience with red-teaming, prompt injection, or adversarial testing of AI systems.

Tools and skills named

Words the posting leans on

• research12×
• agent10×
• experience6×
• risk5×
• capabilities4×
• evaluation4×
• agent robustness3×
• building3×
• design3×
• industry3×
• systems3×
• build2×
• failure modes2×
• frontier2×
• harnesses2×
• labs2×

Counted from the posting after the mission statement and the legal notices are set aside. The ones near the top are the ones a screener is looking for.

More open at Scale AI

• AI Advisory ConsultantSan Francisco, CA; New York, NY
• AI Advisory PrincipalSan Francisco, CA; New York, NY
• AI Applications Ops Manager, GPSDoha, Qatar
• AI Builder InternSan Francisco, CA
• AI Infrastructure Engineer, Model Serving PlatformSan Francisco, CA; New York, NY
• AI Infrastructure Engineer, Sandbox PlatformLondon, UK