Director, Research - AI Evals

The shape of it

In the posting’s own words

The Figma Research team is hiring an Director, Research - AI Evals to own how we measure the quality of Figma's AI-powered experiences. As Figma ships more AI capabilities across our products, the question "is this really good?" has never mattered more — and answering it rigorously is what this role exists to do. You'll define what "good" means for our AI features, build the frameworks and quality bars to measure it, and turn that into trusted signal that product teams rely on to decide what to ship.

What it asks for · 6

• 10+ years of experience in product, research, applied research, or a closely related field, including 2+ years of management experience
• Direct, hands-on experience owning the evaluation of AI/LLM-powered products
• Expertise designing and running AI evaluation — human evaluation programs, rubric and benchmark/golden-dataset construction, inter-rater reliability — and sound judgment about when and how to apply automated/model-based approaches (e.g., LLM-as-judge), including their limitations
• Strength across both qualitative and quantitative methods, comfort with data and metrics, and the ability to reason about model behavior
• Demonstrated success in identifying the riskiest assumptions behind an ambiguous quality question, prioritizing them, and designing right-sized evaluation to build confidence
• A proven track record of gaining buy-in from executive and cross-disciplinary stakeholders — transcending methodology to articulate a larger user story and the "so what" to inspire action

Also a plus

• Experience building or co-building automated evaluation pipelines and regression testing in partnership with engineering, or familiarity with eval tooling (e.g., Braintrust, LangSmith, DeepEval, or equivalents)
• Experience standing up a new function, practice, or discipline from scratch
• 2+ years in product design, user-centric product management, data science, product development, and/or front-end engineering
• A familiarity and depth of experience using Figma's products

What the job covers

• Own AI evaluation methods and operations for Figma's AI-powered experiences — define quality dimensions, design how we measure them, and turn results into decision-ready signal
• Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars, combining human evaluation with automated/model-based approaches (e.g., LLM-as-judge) where appropriate
• Partner with engineering to stand up repeatable, reproducible evaluation pipelines and regression testing, so evaluation is a routine part of how AI features are built and shipped
• Produce clear readouts and dashboards that let stakeholders confidently make go/no-go and prioritization decisions
• Socialize a shared definition of quality so evaluation standards are adopted across teams rather than re-invented — and advocate for evaluation as a strategic partner in the product process
• Manage a small team to execute our AI evals in partnership with contractors, internal staff, and/or LLMs

Tools and skills named

Words the posting leans on

• evaluation14×
• product13×
• experience9×
• quality7×
• design6×
• engineering4×
• research4×
• build3×
• evals3×
• features3×
• human evaluation3×
• measure3×
• ai-powered experiences2×
• ai/llm-powered products2×
• and/or2×
• approaches e.g2×

Counted from the posting after the mission statement and the legal notices are set aside. The ones near the top are the ones a screener is looking for.

More open at Figma

• Account Executive, Emerging Enterprise (Berlin, Germany)Berlin, Germany
• Account Executive, EnterpriseSan Francisco, CA • New York, NY • United States
• Account Executive, Enterprise (Bengaluru, India)Bengaluru, India
• Account Executive, Enterprise (Berlin, Germany)Berlin, Germany
• Account Executive, Enterprise, Mandarin Speaking (Singapore)Singapore
• Account Executive, Enterprise (Paris, France)Paris, France