Staff Software Engineer - ML Observability

The shape of it

In the posting’s own words

As a Staff Engineer, you’ll lead the development of new features and foundational capabilities within Datadog’s LLM Observability product. You will shape product direction, drive experimentation, and apply your deep understanding of both AI systems and software engineering to solve open-ended problems in the fast-moving AI landscape. Your work will directly impact how our customers monitor, troubleshoot, and optimize LLM-based applications in production.

What it asks for · 8

• You have a BS/MS/PhD in a Computer Science, Engineering or related scientific field or equivalent experience
• Deep understanding of distributed systems and scalable backend architectures
• Hands-on experience building and shipping LLM-powered or GenAI applications.
• Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
• Ability to thrive in ambiguous, fast-changing spaces and have a product-oriented mindset
• You’re excited to shape the next generation of AI observability tools from the ground up
• Communicate clearly, think rigorously, and take pride in clean, maintainable code
• Experience with observability tools/platforms

What the job covers

• Drive design and implementation of LLM observability features.
• Ideate, prototype, and scale new product features to provide insights and drive improvements for generative AI systems
• Work cross-functionally with other eng teams, product, UX, and applied science to iterate fast and find product-market fit
• Develop and extend tools for tracing, evaluating, and debugging LLMs
• Influence architecture decisions and mentor engineers to build resilient, high-performance systems
• Stay close to customer pain points and use those insights to guide product and engineering priorities
• Stay current with industry trends and advancements in machine learning and observability, driving innovation within the team

Degree language

• You have a BS/MS/PhD in a Computer Science, Engineering or related scientific field or equivalent experience

Tools and skills named

Words the posting leans on

• observability8×
• systems6×
• product5×
• drive4×
• engineering4×
• features4×
• tools4×
• builds3×
• customers3×
• experience3×
• llm observability3×
• model3×
• applications2×
• architecture2×
• building2×
• deep understanding2×

Counted from the posting after the mission statement and the legal notices are set aside. The ones near the top are the ones a screener is looking for.

More open at Datadog

• AI Research Engineer - Datadog AI Research (DAIR)Paris, France
• AI Research Scientist - Datadog AI Research (DAIR)Paris, France
• AI Research Scientist - Datadog AI Research (DAIR)New York, New York, USA; Pittsburgh, Pennsylvania, USA
• Area Vice President, Sales EngineeringBoston, Massachusetts, USA; Denver, Colorado, USA; New York, New York, USA; San Francisco, California, USA
• Associate Field Marketing Manager (US East and Canada)New York, New York, USA
• Associate Growth Marketing ManagerNew York, New York, USA; San Francisco, California, USA