Senior Software Engineer - Incident Insights & Readiness

The shape of it

In the posting’s own words

The Incident Insights & Readiness SRE team at Datadog fosters a resilient culture by using incidents as learning opportunities and catalysts for growth. Our users are Datadog engineers, and we build the software, tooling, and operational frameworks that help them prepare for, respond to, and learn from incidents. We work closely with engineering teams across Datadog to analyze incidents and turn those insights into better tools, stronger incident response, and organizational learning. Our efforts empower Datadog to navigate unexpected failures confidently, efficiently, and with a commitment to continuous learning and systems improvement.

What it asks for · 8

• At least 5 years of experience building software that solves real user problems. Experience designing new features and collaborating on code and technical design reviews. We primarily develop in Go and Python, with a bit of TypeScript.
• Experience building or operating distributed systems, with familiarity with Kubernetes and an understanding of complex failure modes.
• Demonstrated ability to independently own ambiguous technical problems from design through delivery while balancing long-term engineering quality with pragmatic execution.
• Experience analyzing incidents, identifying systemic risks, and driving engineering improvements informed by operational learnings.
• Experience participating in on-call rotations and improving incident response processes. Experience serving as an incident commander or incident coordinator is a plus.
• Empathy, collaboration, and communication skills in English to cultivate strong relationships across various teams in the organization
• Experience mentoring engineers, driving cross-functional initiatives, and influencing technical direction without relying on organizational authority.
• We welcome candidates from a variety of backgrounds, including software engineering, site reliability engineering, production engineering, infrastructure, and other roles focused on building reliable systems or improving incident response.

What the job covers

• Own and improve the on-call experience for the company by establishing best practices and building platforms to support on-call rotations and compensation.
• Define how we respond to incidents, lead the design and implementation of software to streamline the process, and collaborate with product teams to improve incident response across Datadog. Our aim is to fully support our incident responders in dealing with complexity.
• Contribute to the post-mortem process for the company, collaborating with teams on writing them, and identifying opportunities to reduce friction and enhance learning value for the organization. Our team also runs a weekly postmortem reading group.
• Support various teams in facilitating incident reviews that emphasize learning and blamelessness. Help them share their learnings across the organization to improve the resilience of our people.
• Provide technical leadership and day-to-day coaching to team members, accelerating their growth through design reviews, collaborative problem-solving and operational excellence best practices.
• Train our on-callers in incident and post-mortem processes, sharing expertise in incident management best practices. This involves both introducing newcomers to on-call responsibilities and refreshing the knowledge of existing engineers.
• Lead cross-functional initiatives in engineering organizations across Datadog, embedding with teams to understand their challenges and drive lasting improvements to reliability and operational excellence.

Tools and skills named

Words the posting leans on

• incident21×
• learning10×
• engineering8×
• experience8×
• incident response5×
• operational5×
• software5×
• build4×
• building4×
• culture4×
• design4×
• engineers4×
• improvement4×
• insights4×
• on-call4×
• organization4×

Counted from the posting after the mission statement and the legal notices are set aside. The ones near the top are the ones a screener is looking for.

More open at Datadog

• AI Research Engineer - Datadog AI Research (DAIR)Paris, France
• AI Research Scientist - Datadog AI Research (DAIR)New York, New York, USA; Pittsburgh, Pennsylvania, USA
• AI Research Scientist - Datadog AI Research (DAIR)Paris, France
• Area Vice President, Sales EngineeringBoston, Massachusetts, USA; Denver, Colorado, USA; New York, New York, USA; San Francisco, California, USA
• Associate Field Marketing Manager (US East and Canada)New York, New York, USA
• Associate Growth Marketing ManagerNew York, New York, USA; San Francisco, California, USA