Senior Software Engineer - Husky Storage

The shape of it

In the posting’s own words

As a Senior Software Engineer on this team, you'll work hands-on with a distributed storage system at petabyte scale, and help us evolve our platform to support the next level of scale that will power the AI era. We will move from a custom storage format towards open standards (Parquet, Arrow, DataFusion) enabling on prem solutions, increasingly complex analytical/batch workloads, and powering AI observability, sustainably into the future. You'll partner closely with Husky Ingestion, Metadata, and Platform, and increasingly with Lakehouse on the Iceberg-compatible boundary between Husky and analytical/batch workloads. You will also work with the open source community to build the next generation of reusable software solutions that can be operated at scale.

What it asks for · 4

• Enthusiastic and experienced with large-scale distributed systems and/or storage engines
• Comfortable with improving and operating large distributed systems that customers rely on for critical business functions
• Strong fundamentals in columnar/file storage formats (Parquet, ORC, Arrow) or comparable systems
• Bonus : experience with Go, Java, or Rust; with compaction or compression/encoding schemes (RLE, dictionary, delta, gorilla); with object storage (S3/GCS/Azure Blob) at scale, including caching and cost optimization; with Kafka and stream processing; or with Kubernetes

What the job covers

• Drive Husky to the next level of adoption by migrating our storage format to open source Parquet and re-imagine compaction on a format-agnostic, DataFusion-based substrate
• Drive concrete cost-reduction work: block-level inverted index compression, combining the database with external objects at the row level, and storage tiering/caching
• Scale our solution to enable customers to manage event versioning at scale, using temporal-database techniques.
• Play an influential role in operating and deciding how to best operate our service going forward.
• Work across the write path (with Ingestion) and toward a cleaner Iceberg-based handoff to ADP/Lakehouse for batch/analytical workloads

Tools and skills named

Words the posting leans on

• storage7×
• scale6×
• husky5×
• distributed4×
• event4×
• systems4×
• culture3×
• drive3×
• increasingly3×
• next3×
• open source3×
• parquet3×
• platform3×
• solutions3×
• analytical/batch workloads2×
• arrow2×

Counted from the posting after the mission statement and the legal notices are set aside. The ones near the top are the ones a screener is looking for.

More open at Datadog

• AI Research Engineer - Datadog AI Research (DAIR)Paris, France
• AI Research Scientist - Datadog AI Research (DAIR)New York, New York, USA; Pittsburgh, Pennsylvania, USA
• AI Research Scientist - Datadog AI Research (DAIR)Paris, France
• Applied Science InternParis, France
• Area Vice President, Sales EngineeringBoston, Massachusetts, USA; Denver, Colorado, USA; New York, New York, USA; San Francisco, California, USA
• Commercial Account ExecutiveSeoul, South Korea