Applied ML/AI Engineer - Monitoring

About the job

The monitoring team implements the foundational capabilities of Sifflet: detecting data quality issues across a wide range of data warehouses and databases.

Sifflet’s monitoring capabilities rely heavily on machine learning (ML) techniques. Most advanced data quality checks are based on time series forecasting models that detect unexpected distribution changes while accounting for seasonality and one-off events. Additionally, ML-based features are present throughout our product, be it for intelligent alert grouping, automated incident description, or automated monitor suggestions.

As a machine learning engineer on the monitoring team, you will:

• Build automated data profiling systems that learn normal data patterns and detect deviations.

• Deploy time series forecasting models that understand data seasonality and business cycles.

• Create intelligent alerting systems that reduce noise through ML-powered incident correlation.

• Implement generative AI workflows across the product, such as enabling users to describe their monitoring needs in natural language.

• Contribute to product decisions and identify areas where adding ML/AI-based features can solve customer pain points.

As we’re a small team, you will be expected to design, implement, deploy and maintain your projects in production, and integrate them with other services. Thus, this role includes a significant software engineering component.

Some projects you could be working on

• Automated monitor recommendations based on data profiling metrics.

• Automated root cause analysis of any data quality incident, building upon the many sources of metadata Sifflet collects (table lineage, query history, past monitor failures…).

Our stack

• The monitoring engine is built with Python 3 and its large data/ML ecosystem (notably PyTorch).

• The web API is written in (modern) Java with Spring Boot 3, the web frontend is a VueJS application written in Typescript. You may occasionally need to make minor changes to this code base.

• Infrastructure: Kubernetes (AWS EKS clusters), MySQL (on AWS RDS), Temporal for job orchestration

• Plus a few supporting services: Gitlab CI, Prometheus/Loki/Grafana, Sentry…

While not directly part of our stack, expect to gain a lot of knowledge on many products in the modern data ecosystem. The subtleties of BigQuery or Snowflake will soon be very familiar to you.