Anomaly Detection for European Steel Manufacturer

Background and Problem

A leading European steel manufacturer was losing millions of euros annually to scrap metal from their heat treatment furnaces, with no perceivable patterns. Terabytes of valuable sensor data sat trapped across three disconnected databases, without a way for shop-floor domain experts to inspect it.

Result

Our solution saved multiple millions of euros in scrap reduction by identifying a critical furnace lid malfunction that had gone undetected. We developed a live dashboard which became a permanent fixture on the shop floor, enabling operators to spot and address issues immediately. Through data analysis and anomaly detection modeling, we discovered that temperature differentials (often caused by improperly sealed furnace lids) were the primary cause of quality issues, leading to targeted operator training and further scrap reduction.

Image for illustration by danjoualex from Pixabay

Description of the solution

We built a comprehensive data platform that transformed sensor readings into actionable intelligence. The solution consisted of three integrated components:

Data Engineering Infrastructure: We implemented a modern data lake architecture using Spark for distributed processing and Airflow for orchestration. Robust, auto-scaling streaming pipelines processed terabytes of historical and real-time data, with automatic retry mechanisms ensuring reliability.

Interactive Diagnostics Dashboard: Working closely with furnace operators, we developed a Plotly Dash dashboard that visualized critical metrics and anomalies in real-time. This tool directly led to the discovery of the faulty furnace lid that was causing millions in losses.

Anomaly Detection System: I implemented an XGBoost-based anomaly detection model trained on time-series sensor data. The model analyzed patterns across hundreds of sensor streams to identify deviations from normal operating conditions. A key insight was that a lot of scrap could be saved by ensuring the furnace was shut properly.

Technologies used

Python, Apache Spark, Airflow, Django, Plotly Dash, XGBoost, Hive, Livy, supervisord