01
Data Pipeline Design
Loaded predictive-maintenance multimodal data into ADLS Gen2 and designed preprocessing based on a Medallion Architecture.
DATA ENGINEERING · AI
Facflexity
Industrial Predictive
Maintenance Pipeline
PERIOD
2025.12
TEAM
Microsoft Data School · 5 Members
ROLE
Data Pipeline · Model Training · Optimization
Databricks · Spark · Delta Lake · PyTorch · MLflow
01 — OVERVIEW
Building a scalable data and AI pipeline for industrial predictive maintenance.
Facflexity is an industrial predictive maintenance project that processes large-scale multimodal sensor and thermal-image data to predict equipment conditions and support operational decision-making.
The project combined Databricks-based data processing, multimodal model training, and inference optimization while addressing bottlenecks caused by large-scale data.
02 — ARCHITECTURE
From raw multimodal data to predictive inference.
Predictive-maintenance data was stored in ADLS Gen2 and processed through a Medallion Architecture in Databricks before being used for model training and inference.
ADLS Gen2
Raw Data Storage
→
Databricks
Data Processing
→
Bronze
Raw Ingestion
→
Silver
Preprocessing
→
Gold
Model-ready Data
→
LSTM + CNN
Multimodal Model
→
AutoML
LightGBM
→
Inference
Equipment State
03 — MY ROLE
02
Model Training
Trained and compared an LSTM + CNN multimodal model and an AutoML-based LightGBM model.
03
Pipeline Troubleshooting
Resolved source-data loading delays, timeouts, and Driver OOM issues by redesigning file discovery and large-scale data loading.
04
Inference Optimization
Analyzed equipment-state transition intervals and redesigned the inference cycle to reduce unnecessary operational calls.
04 — DATA LOADING
CHALLENGE
Large-scale source data caused loading delays and timeouts.
Loading the original dataset required discovering large numbers of files and resolving their schemas. This increased preprocessing overhead and caused loading delays and timeout failures.
BEFORE
Broad File Discovery
A broad file-search range and schema inference increased unnecessary loading overhead.
AFTER
Explicit Schema + Glob Pattern
Explicit schemas and Glob-based file filtering reduced unnecessary file discovery and stabilized loading.
SOLUTION
I constrained the file-search range with Glob patterns and defined schemas explicitly, removing unnecessary discovery work during ingestion.
05 — DRIVER OOM
MEMORY BOTTLENECK
Distributed processing became a single-node memory bottleneck.
The thermal dataset contained approximately 19,200 features per row across roughly 90,000 records. Converting the distributed Spark dataset into Pandas concentrated the data in Driver memory and caused OOM failures during model training.
BEFORE
Spark DataFrame
Distributed Data
Pandas
Collect to Driver
Driver Memory
Full Dataset
OOM
Training Failure
AFTER
Spark
Distributed Data
Filter
by device_id
Chunk
Subset Loading
PyTorch
Model Training
Memory Cleanup
gc · Cache Clearing
Next Chunk
Sequential Processing
SOLUTION
Instead of collecting the full distributed dataset into Driver memory, I filtered data by device_id, loaded each subset sequentially, trained the model chunk by chunk, and released memory before loading the next subset.
RESULT
The redesigned loading strategy eliminated the Driver OOM bottleneck and enabled stable end-to-end PyTorch model training on the large-scale dataset.
06 — INFERENCE OPTIMIZATION
Redesigning inference around equipment-state transitions.
Rather than keeping the original 30-second inference interval fixed, I analyzed equipment-state transition intervals and redesigned the inference cycle to 90 seconds.
INFERENCE CYCLE
30s → 90s
Cycle redesigned based on equipment-state transition intervals.
ESTIMATED MONTHLY CALLS
≈ 67% ↓
Estimated reduction in the number of monthly inference calls.
ESTIMATED INFERENCE TIME
323h → 108h
Calculated reduction in total monthly inference time.
These figures were calculated from the redesigned inference schedule. They represent estimated operational impact rather than long-term production measurements.
07 — RESULT
01
Stable Data Loading
Reduced loading bottlenecks through explicit schemas and constrained file discovery.
02
Stable Model Training
Resolved Driver OOM through device-based sequential chunk loading and memory management.
03
Inference Efficiency
Redesigned inference scheduling and quantified the estimated reduction in monthly compute workload.
08 — WHAT I LEARNED
Understanding the data can change the entire system design.
This project showed me that analyzing data volume, distribution, and equipment-state patterns can reveal optimization opportunities beyond model performance. Data-loading strategy and inference design directly affect system stability and operational efficiency.
←Back to Projects
Facflexity · 2025