November 2024 MLOps / Machine Learning / GenAI / Explainability / Model Monitoring / XGBoost
Customer Churn Prediction — MLOps + GenAI.
Leakage-safe churn model with Optuna-tuned XGBoost, SHAP-grounded LLM explanations, and a full MLOps loop — deployed live.
Overview
End-to-end churn prediction on the IBM Telco dataset (7,043 records): six models benchmarked under 5-fold stratified CV on PR-AUC, XGBoost tuned with Optuna (PR-AUC 0.62→0.67) with isotonic calibration and a cost-based decision threshold from an explicit 5:1 FN:FP cost — test PR-AUC 0.66 / ROC-AUC 0.85 / recall 0.87. A SHAP-grounded LLM explanation layer feeds per-prediction drivers into Pydantic-validated explanations with RAG-retrieved retention tactics (sentence-transformers + FAISS); a two-tier faithfulness eval caught a real grounding bug and raised faithfulness 0.72→0.90.
Highlights
- PR-AUC 0.62→0.67 via Optuna HPO
- isotonic calibration + cost-based threshold (5:1 FN:FP)
- test ROC-AUC 0.85, recall 0.87
- SHAP-grounded LLM explanations with RAG retention tactics
- faithfulness eval raised 0.72→0.90 (n=50, seeded)
- Single-source Pandera data contract enforced across train and serve
- Evidently drift monitoring with a 30%-drift retrain rule
- MLflow champion/challenger registry on DagsHub with gated promotion
- CI model-quality gate blocks merges below a PR-AUC floor
- Docker image cut 57% (11.4→4.9 GB)
- deployed live to Hugging Face Spaces (FastAPI + Streamlit), ~399 tests
Stack
Python · XGBoost · Optuna · SHAP · MLflow · DVC · Pandera · Evidently · FastAPI · Docker · FAISS · GitHub Actions