Customer Churn Prediction

End-to-end machine learning pipeline that predicts which telecom customers are likely to cancel their subscription — from raw data to a deployed REST API.

Primary dataset: IBM Telco Customer Churn — 7,043 customers, 21 features
Best model: LightGBM + Optuna · AUC-PR 0.668 · AUC-ROC 0.847
Generalisation: same pipeline applied to a second domain (E-Commerce) · AUC-PR 0.996
Business impact: cost-optimal threshold reduces misclassification cost by 43% vs default (£37k → £65k on test set)
Stack: Python 3.11 · scikit-learn · LightGBM · XGBoost · Optuna · SHAP · MLflow · FastAPI

---

Project Structure

customer-churn-prediction/
├── src/
│   ├── features/pipeline.py     # canonical feature engineering (used by notebooks + API)
│   ├── models/train.py          # pipeline builder + MLflow logging
│   ├── evaluation/metrics.py    # ROC/PR/SHAP plot helpers
│   ├── monitoring/drift.py      # PSI/KS drift detection (from scratch)
│   └── serving/app.py           # FastAPI prediction endpoint
├── notebooks/
│   ├── 01_eda.ipynb             # exploratory data analysis
│   ├── 02_feature_engineering.ipynb
│   ├── 03_model_training.ipynb  # CV, SHAP, calibration, MLflow, model export
│   ├── 04_cost_sensitive.ipynb  # business cost matrix, threshold optimisation
│   ├── 05_optuna_tuning.ipynb   # Bayesian HPO with Optuna TPE (50 trials, AUC-PR +5.1%)
│   ├── 06_generalization.ipynb  # same pipeline on E-Commerce dataset (AUC-PR 0.996)
│   └── 07_drift_monitoring.ipynb # PSI/KS drift on a simulated production batch
├── tests/
│   ├── test_pipeline.py         # data-leakage test + dummy baseline test
│   └── test_drift.py            # PSI/KS correctness (10 tests)
├── data/
│   ├── raw/                     # original CSV (not committed)
│   └── processed/               # train/test splits (not committed)
├── saved_models/                # trained pipeline pickle
├── reports/figures/             # all EDA and evaluation plots
├── Dockerfile
└── pyproject.toml

---

Results

IBM Telco — Model Comparison

Model	AUC-PR	AUC-ROC	Overfit gap
Dummy baseline	0.265	0.500	—
XGBoost	0.631	0.826	0.04
LightGBM (default)	0.636	0.830	0.03
LightGBM + Optuna	0.668	0.847	0.03

AUC-PR is the primary metric because the dataset is imbalanced (26.5% churn rate). Accuracy would be misleading here.

Optuna ran 50 TPE trials optimising 5-fold CV AUC-PR. The most impactful hyperparameter was learning_rate.

Generalisation — E-Commerce Dataset

The same sklearn Pipeline (preprocessing + LightGBM) was applied to a second domain with no structural changes — only column names and domain-specific engineered features differ.

Dataset	Domain	Customers	Churn rate	AUC-PR	AUC-ROC
IBM Telco	Telecom	7,043	26.5%	0.668	0.847
E-Commerce	Online retail	5,630	16.8%	0.996	0.999

Top SHAP features on E-Commerce: Tenure, Complain, NumberOfAddress, MaritalStatus_Single, WarehouseToHome.

Test set (LightGBM, threshold = 0.5):

• Correctly identified 266 out of 374 churners (71% recall)
• 230 false positives out of 1,035 non-churners (22% false alarm rate)
• Best F1 threshold: ~0.40

Cost-Sensitive Analysis

In churn prediction, a missed churner (false negative) is far more expensive than a false alarm (false positive). The default 0.5 threshold treats both errors equally — this section corrects that.

Cost assumptions (illustrative):

• False Negative (missed churner): £500 — lost customer lifetime value
• False Positive (false alarm): £50 — wasted retention campaign spend
• FN/FP cost ratio: 10x

Threshold comparison (test set, 1,409 customers):

Threshold	Churners caught	Missed	Total cost
Default (0.50)	266 / 374	108	£65,500
Best F1 (0.40)	288 / 374	86	£57,350
Cost-optimal (0.05)	363 / 374	11	£37,150

Projected annual saving (100k customers): ~£2M vs default threshold.

Two levers are applied in combination:

1. Threshold optimisation — shift the decision boundary to reflect the asymmetric cost
2. Cost-sensitive retraining — assign higher sample weights to churners during training (weight = FN/FP ratio = 10)

A sensitivity analysis shows the optimal threshold ranges from 0.46 (low cost ratio) to 0.01 (high cost ratio) — confirming that the right threshold must always be derived from actual business costs, not set arbitrarily.

---

Model Monitoring — PSI/KS Drift Detection

A validated model's performance guarantee has a shelf life: population, pricing, and contract mix all shift after deployment. src/monitoring/drift.py implements Population Stability Index (PSI) and the Kolmogorov-Smirnov (KS) statistic from scratch, with standard PSI alarm bands (<0.10 stable, 0.10-0.25 moderate, >0.25 significant).

Notebook 07 simulates a 6-months-post-deployment production batch (price hike, a new-customer acquisition wave, resampling toward higher-churn payment/contract segments) against an untouched control batch:

Feature	Type	PSI (drifted)	PSI (control)	Drift level (drifted)
tenure	numeric	1.914	0.007	significant
MonthlyCharges	numeric	1.309	0.012	significant
charge_per_service	numeric	1.081	0.010	significant
PaymentMethod	categorical	0.320	0.000	significant
Contract	categorical	0.217	0.000	moderate
InternetService, total_services, TotalCharges, Dependents, gender	—	≤ 0.089	≤ 0.001	stable

Score drift (the metric that actually drives a retraining decision): PSI = 0.612 (significant) on the drifted batch vs PSI = 0.012 (stable) on the control batch — the monitor stays quiet when nothing moved and fires clearly when it did.

---

Top SHAP Features

Rank	Feature	Direction
1	tenure	longer tenure → lower churn risk
2	is_long_contract	2-year contracts strongly reduce churn
3	charge_per_service	high cost per service → higher churn
4	Contract_Two year	confirms contract signal
5	MonthlyCharges	high charges → higher churn

---

Key EDA Findings

Segment	Churn Rate
Month-to-month contract	42.7%
First 12 months (new customers)	47.7%
Electronic check payment	45.3%
Fiber optic internet	41.9%
Two-year contract	2.8%
37+ months tenure	11.9%

---

Quickstart

1. Install dependencies

pip install -e ".[dev]"

2. Download data

kaggle datasets download -d blastchar/telco-customer-churn \
  -p data/raw --unzip

3. Run notebooks in order

notebooks/01_eda.ipynb
notebooks/02_feature_engineering.ipynb
notebooks/03_model_training.ipynb
notebooks/04_cost_sensitive.ipynb
notebooks/07_drift_monitoring.ipynb

4. Run tests

pytest tests/ -v --cov=src

5. Start the API

uvicorn src.serving.app:app --reload

---

API

GET /health

{"status": "ok", "model": "LightGBM churn pipeline"}

POST /predict

Request body (example — high-risk customer):

{
  "tenure": 2,
  "MonthlyCharges": 95.5,
  "TotalCharges": 191.0,
  "gender": "Female",
  "SeniorCitizen": 0,
  "Partner": "No",
  "Dependents": "No",
  "PhoneService": "Yes",
  "MultipleLines": "No",
  "InternetService": "Fiber optic",
  "OnlineSecurity": "No",
  "OnlineBackup": "No",
  "DeviceProtection": "No",
  "TechSupport": "No",
  "StreamingTV": "No",
  "StreamingMovies": "No",
  "Contract": "Month-to-month",
  "PaperlessBilling": "Yes",
  "PaymentMethod": "Electronic check"
}

Response:

{
  "churn_probability": 0.9102,
  "churn_prediction": true,
  "risk_tier": "high"
}

risk_tier values: "low" (< 0.4) · "medium" (0.4–0.7) · "high" (≥ 0.7)

---

Docker

docker build -t churn-api .
docker run -p 8000:8000 churn-api

---

Run with Docker

Interactive API docs available at http://localhost:8000/docs after starting the server.

---

Technical Notes

• No data leakage: all transformers are fit on training data only and applied to test data via sklearn Pipeline
• Imbalanced classes: handled with scale_pos_weight in both XGBoost and LightGBM
• Cost-sensitive learning: churners assigned sample weight = FN/FP cost ratio (10x) during retraining
• Threshold optimisation: decision boundary set by minimising total business cost, not F1
• Explainability: SHAP TreeExplainer provides both global feature importance and per-customer waterfall explanations
• Experiment tracking: all runs logged to MLflow (metrics, params, model artifacts, figures)
• Code quality: ruff + black linting enforced via pyproject.toml; pytest with coverage via pytest-cov
• Drift monitoring: PSI (quantile-binned for numeric, proportion-based for categorical) and KS implemented from scratch in src/monitoring/drift.py, verified against scipy.stats.ks_2samp to 1e-9