MLOps Project - ML Inference API with Monitoring & Alerting

A complete MLOps solution featuring automated ML model training, inference API, comprehensive monitoring with Prometheus/Grafana, and AWS deployment with Infrastructure as Code.

---

📸 Screenshots

Grafana Dashboard

Prometheus Monitoring

Prometheus Targets

API Endpoints

Health Endpoint	Stats Endpoint

Metrics Endpoint

---

📋 Table of Contents

• Screenshots
• Project Overview
• Features & Rubric
• Architecture
• Live Deployment URLs
• Quick Start
• Project Structure
• API Endpoints
• Prometheus Metrics
• Alerts Configuration
• Grafana Dashboard
• AWS Deployment
• Slack Integration
• Configuration
• Testing
• Troubleshooting
• Security Notes

---

🎯 Project Overview

This MLOps project implements a complete machine learning pipeline with:

• Data Ingestion: Fetching records from a data lake endpoint with error handling
• ML Model Training: RandomForest/GradientBoosting classifier with auto-retraining
• Inference API: FastAPI-based prediction service with /predict and /metrics endpoints
• Monitoring Stack: Prometheus + Grafana + Alertmanager with Docker Compose
• Alerting: Slack notifications for critical events (503 errors, drift, accuracy drops)
• AWS Deployment: EC2 deployment with Terraform IaC

---

✨ Features & Rubric

Core Features (100 points)

#	Feature	Points	Status	Description
1	Data Ingestion	15 pts	✅ Complete	Fetches from /records endpoint, handles 503 errors
2	ML Model Training	15 pts	✅ Complete	RandomForest classifier, trains until accuracy ≥ 0.8
3	Auto-Retraining	10 pts	✅ Complete	Automatic retrain when accuracy drops below threshold
4	AWS Deployment	15 pts	✅ Complete	Deployed on EC2 with public endpoints
5	Prometheus Metrics	10 pts	✅ Complete	All required metrics exposed at /metrics
6	Prom + Grafana + Alertmanager	15 pts	✅ Complete	Full monitoring stack with Docker Compose
7	Alerts	10 pts	✅ Complete	Slack alerts for all required events
8	Docker Compose	5 pts	✅ Complete	Complete orchestration of all services
9	Documentation	5 pts	✅ Complete	Comprehensive README and inline docs

Bonus Features

Feature	Status	Description
Grafana Dashboard	✅ Complete	Pre-configured MLOps dashboard with all metrics
CI/CD Pipeline	✅ Complete	GitHub Actions for Docker build and deployment
Infrastructure as Code	✅ Complete	Terraform scripts for AWS provisioning

---

🏗️ Architecture

%%{init: {'theme':'dark', 'themeVariables': {'primaryColor':'#1e3a8a','primaryTextColor':'#ffffff','primaryBorderColor':'#3b82f6','lineColor':'#60a5fa','secondaryColor':'#1e40af','tertiaryColor':'#1e293b','background':'#0f172a','mainBkg':'#1e293b','secondBkg':'#334155','textColor':'#e2e8f0','border1':'#475569','border2':'#64748b','arrowheadColor':'#60a5fa','clusterBkg':'#1e293b','clusterBorder':'#3b82f6','defaultLinkColor':'#60a5fa','titleColor':'#ffffff','edgeLabelBackground':'#1e293b','actorBorder':'#3b82f6','actorBkg':'#1e40af','actorTextColor':'#ffffff','actorLineColor':'#60a5fa','signalColor':'#60a5fa','signalTextColor':'#ffffff','labelBoxBkgColor':'#1e40af','labelBoxBorderColor':'#3b82f6','labelTextColor':'#ffffff','loopTextColor':'#ffffff','noteBkgColor':'#1e40af','noteBorderColor':'#3b82f6','noteTextColor':'#ffffff','activationBorderColor':'#60a5fa','activationBkgColor':'#1e3a8a','sequenceNumberColor':'#ffffff','sectionBkgColor':'#1e293b','altBkgColor':'#1e40af','sectionBorderColor':'#3b82f6','altBorderColor':'#3b82f6','excludeBkgColor':'#0f172a','excludeBorderColor':'#475569'}}}%%

graph TB
    subgraph External["🌐 External Services"]
        DataLake["📊 Data Lake<br/>149.40.228.124:6500<br/>/records endpoint"]
        Slack["💬 Slack Workspace<br/>#all-mlosping<br/>Webhook Integration"]
        Users["👥 API Users<br/>HTTP Clients"]
    end

    subgraph AWS["☁️ AWS EC2 Instance (13.53.39.56)"]
        subgraph Docker["🐳 Docker Compose Orchestration"]
            subgraph MLService["🤖 ML Inference Service (FastAPI :5000)"]
                Ingestion["📥 Data Ingestion<br/>• Fetch records every 15s<br/>• Handle 503 errors<br/>• Retry logic (3 attempts)"]
                Training["🎯 Model Training<br/>• RandomForest/GradientBoosting<br/>• Auto-retrain when accuracy < 0.8<br/>• Min 50 records required"]
                DriftDet["🔍 Drift Detection<br/>• KS Test every 100 records<br/>• Distribution shift detection<br/>• Feature tracking"]
                FeatureTrack["📋 Feature Tracking<br/>• Schema validation<br/>• Feature addition/removal<br/>• Change detection"]
                PredictAPI["🔮 Prediction API<br/>• POST /predict<br/>• GET /health<br/>• GET /stats"]
                MetricsExp["📈 Metrics Exporter<br/>• /metrics endpoint<br/>• Prometheus format<br/>• 8+ custom metrics"]
            end

            subgraph Monitoring["📊 Monitoring Stack"]
                Prometheus["⚡ Prometheus :9090<br/>• Scrape metrics every 15s<br/>• Alert rule evaluation<br/>• Time-series storage"]
                Grafana["📉 Grafana :3000<br/>• Pre-configured dashboards<br/>• Real-time visualization<br/>• MLOps metrics panels"]
                AlertManager["🚨 Alertmanager :9093<br/>• Alert routing<br/>• Deduplication<br/>• Slack integration"]
            end

            subgraph Exporters["📡 System Exporters"]
                NodeExp["🖥️ Node Exporter :9100<br/>• CPU, Memory, Disk<br/>• System metrics<br/>• Infrastructure health"]
                BlackboxExp["🔎 Blackbox Exporter :9115<br/>• HTTP endpoint probing<br/>• Data Lake availability<br/>• ML Service health checks"]
            end
        end

        ModelStorage["💾 Model Storage<br/>Docker Volume<br/>• trained_model.joblib<br/>• scaler.joblib<br/>• data_schema.json<br/>• feature_history.json"]
    end

    subgraph IaC["🛠️ Infrastructure as Code"]
        Terraform["📝 Terraform<br/>• EC2 provisioning<br/>• Security groups<br/>• Auto-scaling config"]
    end

    %% Data Flow
    DataLake -->|HTTP GET /records<br/>Every 15s| Ingestion
    Ingestion -->|Records + Schema| Training
    Ingestion -->|Records| DriftDet
    Ingestion -->|Schema| FeatureTrack
    
    Training -->|Save Model| ModelStorage
    ModelStorage -->|Load Model| PredictAPI
    
    DriftDet -->|Drift Events| MetricsExp
    FeatureTrack -->|Feature Changes| MetricsExp
    Training -->|Accuracy Metrics| MetricsExp
    Ingestion -->|Ingestion Stats| MetricsExp
    PredictAPI -->|Request Metrics| MetricsExp
    
    MetricsExp -->|Scrape /metrics<br/>Every 10s| Prometheus
    NodeExp -->|Scrape /metrics<br/>Every 15s| Prometheus
    BlackboxExp -->|Probe Metrics<br/>Every 15s| Prometheus
    
    Prometheus -->|Query Metrics| Grafana
    Prometheus -->|Evaluate Alerts| AlertManager
    AlertManager -->|Send Notifications| Slack
    
    BlackboxExp -->|Probe HTTP| DataLake
    BlackboxExp -->|Probe /health| PredictAPI
    
    Users -->|POST /predict<br/>GET /health| PredictAPI
    
    Terraform -->|Provision| AWS

    %% Styling
    classDef external fill:#1e3a8a,stroke:#3b82f6,stroke-width:2px,color:#ffffff
    classDef mlService fill:#1e40af,stroke:#60a5fa,stroke-width:2px,color:#ffffff
    classDef monitoring fill:#1e293b,stroke:#3b82f6,stroke-width:2px,color:#ffffff
    classDef exporter fill:#334155,stroke:#60a5fa,stroke-width:2px,color:#ffffff
    classDef storage fill:#1e40af,stroke:#3b82f6,stroke-width:2px,color:#ffffff
    classDef iac fill:#0f172a,stroke:#475569,stroke-width:2px,color:#ffffff

    class DataLake,Slack,Users external
    class Ingestion,Training,DriftDet,FeatureTrack,PredictAPI,MetricsExp mlService
    class Prometheus,Grafana,AlertManager monitoring
    class NodeExp,BlackboxExp exporter
    class ModelStorage storage
    class Terraform iac

Loading

Data Pipeline Flow

%%{init: {'theme':'dark', 'themeVariables': {'primaryColor':'#1e3a8a','primaryTextColor':'#ffffff','primaryBorderColor':'#3b82f6','lineColor':'#60a5fa','secondaryColor':'#1e40af','tertiaryColor':'#1e293b','background':'#0f172a','mainBkg':'#1e293b','secondBkg':'#334155','textColor':'#e2e8f0','border1':'#475569','border2':'#64748b','arrowheadColor':'#60a5fa','clusterBkg':'#1e293b','clusterBorder':'#3b82f6','defaultLinkColor':'#60a5fa','titleColor':'#ffffff','edgeLabelBackground':'#1e293b','actorBorder':'#3b82f6','actorBkg':'#1e40af','actorTextColor':'#ffffff','actorLineColor':'#60a5fa','signalColor':'#60a5fa','signalTextColor':'#ffffff','labelBoxBkgColor':'#1e40af','labelBoxBorderColor':'#3b82f6','labelTextColor':'#ffffff','loopTextColor':'#ffffff','noteBkgColor':'#1e40af','noteBorderColor':'#3b82f6','noteTextColor':'#ffffff','activationBorderColor':'#60a5fa','activationBkgColor':'#1e3a8a','sequenceNumberColor':'#ffffff','sectionBkgColor':'#1e293b','altBkgColor':'#1e40af','sectionBorderColor':'#3b82f6','altBorderColor':'#3b82f6','excludeBkgColor':'#0f172a','excludeBorderColor':'#475569'}}}%%

flowchart LR
    Start([🚀 Pipeline Start]) --> DataIngest[📥 Data Ingestion<br/>Fetch from Data Lake<br/>Handle 503 errors<br/>Retry with backoff]
    
    DataIngest -->|Success| SchemaCheck[📋 Schema Validation<br/>Extract feature schema<br/>Track changes]
    DataIngest -->|503 Error| Alert1[🚨 Alert: Data Lake Unavailable<br/>Increment metric<br/>Notify Slack]
    Alert1 --> DataIngest
    
    SchemaCheck -->|New Features| Alert2[🚨 Alert: Feature Added<br/>Update tracking]
    SchemaCheck -->|Removed Features| Alert3[🚨 Alert: Feature Removed<br/>Flag for retraining]
    SchemaCheck -->|No Changes| Accumulate[📊 Data Accumulation<br/>Store records<br/>Count processed]
    
    Alert2 --> Accumulate
    Alert3 --> Accumulate
    
    Accumulate --> DriftCheck{🔍 Drift Check<br/>Every 100 records<br/>KS Test}
    
    DriftCheck -->|Drift Detected| Alert4[🚨 Alert: Distribution Drift<br/>Notify team]
    DriftCheck -->|No Drift| ModelCheck{🎯 Model Check<br/>Accuracy < 0.8?<br/>Records >= 50?}
    Alert4 --> ModelCheck
    
    ModelCheck -->|Need Training| Train[🤖 Model Training<br/>RandomForest/GradientBoosting<br/>Train until accuracy ≥ 0.8<br/>Max 100 iterations]
    ModelCheck -->|Model OK| Serve[🔮 Model Serving<br/>Ready for predictions]
    
    Train -->|Success| SaveModel[💾 Save Model<br/>trained_model.joblib<br/>scaler.joblib<br/>metadata.json]
    Train -->|Low Accuracy| Alert5[🚨 Alert: Accuracy Drop<br/>Below threshold]
    Alert5 --> Train
    
    SaveModel --> Serve
    
    Serve --> Predict[📡 Prediction API<br/>POST /predict<br/>Measure latency<br/>Track requests]
    
    Predict -->|High Latency| Alert6[🚨 Alert: Response Delay<br/>P95 > 2s]
    Predict -->|Normal| Metrics[📈 Export Metrics<br/>Prometheus format<br/>8+ custom metrics]
    
    Alert6 --> Metrics
    
    Metrics --> Prometheus[⚡ Prometheus<br/>Scrape & Store<br/>Evaluate alerts]
    
    Prometheus --> Grafana[📉 Grafana<br/>Visualize metrics<br/>Real-time dashboards]
    Prometheus --> AlertMgr[🚨 Alertmanager<br/>Route alerts<br/>Deduplicate]
    
    AlertMgr --> Slack[💬 Slack Notifications<br/>#all-mlosping<br/>Rich alerts]
    
    Serve -->|Continuous| DataIngest
    
    classDef ingest fill:#1e40af,stroke:#60a5fa,stroke-width:2px,color:#ffffff
    classDef process fill:#1e293b,stroke:#3b82f6,stroke-width:2px,color:#ffffff
    classDef train fill:#1e3a8a,stroke:#3b82f6,stroke-width:2px,color:#ffffff
    classDef serve fill:#334155,stroke:#60a5fa,stroke-width:2px,color:#ffffff
    classDef monitor fill:#0f172a,stroke:#475569,stroke-width:2px,color:#ffffff
    classDef alert fill:#7f1d1d,stroke:#ef4444,stroke-width:2px,color:#ffffff
    
    class DataIngest,Accumulate ingest
    class SchemaCheck,DriftCheck,ModelCheck process
    class Train,SaveModel train
    class Serve,Predict,Metrics serve
    class Prometheus,Grafana,AlertMgr,Slack monitor
    class Alert1,Alert2,Alert3,Alert4,Alert5,Alert6 alert

Loading

---

🌐 Live Deployment URLs

Production (AWS EC2)

Service	URL	Description
ML API - Health	http://13.53.39.56:5000/health	Health check endpoint
ML API - Predict	http://13.53.39.56:5000/predict	Prediction endpoint (POST)
ML API - Metrics	http://13.53.39.56:5000/metrics	Prometheus metrics
Grafana	http://13.53.39.56:3000	Dashboards (admin/admin123)
Prometheus	http://13.53.39.56:9090	Metrics & Queries
Prometheus Targets	http://13.53.39.56:9090/targets	Scrape target status
Alertmanager	http://13.53.39.56:9093	Alert management UI
Node Exporter	http://13.53.39.56:9100/metrics	System metrics
Blackbox Exporter	http://13.53.39.56:9115	Probe metrics

External Data Lake

Service	URL
Data Lake Records	http://149.40.228.124:6500/records

---

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Python 3.11+ (for local development)
• AWS CLI (for AWS deployment)
• Terraform (for infrastructure provisioning)

Local Development

Step 1: Clone and Setup

# Clone the repository
git clone <repository-url>
cd mlops-inference-pipeline

# Create virtual environment
python -m venv venv

# Activate (Windows PowerShell)
.\venv\Scripts\Activate.ps1

# Activate (Linux/Mac)
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Step 2: Configure Environment Variables

# Copy example file
cp credentials.env.example credentials.env

# Edit credentials.env with your values:
# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
# SLACK_CHANNEL=#your-channel

Step 3: Start with Docker Compose

# Set Slack webhook (PowerShell)
$env:SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

# Start all services
docker-compose up -d --build

# View logs
docker-compose logs -f

Step 4: Verify Deployment

# Health check
curl http://localhost:5000/health

# Make a prediction
curl -X POST http://localhost:5000/predict \
  -H "Content-Type: application/json" \
  -d '{"features": [1.5, 2.0, 3.5]}'

# Check metrics
curl http://localhost:5000/metrics

---

📁 Project Structure

mlops-inference-pipeline/
├── src/                              # Source code
│   ├── __init__.py
│   ├── config.py                     # Configuration (environment-based)
│   ├── data_ingestion.py             # Data lake ingestion with 503 handling
│   ├── drift_detection.py            # Distribution drift detection (KS test)
│   ├── feature_tracking.py           # Feature schema change tracking
│   ├── metrics.py                    # Prometheus metrics definitions
│   ├── ml_service.py                 # FastAPI application
│   ├── model_training.py             # ML model training & prediction
│   └── slack_alerts.py               # Slack notification service
│
├── prometheus/                       # Prometheus configuration
│   ├── prometheus.yml                # Main config with scrape targets
│   ├── alert_rules.yml               # Alert rule definitions
│   └── recording_rules.yml           # Recording rules for aggregations
│
├── alertmanager/                     # Alertmanager configuration
│   ├── alertmanager.yml              # Slack integration config
│   └── templates/
│       └── slack.tmpl                # Slack message template
│
├── grafana/                          # Grafana configuration
│   ├── dashboards/
│   │   ├── mlops_dashboard.json      # Main MLOps dashboard
│   │   └── mlops_dashboard_import.json # Import-ready version
│   └── provisioning/
│       ├── dashboards/
│       └── datasources/
│
├── aws/                              # AWS deployment files
│   ├── terraform/
│   │   ├── main.tf                   # EC2, Security Groups, etc.
│   │   ├── variables.tf              # Variable definitions
│   │   ├── outputs.tf                # Output values
│   │   ├── user_data.sh              # EC2 bootstrap script
│   │   └── terraform.tfvars.example  # Example variables
│   └── scripts/
│       ├── deploy.sh                 # Automated deployment script
│       ├── deploy-manual.sh          # Manual deployment steps
│       └── deploy.ps1                # PowerShell deployment
│
├── scripts/                          # Setup scripts
│   ├── setup.sh                      # Linux/Mac setup
│   └── setup.ps1                     # Windows setup
│
├── .github/workflows/                # CI/CD pipelines
│   ├── ci-cd.yml                     # Main CI/CD workflow
│   └── docker-build.yml              # Docker build workflow
│
├── models/                           # Saved ML models (gitignored)
│
├── docker-compose.yml                # Docker orchestration
├── Dockerfile                        # ML service container
├── requirements.txt                  # Python dependencies
├── credentials.env.example           # Environment template
├── ec2_setup.sh                      # EC2 initialization script
├── deploy_now.py                     # Python deployment script
└── README.md                         # This file

---

🔌 API Endpoints

Health & Status

Endpoint	Method	Description	Response
/health	GET	Health check	{"status": "healthy", ...}
/metrics	GET	Prometheus metrics	Text format metrics
/model/info	GET	Model information	{"model_type": "...", "accuracy": ...}
/stats	GET	Current statistics	{"records_processed": ..., ...}

Prediction

Endpoint: POST /predict

Request:

{
  "features": [1.5, 2.0, 3.5, 4.0]
}

Response:

{
  "prediction": 1,
  "confidence": 0.87,
  "model_version": "1.0.0"
}

Example (PowerShell):

Invoke-RestMethod -Method POST -Uri "http://13.53.39.56:5000/predict" `
  -ContentType "application/json" `
  -Body '{"features": [1.5, 2.0, 3.5]}'

Example (curl):

curl -X POST http://13.53.39.56:5000/predict \
  -H "Content-Type: application/json" \
  -d '{"features": [1.5, 2.0, 3.5]}'

Manual Operations

Endpoint	Method	Description
/model/retrain	POST	Trigger manual model retraining
/ingest	POST	Manually trigger data ingestion

Demo Endpoints

Endpoint	Method	Description
/demo/simulate-events	POST	Trigger all demo events (for testing)
/demo/trigger-datalake-error	POST	Simulate Data Lake 503 error
/demo/trigger-drift	POST	Simulate distribution drift
/demo/trigger-feature-change	POST	Simulate feature schema changes

API Endpoint Screenshots

Health Endpoint (/health):

Stats Endpoint (/stats):

---

📊 Prometheus Metrics

All metrics are exposed at /metrics endpoint.

Required Metrics

Metric	Type	Description
model_accuracy	Gauge	Current model accuracy (0-1)
records_processed_total	Counter	Total records processed from data lake
retrain_count_total	Counter	Number of model retraining events
distribution_drift_detected	Counter	Distribution drift detection events
feature_added	Counter	New features detected in schema
feature_removed	Counter	Features removed from schema
datalake_unavailable	Counter	Data lake 503 error count
response_delay_seconds	Histogram	API response latency

Additional Metrics

Metric	Type	Description
ingestion_errors_total	Counter	Data ingestion error count
prediction_requests_total	Counter	Total prediction requests
model_training_duration_seconds	Histogram	Model training time

Example Prometheus Queries

# Current model accuracy
model_accuracy

# Prediction rate per minute
rate(prediction_requests_total[5m]) * 60

# 95th percentile response time
histogram_quantile(0.95, rate(response_delay_seconds_bucket[5m]))

# Data lake availability (1 = available)
1 - (increase(datalake_unavailable[5m]) > 0)

Prometheus Screenshots

Prometheus UI:

Prometheus Targets:

Metrics Endpoint Output:

---

🚨 Alerts Configuration

Alert Rules

All alerts are defined in prometheus/alert_rules.yml:

Alert	Severity	Trigger Condition	Description
DataLakeUnavailable	Critical	datalake_unavailable > 0	Data lake returned 503
FeatureAdded	Warning	feature_added > 0	New feature detected in schema
FeatureRemoved	Warning	feature_removed > 0	Feature removed from schema
DistributionDriftDetected	Warning	distribution_drift_detected > 0	KS test detected distribution shift
ModelAccuracyLow	Critical	model_accuracy < 0.8	Model accuracy below threshold
HighResponseDelay	Warning	p95 latency > 2s	High API response time
ModelRetraining	Info	retrain_count_total increased	Model is being retrained

Alert Flow

Event Detected → Prometheus Alert → Alertmanager → Slack Channel
                                                    (#all-mlosping)

---

📈 Grafana Dashboard

Accessing Grafana

1. URL: http://13.53.39.56:3000
2. Login: admin / admin123

Adding Prometheus Data Source

1. Go to ☰ → Connections → Data sources
2. Click "Add data source"
3. Select Prometheus
4. Set URL: http://prometheus:9090
5. Click "Save & Test"

Importing the Dashboard

1. Go to ☰ → Dashboards
2. Click "New" → "Import"
3. Upload grafana/dashboards/mlops_dashboard_import.json
4. Select Prometheus as the data source
5. Click "Import"

Dashboard Panels

Panel	Type	Metric
Model Accuracy	Gauge	model_accuracy
Records Processed	Stat	records_processed_total
Retrain Count	Stat	retrain_count_total
Drift Events	Stat	distribution_drift_detected
Data Lake Status	Stat	datalake_unavailable
Features Added/Removed	Stat	feature_added, feature_removed
Response Latency	Time Series	response_delay_seconds percentiles
Accuracy History	Time Series	model_accuracy over time

Dashboard Screenshot

---

☁️ AWS Deployment

EC2 Instance Details

Property	Value
Public IP	13.53.39.56
Instance Type	t2.micro / t3.micro
OS	Ubuntu 22.04 LTS
SSH User	ubuntu
Key Pair	your-key.pem

Security Group Ports

Port	Service	Source
22	SSH	Your IP
3000	Grafana	0.0.0.0/0
5000	ML API	0.0.0.0/0
9090	Prometheus	0.0.0.0/0
9093	Alertmanager	0.0.0.0/0
9100	Node Exporter	0.0.0.0/0

Deployment with Terraform

cd aws/terraform

# Initialize Terraform
terraform init

# Copy and configure variables
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your AWS credentials

# Plan deployment
terraform plan

# Apply deployment
terraform apply

Manual EC2 Deployment

# SSH into EC2
ssh -i "your-key.pem" ubuntu@13.53.39.56

# Clone repository
git clone <repository-url> /home/ubuntu/mlops_project
cd /home/ubuntu/mlops_project

# Run setup script
chmod +x ec2_setup.sh
sudo ./ec2_setup.sh

EC2 Setup Script (ec2_setup.sh)

The setup script:

1. Installs Docker and Docker Compose
2. Configures permissions
3. Sets environment variables
4. Starts all services with Docker Compose

---

💬 Slack Integration

Slack Configuration

Setting	Value
App Name	MLOpsAlerts
Workspace	MLOsping
Channel	#all-mlosping

Setting Up Slack Webhook

1. Go to https://api.slack.com/apps
2. Click "Create New App" → "From scratch"
3. Name: MLOpsAlerts, Workspace: Your workspace
4. Go to "Incoming Webhooks" → Enable
5. Click "Add New Webhook to Workspace"
6. Select channel #all-mlosping
7. Copy the webhook URL

Configuring Alerts

Update alertmanager/alertmanager.yml:

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#all-mlosping'
        send_resolved: true

Test Slack Integration

curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test alert from MLOps!"}' \
  YOUR_SLACK_WEBHOOK_URL

---

⚙️ Configuration

Environment Variables

Variable	Default	Description
SLACK_WEBHOOK_URL	-	Slack incoming webhook URL (required)
SLACK_CHANNEL	#mlops-alerts	Slack channel for alerts
DATA_LAKE_URL	http://149.40.228.124:6500	Data lake base URL
API_PORT	5000	ML service port
MIN_ACCURACY_THRESHOLD	0.8	Minimum acceptable model accuracy
DRIFT_THRESHOLD	0.05	KS test p-value threshold
DRIFT_CHECK_INTERVAL	100	Check drift every N records
GRAFANA_ADMIN_PASSWORD	admin123	Grafana admin password

Docker Compose Services

Service	Image	Port
ml-service	Custom (Dockerfile)	5000
prometheus	prom/prometheus	9090
grafana	grafana/grafana	3000
alertmanager	prom/alertmanager	9093
node-exporter	prom/node-exporter	9100
blackbox	prom/blackbox-exporter	9115

---

🧪 Testing

Quick Test Commands

# Health check
Invoke-RestMethod http://13.53.39.56:5000/health

# Make prediction
Invoke-RestMethod -Method POST -Uri "http://13.53.39.56:5000/predict" `
  -ContentType "application/json" -Body '{"features": [1.5, 2.0]}'

# Check metrics
(Invoke-WebRequest http://13.53.39.56:5000/metrics).Content

# Check Prometheus targets
Invoke-RestMethod http://13.53.39.56:9090/api/v1/targets

Test Scripts

# Run all tests
python run_all_tests.py

# Quick endpoint test
python quick_test.py

# Test specific endpoints
python test_endpoints.py

Expected Test Results

Test	Expected Result
Health Check	{"status": "healthy"}
Prediction	{"prediction": 0/1, "confidence": 0.xx}
Metrics	Text containing model_accuracy, records_processed_total
Prometheus	Status 200, targets UP
Grafana	Login page loads

---

🔧 Troubleshooting

Common Issues

Docker not running

# Check Docker status
docker --version
docker-compose --version

# Start Docker service (Linux)
sudo systemctl start docker

Prometheus permission denied

# Fix permissions on EC2
sudo chmod -R 755 prometheus/ alertmanager/
sudo chown -R 65534:65534 prometheus/
sudo docker-compose restart prometheus alertmanager

Port not accessible

1. Check AWS Security Group has the port open
2. Check EC2 firewall: sudo ufw status
3. Verify container is running: docker-compose ps

Slack alerts not working

# Test webhook manually
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test"}' YOUR_WEBHOOK_URL

# Check alertmanager logs
docker-compose logs alertmanager

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f ml-service
docker-compose logs -f prometheus
docker-compose logs -f alertmanager
docker-compose logs -f grafana

Restart Services

# Restart all
docker-compose restart

# Restart specific service
docker-compose restart ml-service

# Full rebuild
docker-compose down
docker-compose up -d --build

---

🔒 Security Notes

• NEVER commit credentials.env to version control
• .gitignore excludes all credential files
• Use environment variables for sensitive data
• Rotate Slack webhooks periodically
• Restrict AWS Security Group to necessary IPs in production
• Use HTTPS in production (configure with nginx/traefik)

---

📝 Project Information

Field	Value
Course	MLOps
Semester	Spring 2025
Author	Nouman Hafeez
Repository	mlops-inference-pipeline

---

📄 License

MIT License - See LICENSE file for details.