💳 Credit Card Fraud Classification with Apache Spark

This repository contains a comprehensive implementation of Logistic Regression for the classification of fraudulent credit card transactions. The project demonstrates three distinct levels of implementation using Apache Spark to showcase scalability and distributed computing paradigms.

📌 Project Overview

The objective is to assign labels (Fraud vs. Genuine) to transaction instances. We focus on building a scalable classifier that can process large-scale data in parallel while ensuring robust performance.

🛠 Tech Stack

• Language: Python (PySpark) or Scala.
• Framework: Apache Spark (Structured API & Core RDD).
• Core Concepts: Catalyst Optimizer, Tungsten Execution Engine, and RDD Lineage.

📂 Dataset

• Name: Credit Card Fraud Detection Dataset.
• Source: Kaggle - Credit Card Fraud.
• Task: Binary Classification using Logistic Regression.

🚀 Implementation Approaches

1. Structured API Implementation (High-Level)

• Module: spark.ml.
• Methodology: Utilizing VectorAssembler for feature engineering and the LogisticRegression estimator.
• Metrics: Evaluated based on Accuracy, AUC, Precision, and Recall.

2. MLlib RDD-Based Implementation

• Module: spark.mllib.
• Methodology: Data is converted into an RDD of LabeledPoint objects. Models are trained using built-in functions like LogisticRegressionWithSGD or LogisticRegressionWithLBFGS.
• Optimization: Fine-tuning via iteration counts and step-sizes (learning rates).

3. Low-Level Operations (Manual Implementation)

• Module: Spark Core (Fundamental RDD Transformations).
• Methodology: Manual decomposition of the Logistic Regression algorithm into scalable operations.
• Key Components: * Custom Dot Product calculation.
  • Sigmoid Function implementation.
  • Gradient Descent update mechanism using map and reduce.

📊 Grading Criteria

The implementation adheres to the following course requirements:

• High-level Structured API: 1.25 points.
• MLlib RDD-based: 2.0 points.
• Low-level Operations: 2.0 points.
• Note: Bonus points awarded for Scala-based implementations.

📁 Repository Structure

<RepresentativeID>/
├── docs/
│   ├── Report.pdf          # Approach, detailed explanations, and screenshots
│   └── drive_link.txt      # Link to predicted CSV results
├── src/
│   └── Classification/
│       ├── Structured_API/ # Folder code for Structured API Implementation
│       ├── MLlib_RDD_Based/# Folder code for MLlib RDD-Based Implementation
│       └── Low_Level/      # Folder code for Low-Level Operations Implementation
└── README.md               # Execution instructions