Big Data Learning Project: MovieLens Analytics

This project is designed to help you learn the concepts of Big Data Analytics as outlined in your syllabus. It simulates a "Big Data" pipeline using Python, SQLite, and Spark to analyze a synthetic movie ratings dataset.

Modules Covered

Module I & II: Big Data, Hadoop, MapReduce

• Concept: Processing large datasets by breaking them down (Map) and aggregating results (Reduce).
• Implementation:
  • mapper.py: Parses input lines and emits movieId and rating.
  • reducer.py: Aggregates ratings for each movie to calculate the average.
  • run_mapreduce.sh: Simulates the Hadoop Streaming process using Unix pipes (|).

Module III & IV: Hive, SQL-like Querying

• Concept: Structuring data into tables and running SQL-like queries (HiveQL) for analysis.
• Implementation:
  • hive_simulation.py: Uses SQLite and Pandas to mimic a Hive Metastore.
  • Loads CSV data into tables.
  • Runs queries to find top movies and analyze user activity, simulating Hive's aggregation capabilities.

Module V: Spark, RDDs

• Concept: In-memory distributed processing for faster analytics using DataFrames and RDDs.
• Implementation:
  • spark_analytics.py: Uses PySpark (Apache Spark's Python API).
  • Demonstrates loading data into DataFrames.
  • Performs complex transformations like exploding genres and joining datasets.
  • Demonstrates RDD (Resilient Distributed Dataset) operations (map, reduceByKey) to count ratings per user.

How to Run

1. Generate Data:

   python3 big_data_project/generate_data.py

   This creates movies.csv, users.csv, and ratings.csv in big_data_project/data/.

2. Run MapReduce Simulation:

   chmod +x big_data_project/run_mapreduce.sh big_data_project/mapper.py big_data_project/reducer.py
   ./big_data_project/run_mapreduce.sh

3. Run Hive/SQL Simulation:

   python3 big_data_project/hive_simulation.py

4. Run Spark Analytics:

   python3 big_data_project/spark_analytics.py

Learning Exercises

To deepen your understanding, try the following:

• MapReduce: Modify mapper.py and reducer.py to count the number of ratings per user instead of average rating per movie.
• Hive: Add a new query in hive_simulation.py to find the genre with the highest number of movies.
• Spark: Modify spark_analytics.py to filter out movies with an average rating below 3.0 before showing the results.