Apache Hadoop 3.4.0 Documentation on Debian

Table of Contents

1. Prerequisites
2. Lab 23: Creating HDFS
3. Lab 24: Installing Hadoop Framework
4. Lab 25: Query Processing in HDFS
5. Managing Files and Running Jobs
6. Cheat Sheet

---

Prerequisites

1. Java Installation

   • Ensure Java 8 or higher is installed:

     java --version

   • Install Java if needed:

     sudo apt update
     sudo apt install openjdk-17-jdk -y

2. SSH Configuration

   • Install SSH:

     sudo apt install openssh-server openssh-client -y

   • Set up passwordless SSH:

     ssh-keygen -t rsa -P ""
     cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

---

Lab 23: Creating HDFS

Step 1: Download and Install Hadoop 3.4.0

1. Download Hadoop:

   wget https://downloads.apache.org/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz

2. Extract Hadoop:

   tar -xvzf hadoop-3.4.0.tar.gz
   sudo mv hadoop-3.4.0 /usr/local/hadoop

3. Set Hadoop Environment Variables: Edit .bashrc:

   nano ~/.bashrc

   Add:

   export HADOOP_HOME=/usr/local/hadoop
   export HADOOP_INSTALL=$HADOOP_HOME
   export HADOOP_MAPRED_HOME=$HADOOP_HOME
   export HADOOP_COMMON_HOME=$HADOOP_HOME
   export HADOOP_HDFS_HOME=$HADOOP_HOME
   export YARN_HOME=$HADOOP_HOME
   export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

   Apply changes:

   source ~/.bashrc

Step 2: Configure Hadoop

1. Set JAVA_HOME: Find Java path:

   readlink -f $(which java)

   Update hadoop-env.sh:

   nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

   Add:

   export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64

2. Configure HDFS:

   • Edit core-site.xml:

     nano $HADOOP_HOME/etc/hadoop/core-site.xml

     Add:

     <configuration>
       <property>
         <name>fs.defaultFS</name>
         <value>hdfs://localhost:9000</value>
       </property>
     </configuration>

   • Edit hdfs-site.xml:

     nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

     Add:

     <configuration>
       <property>
         <name>dfs.replication</name>
         <value>1</value>
       </property>
       <property>
         <name>dfs.namenode.name.dir</name>
         <value>file:///usr/local/hadoop_data/hdfs/namenode</value>
       </property>
       <property>
         <name>dfs.datanode.data.dir</name>
         <value>file:///usr/local/hadoop_data/hdfs/datanode</value>
       </property>
     </configuration>

   • Create HDFS Directories:

     sudo mkdir -p /usr/local/hadoop_data/hdfs/namenode
     sudo mkdir -p /usr/local/hadoop_data/hdfs/datanode
     sudo chown -R $USER:$USER /usr/local/hadoop_data

   • Format the NameNode:

     hdfs namenode -format

Step 3: Start HDFS

1. Start HDFS:

   start-dfs.sh

2. Verify Services: Check that NameNode and DataNode are running:

   jps

3. Access HDFS Web UI: Navigate to:

   http://localhost:9870/
   

---

Lab 24: Installing Hadoop Framework (YARN)

Step 1: Configure YARN

1. Edit yarn-site.xml:

   nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

   Add:

   <configuration>
     <property>
       <name>yarn.resourcemanager.hostname</name>
       <value>localhost</value>
     </property>
     <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
     </property>
   </configuration>

2. Start YARN:

   start-yarn.sh

3. Verify YARN Services: Check that ResourceManager and NodeManager are running:

   jps

---

Lab 25: Query Processing in HDFS (Using MapReduce)

Step 1: Write the MapReduce Program

1. Create WordCount Java Program:

   nano WordCount.java

   Insert the following code:

   import java.io.IOException;
   import org.apache.hadoop.conf.Configuration;
   import org.apache.hadoop.fs.Path;
   import org.apache.hadoop.io.IntWritable;
   import org.apache.hadoop.io.Text;
   import org.apache.hadoop.mapreduce.Job;
   import org.apache.hadoop.mapreduce.Mapper;
   import org.apache.hadoop.mapreduce.Reducer;
   import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
   import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
   
   public class WordCount {
       public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {
           private final static IntWritable one = new IntWritable(1);
           private Text word = new Text();
   
           public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
               String[] words = value.toString().split("\\s+");
               for (String w : words) {
                   word.set(w);
                   context.write(word, one);
               }
           }
       }
   
       public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
           private IntWritable result = new IntWritable();
   
           public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
               int sum = 0;
               for (IntWritable val : values) {
                   sum += val.get();
               }
               result.set(sum);
               context.write(key, result);
           }
       }
   
       public static void main(String[] args) throws Exception {
           Configuration conf = new Configuration();
           Job job = Job.getInstance(conf, "word count");
           job.setJarByClass(WordCount.class);
           job.setMapperClass(TokenizerMapper.class);
           job.setCombinerClass(IntSumReducer.class);
           job.setReducerClass(IntSumReducer.class);
           job.setOutputKeyClass(Text.class);
           job.setOutputValueClass(IntWritable.class);
           FileInputFormat.addInputPath(job, new Path(args[0]));
           FileOutputFormat.setOutputPath(job, new Path(args[1]));
           System.exit(job.waitForCompletion(true) ? 0 : 1);
       }
   }

Step 2: Compile and Run the Program

1. Compile the Program:

   hadoop com.sun.tools.javac.Main WordCount.java
   jar cf wordcount.jar WordCount*.class

2. Create an Input File:

   echo "Hello Hadoop Hello HDFS" > input.txt

3. Upload File to HDFS:

   hdfs dfs -mkdir /input
   hdfs dfs -put input.txt /input/input.txt

4. Run the MapReduce Job:

   hadoop jar wordcount.jar WordCount /input/input.txt /output

5. View the Output:

   hdfs dfs -cat /output/part-r-00000

Changing the Content of an Existing Input File in HDFS

1. Remove the Existing Input File from HDFS:

   hdfs dfs -rm /input/input.txt

2. Create a New File with Updated Content Locally:

   echo "New content for Hadoop processing" > new_input.txt

3. Upload the New Input File to HDFS:

   hdfs dfs -put new_input.txt /input/input.txt

4. **Run the Map

Reduce Job Again**:

hadoop jar wordcount.jar WordCount /input/input.txt /output

5. View the Output:

   hdfs dfs -cat /output/part-r-00000

---

Stopping Hadoop Services

1. Stop HDFS:

   stop-dfs.sh

2. Stop YARN:

   stop-yarn.sh

3. Verify Services are Stopped:

   jps

   No Hadoop-related processes should be listed.

---

This comprehensive guide should cover everything from setting up Hadoop to running MapReduce jobs and managing input/output files. If you need further assistance, feel free to ask!