Hadoop Development Environment with Eclipse

A usual Hadoop application needs a jar file for distributed nodes. It is very troublesome to repeat creating jar frequently in development…
So I’ve enable Eclipse to run Hadoop application(Map-Reduce Job) on its standalone mode. It make debug easier so it enable to Run as Debug.

I assume to finish Eclipse setup. See here if not yet.
By the way, the eclipse-plugin of Hadoop is not used. I tried it but couldn’t run it. One of Hadoop 0.20 can’t Run on Hadoop. One of Hadoop 0.21 can’t resister Hadoop’s Name Node furthermore.

Create a project for Hadoop

In this article, Hadoop version is 0.20.2 though the last one is 0.21.0 at a point of writing.
Because 0.21 has many toubles and no document for updating API and Mahout(0.4 and 0.5-SNAPSHOT) supports Hadoop 0.20.2 only.

Now, import Hadoop as an Eclipse project by the following. Need to do manually Hence Hadoop doesn’t have any Maven project.

• Create a new Java Project in Eclipce and name it “hadoop-0.20.2”.
• Download an archive file of Hadoop 0.20.2 (hadoop-0.20.2.tar.gz) and import it into the above project.
  Open Archive File Import dialog(File > Import > General > Archive File from Eclipse menu) and specify hadoop-0.20.2.tar.gz as archive file. Set the above project as Into Folder.
  
• Set Apache Ant library(ant.jar) into the library folder of the project.
  Download apache-ant-1.8.2-bin.zip from here and extract ant.jar from it and copy into $WORKSPACE/hadoop-0.20.2/hadoop-0.20.2/lib/ .
• Rewrite $WORKSPACE/hadoop-0.20.2/.classpath to add source folders and necessary libraries into the project.

  <?xml version="1.0" encoding="UTF-8"?>
  <classpath>
  	<classpathentry kind="src" path="hadoop-0.20.2/src/core"/>
  	<classpathentry kind="src" path="hadoop-0.20.2/src/mapred"/>
  	<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-1.6"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/commons-logging-1.0.4.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/xmlenc-0.52.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/commons-net-1.4.1.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/kfs-0.2.2.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/jets3t-0.6.1.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/jetty-6.1.14.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/jetty-util-6.1.14.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/commons-codec-1.3.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/log4j-1.2.15.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/commons-cli-1.2.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/ant.jar"/>
  	<classpathentry kind="lib" path="hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar"/>
  	<classpathentry kind="output" path="bin"/>
  </classpath>
  

• Refresh the project. Right click the hadoop-0.20.2 project and select “Refresh”.

Then importing Hadoop finishes. It is already running To build Hadoop in background.

Run Word Count Sample

We shall run a sample application on Hadoop in Eclipse for confirmation.

• Create a Java Project for a sample application and name appropriately(e.g. wordcount).
  Click Next in Create Java Project dialog (not Finish!) and set up references of projects and libraries.
  • Add the above hadoop-0.20.2 project at Projects tab.
  • Add hadoop-0.20.2/hadoop-0.20.2/lib/commons-cli-1.2.jar at Libraries tab.

  If you’ve clicked Finish, open a Java Build Path dialog from project Properties.

• Write classes of the sample application.

  // WordCountDriver.java
  import java.io.IOException;
  import java.util.Date;
  import java.util.Formatter;
  
  import org.apache.hadoop.conf.Configuration;
  import org.apache.hadoop.fs.Path;
  import org.apache.hadoop.io.IntWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Job;
  import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  import org.apache.hadoop.util.GenericOptionsParser;
  
  public class WordCountDriver {
  
      public static void main(String[] args) throws IOException,
              InterruptedException, ClassNotFoundException {
          Configuration conf = new Configuration();
          GenericOptionsParser parser = new GenericOptionsParser(conf, args);
          args = parser.getRemainingArgs();
  
          Job job = new Job(conf, "wordcount");
          job.setOutputKeyClass(Text.class);
          job.setOutputValueClass(IntWritable.class);
  
          job.setInputFormatClass(TextInputFormat.class);
          job.setOutputFormatClass(TextOutputFormat.class);
  
          Formatter formatter = new Formatter();
          String outpath = "Out"
                  + formatter.format("%1$tm%1$td%1$tH%1$tM%1$tS", new Date());
          FileInputFormat.setInputPaths(job, new Path("In"));
          FileOutputFormat.setOutputPath(job, new Path(outpath));
  
          job.setMapperClass(WordCountMapper.class);
          job.setReducerClass(WordCountReducer.class);
  
          System.out.println(job.waitForCompletion(true));
      }
  }
  

  // WordCountMapper.java
  import java.io.IOException;
  import java.util.StringTokenizer;
  
  import org.apache.hadoop.io.IntWritable;
  import org.apache.hadoop.io.LongWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Mapper;
  
  public class WordCountMapper extends
          Mapper<LongWritable, Text, Text, IntWritable> {
      private Text word = new Text();
      private final static IntWritable one = new IntWritable(1);
  
      protected void map(LongWritable key, Text value, Context context)
              throws IOException, InterruptedException {
          String line = value.toString();
          StringTokenizer tokenizer = new StringTokenizer(line);
          while (tokenizer.hasMoreTokens()) {
              word.set(tokenizer.nextToken());
              context.write(word, one);
          }
      }
  }
  

  // WordCountReducer.java
  import java.io.IOException;
  
  import org.apache.hadoop.mapreduce.Reducer;
  import org.apache.hadoop.io.IntWritable;
  import org.apache.hadoop.io.Text;
  
  public class WordCountReducer extends
          Reducer<Text, IntWritable, Text, IntWritable> {
      protected void reduce(Text key, Iterable<IntWritable> values,
              Context context) throws IOException, InterruptedException {
          int sum = 0;
          for (IntWritable value : values) {
              sum += value.get();
          }
          context.write(key, new IntWritable(sum));
      }
  }
  

• Create $WORKSPACE/wordcount/bin/log4j.properties as the following.

  log4j.rootLogger=INFO,console
  log4j.appender.console=org.apache.log4j.ConsoleAppender
  log4j.appender.console.target=System.err
  log4j.appender.console.layout=org.apache.log4j.PatternLayout
  log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
  

  This enables Hadoop logs to output Eclipse console.

• Create a sample text data into $WORKSPACE/wordcount/In .
• Run the application. Click Run > Run from Eclipse menu or press ctrl+F11.

It succeeds if the application outputs logs to Eclipse console as the following.

11/02/18 19:52:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
11/02/18 19:52:39 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
11/02/18 19:52:39 INFO input.FileInputFormat: Total input paths to process : 1
11/02/18 19:52:39 INFO mapred.JobClient: Running job: job_local_0001
11/02/18 19:52:39 INFO input.FileInputFormat: Total input paths to process : 1
11/02/18 19:52:39 INFO mapred.MapTask: io.sort.mb = 100
11/02/18 19:52:39 INFO mapred.MapTask: data buffer = 79691776/99614720
11/02/18 19:52:39 INFO mapred.MapTask: record buffer = 262144/327680
11/02/18 19:52:40 INFO mapred.MapTask: Starting flush of map output
11/02/18 19:52:40 INFO mapred.MapTask: Finished spill 0
11/02/18 19:52:40 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
11/02/18 19:52:40 INFO mapred.LocalJobRunner: 
11/02/18 19:52:40 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
11/02/18 19:52:40 INFO mapred.LocalJobRunner: 
11/02/18 19:52:40 INFO mapred.Merger: Merging 1 sorted segments
11/02/18 19:52:40 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 23563 bytes
11/02/18 19:52:40 INFO mapred.LocalJobRunner: 
11/02/18 19:52:40 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
11/02/18 19:52:40 INFO mapred.LocalJobRunner: 
11/02/18 19:52:40 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
11/02/18 19:52:40 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to Out0218195239
11/02/18 19:52:40 INFO mapred.LocalJobRunner: reduce > reduce
11/02/18 19:52:40 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
11/02/18 19:52:40 INFO mapred.JobClient:  map 100% reduce 100%
11/02/18 19:52:40 INFO mapred.JobClient: Job complete: job_local_0001
11/02/18 19:52:40 INFO mapred.JobClient: Counters: 12
11/02/18 19:52:40 INFO mapred.JobClient:   FileSystemCounters
11/02/18 19:52:40 INFO mapred.JobClient:     FILE_BYTES_READ=73091
11/02/18 19:52:40 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=110186
11/02/18 19:52:40 INFO mapred.JobClient:   Map-Reduce Framework
11/02/18 19:52:40 INFO mapred.JobClient:     Reduce input groups=956
11/02/18 19:52:40 INFO mapred.JobClient:     Combine output records=0
11/02/18 19:52:40 INFO mapred.JobClient:     Map input records=94
11/02/18 19:52:40 INFO mapred.JobClient:     Reduce shuffle bytes=0
11/02/18 19:52:40 INFO mapred.JobClient:     Reduce output records=956
11/02/18 19:52:40 INFO mapred.JobClient:     Spilled Records=4130
11/02/18 19:52:40 INFO mapred.JobClient:     Map output bytes=19431
11/02/18 19:52:40 INFO mapred.JobClient:     Combine input records=0
11/02/18 19:52:40 INFO mapred.JobClient:     Map output records=2065
11/02/18 19:52:40 INFO mapred.JobClient:     Reduce input records=2065
true

The result is outputed into $WORKSPACE/wordcount/Out**********/part-r-00000 .
Click Run > Debug if you would like to run it in debug mode. You can also use step runs and breakpoints as well as other Java applications.

If you encounter an Out of Memory error when running, change suitably the heap size configuration -Xmx in eclipse.ini. (thanks to Rick).

-Xmx384m