How-To : Setup Development Environment for Hadoop MapReduce

This post is intended for folks who are looking out for a quick start on developing a basic Hadoop MapReduce application.

We will see how to set up a basic MR application for WordCount using Java, Maven and Eclipse and run a basic MR program in local mode , which is easy for debugging at an early stage.

Assuming JDK 1.6+ is already installed and Eclipse has a setup for Maven plugin and download from default maven repository is not restriced.

Problem Statement  : To count the occurrence of each word appearing in an input file using  Hadoop MapReduce.

Step 1 : Hadoop Maven Dependency

Create a maven project in eclipse and use following code in your pom.xml.

Upon saving it should download all required dependencies for running  a basic Hadoop MapReduce program.

Step 2 : Mapper Program

Map step involves tokenizing the file, traversing the words, and emitting a count of one for each word that is found.

Our mapper class should extend Mapper class and override it’s map method. When this method is called the value parameter of the method will contain a chunk of the lines of file to be processed and the output parameter is used to emit word instances.

In real world clustered setup, this code will run on multiple nodes which will be consumed by set of reducers to process further.

Step 3 : Reducer Program

Our reducer extends the Reducer class and implement logic to sum up each occurrence of word token received from mappers.Output from Reducers will go to the output folder as a text file ( default or as configured in Driver program for Output format) named as  “part-r-00000” along with a _SUCCESS file.

 Step 4 : Driver Program

Our driver program will configure the job by supplying the map and reduce program we just wrote along with various input  , output parameters.

 

That’s It !! We are all set to execute our first MapReduce Program in eclipse in local mode.

Let’s assume there is an input text file called input.txt in folder input which contains following text :

Expected output :

Let’s run this program in eclipse as Java Application :-

We need to give path to input and output folder/file  to the program as argument.Also, note output folder shouldn’t exist before running this program else program will fail.

If this program runs successfully emitting set of lines while it is executing mappers and reducers, we should see a output folder and with following files :

 

Enjoy Learning !!


 

  • Alessandro La Torraca

    Hi Saurz!
    Thank you for your guide. I have a problem with Eclipse following your instruction. I have created a new project with your pom.xml file, but I don’t know where should I creare the 3 classes in order to be able to set WordCount class as main class in my run configuration.
    Thank you in advance for you help

    Alex

    • Great Alessandro !!
      So you took the first step right, for using those classes, cust create any package using New > package and created those classes inside that.And when you have everything in there, just Right click on WordCount and select Run As .. Java Application. That’s it !! Let me know if you face any trouble.

      • Alessandro La Torraca

        Hi! Thank you for your answer. It works now!

        All the best,

        Alex

        • That’s Great !! Happy Learning . Feel free to share this !!

  • Ideally Warning shouldn’t be blocking the execution , you need to add log4j.properties in class path , did you ? But yes, it should be running in spite of that warning.

    • Alessandro La Torraca

      I solved this problem adding this line in main class, which inizialize the log4j

      org.apache.log4j.BasicConfigurator.configure();

  • Guruprasad Prabhu

    Hi Saurabh,

    Thanks for posting the example.

    I was trying to implement the same in Spring STS, but I am getting some exceptions. Below link has error details

    https://www.dropbox.com/s/ul84py2rx8ez489/Exception.TXT?dl=0

    Can you please help solving the issue?

    Is there any prerequisites required for running the program?

    Thanks in advacne
    -Guru

    • saurzcode

      Hi Guru,

      Looks like you are trying to implement MaxTemperature Program and not Word count as mentioned in post.I can’t comment on your error if I wouldn’t know what steps did you follow for that. Please follow the WordCount and try implementing.Once this is successful you can follow Max Temperature on similar path.