Setting hadoop 2.2.0 on ubuntu 12 lts，hadoop2.2.0

Setting hadoop 2.2.0 on ubuntu 12 lts，hadoop2.2.0

# Prerequisite:
(Installing java openjdk-7 on ububtu)

If not having installed java on before than the process is:

$ sudo apt-get install openjdk-7-jdk
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4ubuntu3)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
$ cd /usr/lib/jvm
$ ln -s java-7-openjdk-amd64 jdk

If having already installed java then have the clean openjdk install:

$ sudo rm /var/lib/dpkg/info/oracle-java7-installer* 
$ sudo apt-get purge oracle-java7-installer* 
$ sudo rm /etc/apt/sources.list.d/*java* 
$ sudo apt-get update 
$ sudo add-apt-repository ppa:webupd8team/java 
$ sudo apt-get update 
$ sudo apt-get install oracle-java7-installer

Install openssh-server

$ sudo apt-get install openssh-server

# Add hadoop group and user
(although we can use any user without issue but making a separate will make hadoop user tasks in different user)

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser
$ sudo adduser hduser sudo

# Setting ssh for login at localhost

$ ssh-keygen -t rsa
   (3 times press enter)
$ ssh user-name@localhost mkdir -p .ssh
$ cat .ssh/id_rsa.pub | ssh user-name@localhost 'cat >> .ssh/authorized_keys'
$ ssh user-name@localhost "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
$ ssh localhost
Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.8.0-29-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

Last login: Tue Jan 14 19:27:05 2014 from localhost

#Install hadoop

$ cd ~
$ wget http://www.trieuvan.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
$ sudo tar vxzf hadoop-2.2.0.tar.gz -C /usr/local
$ cd /usr/local
$ sudo mv hadoop-2.2.0 hadoop
$ sudo chown -R hduser:hadoop hadoop

# Setup hadoop environment variables

$cd ~
$vi .bashrc
 
paste following to the end of the file
 
#Hadoop variables
export JAVA_HOME=/usr/lib/jvm/jdk/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
###end of paste

$ cd /usr/local/hadoop/etc/hadoop $ vi hadoop-env.sh   #modify JAVA_HOME export JAVA_HOME=/usr/lib/jvm/jdk/

# Testing hadoop install (Getting version information)

$ hadoop version
Hadoop 2.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using /usr/local/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar

#Configuring hadoop environment

$ cd /usr/local/hadoop/etc/hadoop
$ vi core-site.xml
#Paste following between <configuration>
 
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property>
 
 
$ vi yarn-site.xml
#Paste following between <configuration>
 
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
 
 
$ mv mapred-site.xml.template mapred-site.xml
$ vi mapred-site.xml
#Paste following between <configuration>
 
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>
     <property>

        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>  
    </property>
 
$ cd ~
$ mkdir -p mydata/hdfs/namenode
$ mkdir -p mydata/hdfs/datanode
$ cd /usr/local/hadoop/etc/hadoop
$ vi hdfs-site.xml
Paste following between <configuration> tag
 
<property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/mydata/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/mydata/hdfs/datanode</value>
 </property>

# Format namenode before first use and only once.

hduser@ubuntu40:~$ hdfs namenode -format

# Starting hadoop service

$ start-dfs.sh
....
$ start-yarn.sh
....
 
hduser@ubuntu40:~$ jps
If everything is sucessful, you should see following services running
2583 DataNode
2970 ResourceManager
3461 Jps
3177 NodeManager
2361 NameNode
2840 SecondaryNameNode

If not getting above output on 'jps' then run stop-dfs.sh and stop-yarn.sh and run themmanually from /usr/local/hadoop/sbin/start-dfs.sh and start-yarn.sh.



# Running sample hadoop for complete testing of fresh install

hduser@ubuntu: cd /usr/local/hadoop
hduser@ubuntu:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
 
Number of Maps  = 2
Samples per Map = 5
13/10/21 18:41:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Starting Job
13/10/21 18:41:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
13/10/21 18:41:04 INFO input.FileInputFormat: Total input paths to process : 2
13/10/21 18:41:04 INFO mapreduce.JobSubmitter: number of splits:2
13/10/21 18:41:04 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
...


http://javatute.com/javatute/faces/post/hadoop/2014/setting-hadoop-2.2.0-on-ubuntu-12-lts.xhtml

http://codesfusion.blogspot.in/2013/10/setup-hadoop-2x-220-on-ubuntu.html