Why Configure Eclipse for Apache Hadoop?
- From Eclipse to work on Hadoop files systems (HDFS).
- Create New Directory
- Upload files to HDFS
- Upload Directory to HDFS
- Download from HDFS
- Write & Execute MapReduce program, which runs on Hadoop cluster
Step to Integrate Eclipse to work on Hadoop Cluster
- Hope you have Installed Eclipse, else download & install
- Run your Hadoop server, If haven't, setup Hadoop
- Download hadoop-eclipse-plugin-1.2.1.jar and place the Jar into Eclipse/plugins (instead of downloading you can also build the plugin jar file yourself using "ant")
- Start the eclipse
- $ECLIPSE_HOME/eclipse
- In Eclipse menu click, Window --> Open Perspective --> Others --> MapReduce
- In bottom MapReduce icon click to Add new Hadoop location
- Enter MapReduce & HDFS running port
- for recall, MapReduce port (9001) specified in $HADOOP_HOME/conf/mapred-site.xml
- for recall, HDFS port (9000) specified in $HADOOP_HOME/conf/core-site.xml
- Enter the Hadoop user name
- Once Hadoop location added, DFS Locations will be seen/displayed in Eclipse Project Explorer window, (Windows-->Show View-->Project Explorer)
- Once Hadoop added, DFS Locations will be seen/displayed in Project Explorer window,
- Right click DFS location and click to Connect
- Once connected successfully, it will display all the DFS Folder.
- You can create Directory, Upload files to HDFS location, Download files to local by right click any of the listed Directory.
Possible Error you may get
ERROR
Error: Call to loaclhost/127.0.0.1:9000 failed on connection exception:java.net:ConnectionException
SOLUTION
Make sure you have all the Hadoop daemons up&running.
7 comments:
i have tried in the same way as you described. But i got the connection exception error even if i had started all the daemons and it was running.
hope you will come up with error, as we discussed via chat
I have setup hadoop-2.4.0 and i get the following error
Server IPC version 9 cannot communicate with client version 4
inside eclipse when itry to connect.
can u help?
it seems, you have not used proper plugin for Hadoop 2.X. I hope you have downloaded hadoop-eclipse-plugin-1.2.1.jar as per this post.
for Hadoop2.x, download JAR from here
https://github.com/winghc/hadoop2x-eclipse-plugin/archive/master.zip
Hi Solai,
Nice Post on Setting up the cluster.
A clarification, Master does not have started Datanode services and replication factor has been set to 2.
So two Datanodes are important.
1st on Master.
2nd on Slave.
Ya.. u r right.. in this case, start datanode from datanode m/c.
hadoop-daemons.sh start datanode
Your comments are a refreshing blend of intellect and humility – a rare and appreciated combination. User-friendly Click Test – simplicity at its best.
Post a Comment