Error and Solution : Detailed step by step instruction on Spark over Yarn - Part 1

I have Hadoop cluster setup, decided to Deploy Apache Spark over Yarn.
for test case I have tried different option to summit Saprk job.
Here I have discussed few Exception / issues during
Spark deployment on Yarn.

Error 1)
Initially I Connected Spark via shell using "local" mode. everything working great.

root@boss:/opt/spark-1.1.0-bin-hadoop2.4# ./bin/spark-shell --master local[2]

when I tried to coonect via "master" mode,

root@boss:/opt/spark-1.1.0-bin-hadoop2.4# ./bin/spark-shell --master spark://boss:7077

I can safly enter into spark-shell, Then I summitted the job,
Resource manager(domainname:8088) Acceptted my job but not allowed to run the job.
I have been waiting quite long time then decide to check log files.

14/09/25 15:54:59 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

14/09/25 15:55:14 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
.....

Solution
There was no runnning worker for the master to assign the job. Started the worker for the master running on server "solai" and port "7077"

root@boss:/opt/spark-1.1.0-bin-hadoop2.4# ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://solai:7077

Error 2)
then I decided to enter spark-shell via Yarn.

root@boss:/opt/spark-1.1.0-bin-hadoop2.4# ./bin/spark-shell --master yarn-client

Spark assembly has been built with Hive, including Datanucleus jars on classpath Exception in thread "main" java.lang.Exception: When running with master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. at org.apache.spark.deploy.SparkSubmitArguments.
checkRequiredArguments(SparkSubmitArguments.scala:182) at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:62) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
scala:70) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Solution
as said in error set HADOOP_CONF_DIR in $SPARK_HOME/conf/spark-env.sh (create this file from conf/spark-env-template.sh)

HADOOP_CONF_DIR=/opt/hadoop-2.5.1/etc/hadoop

Error 3)
Successfully deployed Spark on top of Yarn. Next, Tryied to submit the job via Yarn-cluster.

root@slavedk:/opt/spark-1.1.0-bin-hadoop2.4# ./bin/spark-submit --class org.apache.spark.examples.JavaWordCount --master yarn-cluster lib/spark-examples-1.1.0-hadoop2.4.0.jar /sip/ /op/sh2

Container launch failed for container_1412078568642_0005_01_000003 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. This token is expired. current time is 1412083584173 found 1412082408578

OR

Application application_1411747873135_0002 failed 2 times due to AM Container for appattempt_1411747873135_0002_000002 exited with exitCode: -100 due to: Container expired since it was unused.Failing this attempt.. Failing the application

Solution
     One of the options would be increasing lifespan of container by changing the default time. defaut container expairy itervel is 10 sec.. make it as 20sec.
add the below property to the Hadoop yarn-site.xml file. in my case /opt/hadoop-2.5.1/etc/hadoop/yarn-site.xml

<property>
  	<name>yarn.resourcemanager.rm.container-allocation.
        expiry-interval-ms</name>
         <value>2000000</value>
</property>
 

Related posts

Few more issues Apache Spark on Yarn

Working with Hadoop Eco systems tools : Exception and Solution

This is the continue post on Error & Solution during setup Hadoop HA

Here I have discussed few error / issues during Automatic Fail-over configuration a part of the Hadoop HA setup.

Error 1)

Application application_1406816598739_0003 failed 2 times due to Error launching appattempt_1406816598739_0003_000002. Got exception: java.io.IOException: Failed on local exception: java.net.SocketException: Network is unreachable; Host Details : local host is: "localhost/127.0.0.1"; destination host is: "boss":32939;

Solution
no IP address assigned for the node. manually assigned just like

root@boss[bin]#sudo ifconfig eth0 10.184.36.194

Error 2)

root@solaiv[sqoop]# bin/sqoop import --connect jdbc:mysql://localhost/hadoop --table movies --username root -P --split-by id

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

Solution
Download and put the "sqoop-1.4.4-hadoop200.jar" to SQOOP_HOME

Error 3)
PIG, while executing PIG statement from GRUNT

grunt> cnt = foreach grpd generate group, count(words) as nos; Failed to generate logical plan. Nested exception: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve count using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]

Solution
count in pig statement shoud be upper case COUNT

grunt> cnt = foreach grpd generate group, COUNT(words) as nos;

Error 4)
using SQOOP: MySQL to HIVE

root@boss:/opt/sqoop-1.4.4# bin/sqoop import --connect jdbc:mysql://localhost:3306/hadoop --table csurvey --target-dir /sqoop-hive --hive-import --split-by pname --hive-table csurveysu -username root -P

ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot delete /tmp/hadoop-yarn/staging/root/.staging/job_1407336913887_0001. Name node is in safe mode. The reported blocks 197 has reached the threshold 0.9990 of total blocks 197. The number of live datanodes 1 has reached the minimum number 2 . Safe mode will be turned off automatically in 3 seconds.

Solution
My Hadoop NameNode was in safe mode. i.e read-only mode for the HDFS cluster. it can't Write.. until enough datanode up for replication After started up enough datanode still my NameNode in safe mode, just manually OFF safemode by

root@localhost:/opt/hadoop-2.2.0# bin/hadoop dfsadmin -safemode leave

Just check the inconsistency of HDFS cluster by

root@solaiv[bin]#root@localhost:/opt/hadoop-2.2.0# bin/hadoop fsck /

Error 5)
Hive show tables does not display table "sqooptest" , which was imported by SQOOP

hive> show tables tablename I have imported MySQL to HVE using SQOOP, once it done I can able to see from Hadoop File Systes (HDFS), but When I use "show tables tablname" from HIVE console, it throws no table exists

Solution
just enter into hiveQL from where you used the sqoop import commands.

root@solaiv[bin]# cd /opt root@solaiv[opt]# $SQOOP_HOME/bin/sqoop import --connect jdbc:mysql://localhost:3306/hadoop --table huser --hive-import --split-by name --hive-table sqooptest -username root -P execute hive from /opt, if you are trying to enter other-then this dir, you will not able to view root@solaiv[opt]# $HIVE_HOME/bin/hive hive> show tables tablename

More clarity..
By default HIVE meta data stored in derby database. Derby store the data into current working dir. that can be customize by user "hive-default.xml"