issues : Login in RStudio, SWIRL Package installation and R 3.1.0 installation

Tried to access Rstudio via HTTP using default port 8787. http://localhost:8787.

Error
In my first login i was given root as a username and password. got below error

cannot login to rtudio server on Debian (RStudio initialization error) Unable to connect to service (RStudio initialization error)

Solution
Create new user called ruser.

root@solaiv[bin]# adduser ruser

Adding user `ruser' ...
Adding new group `ruser' (1002) ...
Adding new user `ruser' (1002) with group `ruser' ...
Creating home directory `/home/ruser' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully 

Refreshed the browser and login with newly created user credentials

Error
R version 3.1.0, while configure R

root@solaiv[R-3.1.0]# ./configure

configure: error: --with-readline=yes (default) and headers/libs are not available configure: error: --with-x=yes (default) and X11 headers/libs are not available

Solution
do not need R to be built with this library, you can simply set with-readline to "no":

root@solaiv[R-3.1.0]# ./configure --with-x=no --with-readline=no

Error
while installing "SWIRL" packages

> install.packages("swirl")

Cannot find curl-config
ERROR: configuration failed for package "RCurl"
* removing "/usr/local/lib/R/library/RCurl"
ERROR: dependency "RCurl" is not available for 
package "httr"

* removing "/usr/local/lib/R/library/httr"
ERROR: dependencies "httr", "RCurl" are not 
available for package "swirl"

* removing "/usr/local/lib/R/library/swirl"

Solution

> install.packages("swirl", dependencies=TRUE)

still getting the same error, install and re-try

root@solaiv[R-3.1.0]# apt-get install libcurl4-openssl-dev

root@solaiv[R-3.1.0]# apt-get install libxml2-dev

root@solaiv[R-3.1.0]# R

> install.packages("swirl", dependencies=TRUE)

Related posts

Install R on debian

Error & Solution : Automatic Failover configuration (HDFS High Availability for Hadoop 2.X)

This is the continue post on Error & Solution during setup Hadoop HA

Here I have discussed few error / issues during Automatic Failover configuration
a part of the Hadoop HA setup.

Error 1)
If you are converting a non-HA NameNode to be HA, you should run the command "hdfs namenode -initializeSharedEdits", which will initialize the JournalNodes with the edits data from the local NameNode edits directories

root@solaiv[bin]#./hdfs namenode -initializeSharedEdits

ERROR namenode.NameNode: Could not initialize shared edits dir java.io.IOException: Cannot start an HA namenode with name dirs that need recovery. Dir: Storage Directory /app/hadoop2/namenode state: NON_EXISTENT

Solution
create namenode dir in

root@boss[bin]#mkdir -P /app/hadoop2/namenode

Error 2)

root@solaiv[bin]#./hdfs namenode -initializeSharedEdits

namenode.NameNode: Could not initialize shared edits dir The directory is already locked;

Solution
make sure full permission to hadoop.dir for namenode, datanode and journalnode

root@boss[bin]#chmod 777 -R /app/hadoop2/
I have configured all the dirs under /app/hadoop2

root@boss[bin]#ls -l /app/hadoop2/

drwxrwxrwx 2 root root 4096 Nov 29 12:27 datanode
drwxrwxrwx 3 root root 4096 Nov 28 19:38 jn
drwxrwxrwx 3 root root 4096 Nov 29 12:32 namenode

Error 3)
This time when i run the initializeSharedEdits on standby node,

root@standby[bin]#hdfs namenode -initializeSharedEdits

14/06/03 14:42:28 ERROR namenode.NameNode: Could not initialize shared edits dir java.io.FileNotFoundException: No valid image files found at org.apache.hadoop.hdfs.server.namenode.
FSImageTransactionalStorageInspector.
getLatestImages(FSImageTransactionalStorageInspector.java:144)

Solution
Error due to standby node couldn't sync with active namenode
format the satndby namenode

standby@hadoop[bin]#hdfs namenode -format

Error 4)
in order to Initialize standby node. Format standby node namenode and copy the latest checkpoint (FSImage) from master to standby by executing the following command:

root@standby[bin]#hdfs namenode -bootstrapStandby

This command connects with master node to get the namespace metadata and the checkpointed fsimage. This command also ensures that standby node receives sufficient editlogs from the JournalNodes (corresponding to the fsimage). This command fails if JournalNodes are not correctly initialized and cannot provide the required editlogs.

root@standby[bin]#hdfs namenode -bootstrapStandby

org.apache.hadoop.hdfs.qjournal.protocol.
JournalNotFormattedException: Journal Storage Directory /app/hadoop2/jn/mycluster not formatted

10.184.39.147:8485: Journal Storage Directory /app/hadoop2/jn/mycluster not formatted at org.apache.hadoop.hdfs.qjournal.server.Journal.
checkFormatted(Journal.java:453) at org.apache.hadoop.hdfs.qjournal.server.Journal.
getEditLogManifest(Journal.java:636) at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.
getEditLogManifest(JournalNodeRpcServer.java:181) …... FATAL ha.BootstrapStandby: Unable to read transaction ids 3-13784 from the configured shared edits storage qjournal://master:8485;standby:8485/mycluster. Please copy these logs into the shared edits storage or call saveNamespace on the active node. Error: Gap in transactions. Expected to be able to read up until at least txid 13784 but unable to find any edit logs containing txid 3

Solution
I finally solved this by copying data for a 'good' journal node (aka, from 'master') to the unformatted one (aka, standby where i was getting error)

root@master[bin]#scp -r /app/hadoop2/jn/mycluster/ root@standby:/app/hadoop2/jn/

then restarted the journanl node.

root@standby[bin]#../sbin/hadoop-daemon.sh start journalnode

root@standby[bin]#hdfs namenode -bootstrapStandby

Related posts

Error and Solution - Hadoop HA
distributed Hadoop setup
Issue while setup Hadoop cluster

Issues & Solution : HDFS High Availability for Hadoop 2.X

Here I have discussed few error / issues during the Hadoop HA setup.

Error 1)
when I start resourcemanager from active namenode in Hadoop HA ,

root@master:/opt/hadoop-2.2.0# sbin/yarn-daemon.sh start resourcemanager

Problem binding to [master:8025] java.net.BindException: Cannot assign requested address;

Solution
Check your /etc/hosts file, If you have multiple enrty for same IP/localhost, delete and make sure only one valued entry.

Just I removed all other entry for the IP ' 10.184.39.167' from /etc/hosts

10.184.39.167 standby

Error 2)
Once I have configured Haddop HA, strated Haddop cluster

root@master[bin]#hdfs namenode -format

FATAL namenode.NameNode:Exception in namenode join

org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if JNs are ready for formatting. 1 exceptions thrown:

10.184.39.67:8485: Call From standby/10.184.39.62 to master:8485 failed on connection exception: java.net.ConnectException:

Solution
start all the configured journalnode then format the Hadoop namenode

root@master[bin]#../sbin/hadoop-daemon.sh start journalnode
root@master[bin]#hadoop namenode -format

Error 3)
As mentioned in Solution 2 above, journalnode started and hadoop namenode formatted successfully
when I try to start DFS

root@master[bin]#../sbin/start-dfs.sh

java.io.IOException: Cannot start an HA namenode with name dirs that need recovery. Dir: Storage Directory /app/hadoop2/namenode state: NOT_FORMATTED

Solution
  Error due to data sync. b/w active node and standby node, If your distributed Hadoop setup is fresh you may not get this error.
I copied dfs directory from the active namenode to the standby namenode

root@standby[hadoop-2.2.0]#scp -r /app/hadoop2/namenode/* root@master:/app/hadoop2/namenode/

make sure full permission to hadoop.dir

root@master[bin]#chmod 777 -R /app/hadoop2/

re started it.

root@master[bin]#../sbin/start-dfs.sh

Error 4)
Once I have configured Haddop HA, strated Haddop cluster

root@master[bin]#hdfs namenode -format

FATAL namenode.NameNode:Exception in namenode join

org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if JNs are ready for formatting. 1 exceptions thrown:

10.184.39.67:8485: Call From standby/10.184.39.62 to master:8485 failed on connection exception: java.net.ConnectException:

Solution
start all the configured journalnode then format the Hadoop namenode

root@master[bin]#../sbin/hadoop-daemon.sh start journalnode
root@master[bin]#hadoop namenode -format

Error 5)
NameNode doesn't start in Hadoop2.x

root@master[bin]#../sbin/start-dfs.sh

Incorrect configuration namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured. Starting namenodes on [] …. …..

Solution
This error caused due to *.site.xml tag problem, I'v checked my *site.xml file, all seems to be correct. But mistakenly I have commended fs.default.name in core-site.xml.

 
<property>
   <name>fs.default.name</name>
         <value>hdfs://master:9000</value>
</property>

Related posts

Error and Solution - Hadoop HA Automatic Failover
distributed Hadoop setup
Issue while setup Hadoop cluster

Indexing and Searching using IDOL OnDemand

IDOL OnDemand delivers a rich set of web service APIs to enable developers to create ground-breaking data driven apps.To get more details on each HP IDOL OnDemand API click here
In this tutorial we are going to cover only 5 APIs,

• Create Index
• Obejct Store
• View Document
• Find Related Concepts
• Find Similar

For all APIs, calls are made using JQuery

$('div.api').bind('click', function() {
    $.ajax({url:URL,
 success:function(result){    
 //convert into JSON
 var third4 = JSON.stringify(result,undefined,2)     
 //process the data 
  },error:function(data){
//Handling Error 
  }
  }); 
});

Create Index : The Create Text Index API allows you to create a text index, that you can use to add your own content to IDOL OnDemand. Here we are creating index called 'Cancer Prevention'

URL = https://api.idolondemand.com/1/api/sync/createtextindex
/v1?index=Cancer+Prevention&flavor=explorer&
apikey=dac630d2-4aed-45b7-8fc7-99fa87858460

Store Object : The Store Object API takes a file, reference, or an input URL and stores the contents of the document for use in other APIs. It returns the object store reference, which you can pass to other APIs to process the document that you store. In this tutorial API takes Files from url -> http://www.cancer.org/acs/groups/cid/documents/webcontent/002577-pdf.pdf

URL = https://api.idolondemand.com/1/api/sync/storeobject
/v1?url=http%3A%2F%2Fwww.cancer.org%2Facs%2Fgroups%2Fcid%2F
documents%2Fwebcontent%2F002577-pdf.pdf&
apikey=dac630d2-4aed-45b7-8fc7-99fa87858460

view document call: Converts a document to HTML format for viewing in a Web browser. in this snippet we are executing a view document call and and highlighting the phrase 'physical activity' in the document.

URL = https://api.idolondemand.com/1/api/sync/viewdocument/
v1?url=http%3A%2F%2Fwww.cancer.org%2Facs%2Fgroups%2Fcid%2F
documents%2Fwebcontent%2F002577-pdf.pdf&highlight_expression
=physical+activity&start_tag=%3Cb%3E&raw_html=true&
apikey=dac630d2-4aed-45b7-8fc7-99fa87858460

Find Related Concepts Returns the best terms and phrases in documents that match the specified query.

URL = https://api.idolondemand.com/1/api/sync/findrelatedconcepts/
v1?reference=b856d643-bc0a-48f6-88a2-e7489aa33be9&
sample_size=5&apikey=dac630d2-4aed-45b7-8fc7-99fa87858460

Find Similar API : Finds documents that are conceptually similar to your text or a document. Returns documents in the IDOL OnDemand databases that are similar to text or a document that we provide. We can either submit text, an index reference, a file, an Store Objetc reference, or a URL.

 URL = https://api.idolondemand.com/1/api/sync/findsimilar
/v1?text=Hello+World&indexes=wiki_eng&apikey=
dac630d2-4aed-45b7-8fc7-99fa87858460
 

Related posts

Few more API calls using JQuery

IDOL OnDemand API Call using Java

IDOL OnDemand API call using JAVA

IDOL OnDemand delivers a rich set of web service APIs to enable developers to create ground-breaking data driven apps.To view complete set of API here
In this tutorial we are going to cover only 3 APIs,

• Find Similar
• OCR Document
• Sentiment Analysis

Find Similar API : Finds documents that are conceptually similar to your text or a document. Returns documents in the IDOL OnDemand databases that are similar to text or a document that we provide. we can either submit text, an index reference, a file, an object store reference, or a URL.
In this tutorial, Find Similar API call from Wikipedia on the phrase 'Hello World'.

OCR Document API : The OCR Document API extracts text from an image that we provide. The API returns the extracted text, along with information about the location of the detected text in the original image.
In this tutorial, OCR Document API will call from the following image: 'http://www.java-made-easy.com/images/hello-world.jpg' and display the extracted text.

Sentiment Analysis API : The Sentiment Analysis API analyzes text to return the sentiment as positive, negative or neutral. It contains a dictionary of positive and negative words of different types, and defines patterns that describe how to combine these words to form positive and negative phrases.
In this tutorial, Sentiment Analysis API call with the text from the Hello World response (i.e reference link ). Display the Sentiment Score along with the Sentiment Rating

Java Application, in web.xml

<welcome-file-list>
        <welcome-file>test.jsp</welcome-file>
        <welcome-file>test.jsp</welcome-file>
    </welcome-file-list>
    
    <servlet>
        <servlet-name>findsimilar</servlet-name>
        <servlet-class>idol.api.HelloWorld</servlet-class> 
    </servlet>

    <servlet-mapping>
        <servlet-name>findsimilar</servlet-name> 
        <url-pattern>/findsimilar</url-pattern> 
    </servlet-mapping>

in test.isp

<div class="section1">
<h3>In this tutorial</h3>

<div class="block">
<img src="images/similar.jpg" alt="" width="40" height="55" />
<h6>  <a href="findsimilar?api=api1"> servlet </a>  .</h6>
</div>

<div class="block">
<img src="images/ocr.jpg" alt="" width="40" height="55"/>
<h6>  <a href="findsimilar?api=api2"> servlet </a>  </h6>
</div>

<div class="block">
<img src="images/senti.jpg" alt="" width="40" height="55"/>
<h6> <a href="findsimilar?api=api3"> servlet </a>   </h6>
</div>

</div>

In servlet HelloWorld.java

String reference = "";	
String url3 = "https://api.idolondemand.com/1/api/sync/analyzesentiment/v1?url="+reference+"&apikey=dac630d2-4aed-45b7-8fc7-99fa87858460";
	
 protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
       
     
    	String url = "";  	
    	
        String url1 = "https://api.idolondemand.com/1/api/sync/findsimilar/v1?text=Hello+World&indexes=wiki_eng&apikey=dac630d2-4aed-45b7-8fc7-99fa87858460";
        String url2 = "https://api.idolondemand.com/1/api/sync/ocrdocument/v1?url=http%3A%2F%2Fwww.java-made-easy.com%2Fimages%2Fhello-world.jpg&apikey=dac630d2-4aed-45b7-8fc7-99fa87858460";
        
       	String api = request.getParameter("api");	
		
		if (api.equalsIgnoreCase("api1")){
			url = url1;
			}
		else if(api.equalsIgnoreCase("api2")){
			url = url2;
		}
		else{
			url = url3;			
		}
		
           Client1 cl1 = new Client1(); 
			String str =  cl1.run(url);			
			
			if (api.equalsIgnoreCase("api1")){
				 String []a =  str.split("reference");
				 String []b = a[1].split("weight");
				 reference =  b[0].substring(b[0].indexOf("http"), (b[0].indexOf(",",b[0].indexOf("http")))-1 );				 
				 url3 = "https://api.idolondemand.com/1/api/sync/analyzesentiment/v1?url="+reference+"&apikey=dac630d2-4aed-45b7-8fc7-99fa87858460";
				 request.setAttribute("ref", reference);
				}
			
			if (api.equalsIgnoreCase("api2")){			     
				 String []a = str.split("text", 3);
				 String []b = a[2].split("left");				
				 str =  b[0].substring((b[0].indexOf("="))+1, (b[0].indexOf(",",b[0].indexOf("left")))-1 );				 				
				}
			
						
			if (api.equalsIgnoreCase("api3")){
			     String []a = str.split("aggregate", 3);			     
			     if (a.length !=1){
				 str = a[1];}				 
				}
			
			response.setContentType("text/html");
			request.setAttribute("servletName", str); 		    
	    	
	      getServletConfig().getServletContext().getRequestDispatcher(
	    		  "/test2.jsp").forward(request,response); 
	    
    }

in HelloWorld.java, we are using reference global variable and it stores link from the Find Similar API, which will be used for sentiment analysis API,

in Client.java

public String run(String url){
		
		String body = "";
		 HttpClient httpclient = new DefaultHttpClient();
		 HttpGet httpget = new HttpGet(url);
		 System.out.println("httpget : "+httpget);		 
		 BasicHttpResponse response = null;
		 
		 try {
			response = (BasicHttpResponse) httpclient.execute(httpget);			
			StatusLine statusLine = response.getStatusLine();					
			 
			HttpEntity entity = response.getEntity(); 
		    if (entity != null) {		    	
		    	body = EntityUtils.toString(entity);		        	
		    }
		 }catch (Exception e) {
			// TODO: handle exception
			 e.printStackTrace();
		} 
		 finally {
			    System.out.println("Finally !!!");
		 }
		return body;	 
	}

<div class="section2">
<h3> API Response is </h3>
<div style="height: 227px; overflow:scroll;width:632px">
<pre>
<% 
out.print(request.getAttribute("servletName").toString()); 
%>
</pre>
</div>

Output of Sentiment Analysis API

: {
    "sentiment": "neutral",
    "score": -0.06540204218347706
  }
}

IDOL OnDemand API call using JavaScript

IDOL OnDemand delivers a rich set of web service APIs to enable developers to create ground-breaking data driven apps.To view complete set of API here
In this tutorial we are going to cover only 3 APIs,

• Find Similar
• OCR Document
• Sentiment Analysis

Find Similar API : Finds documents that are conceptually similar to your text or a document. Returns documents in the IDOL OnDemand databases that are similar to text or a document that we provide. we can either submit text, an index reference, a file, an object store reference, or a URL.
In this tutorial, Find Similar API call from Wikipedia on the phrase 'Hello World'.

 
$.ajax({url:"https://api.idolondemand.com
/1/api/sync/findsimilar/v1?text=Hello+World&indexes=wiki_eng&
apikey=dac630d2-4aed-45b7-8fc7-99fa87858460",
success:function(result)

{window.reference  = result.documents[0].reference;..}

Output response will be

  "documents": [
    {
      "reference": "http://en.wikipedia.org/wiki/Hell",
      "weight": 89.25,
      "links": [
        "HELL",
        "WORLD"
      ],
      "index": "wiki_eng",
      "title": "Hell"
    },
    {
      "reference": "http://en.wikipedia.org/wiki/Hello world program",
      "weight": 89.07,
      "links": [
        "HELL",
        "WORLD"
      ],
      "index": "wiki_eng",
      "title": "Hello world program"
    }, 

OCR Document API : The OCR Document API extracts text from an image that we provide. The API returns the extracted text, along with information about the location of the detected text in the original image.
In this tutorial, OCR Document API will call from the following image: 'http://www.java-made-easy.com/images/hello-world.jpg' and display the extracted text.

$.ajax({url:"https://api.idolondemand.com/1/api/sync
/ocrdocument/v1?url=http%3A%2F%2Fwww.java-made-easy.com%2Fimages%2F
hello-world.jpg&apikey=dac630d2-4aed-45b7-8fc7-99fa87858460",
success:function(result){
..}

Output response will be

{
  "text_block": [
    {
      "text": "= 5\nJR '\npublic class Hellouorld {\n/ # 'X;
\n* @param args\n*!\n; public static void main
(String[] argsj {\n// TODO Auto-generated method stub El\nSystem
.out.printlnl"Hello world!"1;\n}\n}\n< ) i",
      "left": 0,
      "top": 0,
      "width": 562,
      "height": 472
    }
  ]
}

Sentiment Analysis API : The Sentiment Analysis API analyzes text to return the sentiment as positive, negative or neutral. It contains a dictionary of positive and negative words of different types, and defines patterns that describe how to combine these words to form positive and negative phrases.
In this tutorial, Sentiment Analysis API call with the text from the Hello World response (i.e reference link ). Display the Sentiment Score along with the Sentiment Rating

 var url3 = 'https://api.idolondemand.com/1/api/
sync/analyzesentiment/v1?url='+reference+'&
apikey=dac630d2-4aed-45b7-8fc7-99fa87858460';

$.ajax({url:url3,success:function(result){    
 var sentiment = result.aggregate.sentiment;
 var score = result.aggregate.score;}

Output response will be

 "aggregate": {
    "sentiment": "neutral",
    "score": -0.06540204218347706
  }

Install R in Debian squeeze/BOSS OS

Install R Prog. on Debian Squeeze

R is a Free software environment for Data Analysis and Graphics

• Programming Language
• Data Visualization

By default, Debian squeeze comes up with R packages, you can verify by

root@solaiv[~]#apt-cache search ^r-.*

if you were not find anything add an appropriate entry in

root@solaiv[~]#vi /etc/apt/sources.list

the newest R release can be installed using a command sequence like

root@solaiv[~]#apt-get update
root@solaiv[~]#apt-get install r-base
root@solaiv[~]#apt-get install r-base-dev

to enter to R console

root@solaiv[~]#R
>

simple add example in R

> a <- 5;
> b <-10;
> c <- a+b;
> c
[1] 15

R packages may then be installed by the local user/admin from the CRAN source packages, typically from inside R using the working wit addition R packages

Install packages from CRAN

> install.packages("RJDBC")

Install NEW packeges from downloaded file

> install.packages("file.tar.gz")

loading packages into sessions

> library("strings")
> require("strings")

Related posts

Install RStudio
issues cannot login to RStudio server

Hbase HMaster cannot started/Aborted in Debiain/Cent OS.

When I start Hbase, my HMaster started successfully, but after some time (with in one minute) HMaster aborted all other daemons are still running. Here is the log.

 

2014-05-21 15:45:05,075 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.

java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused

2014-05-21 15:45:16,794 INFO org.apache.hadoop.hbase.master.HMaster: Aborting

2014-05-21 15:45:16,794 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads

2014-05-21 15:45:16,794 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000

2014-05-21 15:45:16,794 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting

2014-05-21 15:45:16,794 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting

2014-05-21 15:45:16,794 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting

java.lang.RuntimeException: HMaster Aborted

at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)

at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java

SOLUTION

In my  hbase-site.xml, I have missed out hbase.rootdir propery

<property> 
     <name>hbase.rootdir</name> 
     <value>hdfs://localhost:54310/hbase</value> 
</property> 

Once I added, restarted the hbase now Hmaster running fine.