Big Data / Data Analytics Job

Data Architect   3-7+ years   

Salary : £70000 - £75000 per annum     location : Central London

Posted: 26/06/2018    Job Type : Permanent

Job Ref: 439101143    Skills & Apply ->

Data Scientist   3-7+ years   

Salary : £232 - £298 per day     location : London

Posted: 18/06/2018    Job Type : Temporary

Job Ref: 775109025    Skills & Apply ->

#AIforAll - AI in India : Use case & scenario explained by NITI AAYOG

post - #AIforAll - focus area for #AI in India

National Strategy for Artificial Intelligence

*) NITI AAYOG released National Strategy for Artificial Intelligence, Download #AIFORALL

Findings in National Strategy for #AI

Top 5 - #AI Personality mentioned in doc

There are around 30 unique personility mentioned in the #AIforAll doc

*) Andrew Ng
*) Von Ahn
*) Sam Altman
*) Olga Russakovsky
*) Marvin Minsky

Scenario discussed in #AI strategy

Box 1: Machine Learning and Deep Learning

---

travel substantial distances for basic and advanced healthcare services. (Box 2: What TMH's Cancer

---

Box 3: Government of India's push for Universal Healthcare Coverage

---

Box 4: AI for India's cancer woes

---

Box 5: Application for soil care

---

Box 7: AI for herbicide optimization

---

Box 9: Creating ‘smart content' for improved interactivity

---

Box 10: Microsoft is helping in predicting drop outs in Andhra

---

Box 12: Effect on R&D

---

Box 13: What India can learn from other economies in terms of AI

---

Box 14: The PRAIRIE Institute

---

reduce operational and maintenance costs while keeping national data secure (Box 11: AIRAWAT)

---

ICTAI Inc., the overall governing body for ICTAIs, could be the DARPA (see Box 16) equivalent of AI

---

Box 16: Defense Advanced Research Projects Agency

---

Box 17: European Organization for Nuclear Research aka CERN

---

Box 18: Impact sourcing through Samasource

---

Box 19: Why the focus on PSUs

---

Box 19: Decoding Explainable AI

---

Box 20: Differential Privacy

---

Box 21: ImageNet

Top 5 - Country mentioned in doc

There are 52 unique place mentioned in the strategy documnts

*) India

*) China

*) Japan

*) France

*) Germany

In next blog will discuss possible use-case from AREAS FOR AI INTERVENTION.

Next :

Massacre at thoothukudi - #SterliteProtest turnd #Sterlitekills - Twitter snipped

Tweet Summary - #SterliteProtest

total tweet mined - 5000

| Original tweet - 429

popular tweet - #SterliteProtest

AdmkFails ::Again and Again fake propaganda by ADMK ITWING those pictures are belongs to 2015 incident n they sharing it now to… https://t.co/ZP6J0gE1GM

---

Ahmedshabbir20 ::Not even a single Minister from TN Govt has the time to visit #Thoothukudi firing victims? Look at what TN Deputy C… https://t.co/SnbxpICTTz

---

Ahmedshabbir20 ::Vedanta owner Anil Aggarwal's son assaulted in London. 3 arrested #Thoothukudi #SterliteProtest #SterliteKillings #MurderOfMakkal

#SterliteProtest - words power

TOP +ve tweets with- #SterliteProtest

jspsampaul ::New India is killing us #Thoothukudi new India is not safe #SterliteProtest New India will be Capitalistic New Indi… https://t.co/rytoYHVaS3

---

VJIndian_ ::Beauty of #AntiSterliteProtest A. Sonia Gandhi clears project Rahul Gandhi opposes project B. Karunanidhi clears p… https://t.co/K5SvoMtv2t

---

Ansari2Thamim ::Salute you Brother. Good journalist should be like you. #SterliteProtest #BanSterlite #DissolveTNGovernment… https://t.co/33MmR9Uqdg

TOP -ve tweets with- #SterliteProtest

albertalbey ::Worst time ever. Divide, exploit, extort, murder, rape. Injustice everywhere, pm and central ministers opens mouth… https://t.co/0XdwO0Xnmz

---

urvashi_valecha ::When you're uninfluential, it takes multiple cancer cases, 100 day protests and deaths in a massacre for any issue… https://t.co/sTyrfZICEm

---

dhanyarajendran ::Note how people who condemn destruction of public property and claim their blood is boiling, have no strong condemn… https://t.co/7XVaqj4CFE

#SterliteProtest : most words

Top 5 mention with #SterliteProtest

@narendramodi - 12

|

@CMOTamilNadu - 8

@OfficeOfOPS - 8

|

@MirrorNow - 5

Top 5 hastag associated #SterliteProtest

#sterliteprotest - 209

|

#thoothukudi - 82

#sterlitekillings - 23

|

#sterlite - 15

List of TOP Deep Learning packages in R

previous post - Apache Spark : steps to Start worker across cluster

Complete list of Deep Learning package in R

R Package	Description
nnet	Software for feed-forward neural networks with a single hidden layer, and for multinomial log-linear models.
neuralnet	Training of neural networks using backpropagation
h2o	R scripting functionality for H2O
RSNNS	Interface to the Stuttgart Neural Network Simulator (SNNS)
tensorflow	Interface to TensorFlow
deepnet	Deep learning toolkit in R
darch	Package for Deep Architectures and Restricted Boltzmann Machines
rnn	Package to implement Recurrent Neural Networks (RRNs)
FCNN4R	Interface to the FCNN library that allows user-extensible ANNs
rcppDL	Implementation of basic machine learning methods with many layers (deep learning), including dA (Denoising Autoencoder), SdA (Stacked Denoising Autoencoder), RBM (Restricted Boltzmann machine) and DBN (Deep Belief Nets)
deepr	Package to streamline the training, fine-tuning and predicting processes for deep learning based on darch and deepnet
MXNetR	Package that brings flexible and efficient GPU computing and state-of-art deep learning to R

Next :

crowd sourced Tamil Movie review - Annadurai

Tweet Summary - #annadurai

total tweet mined - 1958

| Original tweet - 435

Top 5 mention with #annadurai

@vijayantony - 165

|

@mrsvijayantony - 66

|

@realradikaa - 53

@dianachampika - 33

|

@vijayantonyfilm - 28

Top 5 hastag associated #annadurai

#VijayAntony - 354

|

#ndrasena - 41

|

#Thiruttuppayale2 - 30

#DianaChampika - 11

|

#mersal - 10

popular tweet - #annadurai

@avardhaar ::After #Thalapathy #Vijay 's #Mersal Only #VijayAntony 's #Annadurai \nGST Song became Popular - Annadurai Heroine https://t.co/FJ6P98b3c3

@LMKMovieManiac ::#Annadurai 1st half - Has an interesting, attention-grabbing surprise element at the interval block! \nThe film is c https://t.co/kRkeGEgx4q

@sri50 ::#Annadurai is a mass commercial @vijayantony template film in which he impresses in a dual role as identical twins https://t.co/jrYhRQ4aOL

crowd sourced #annadurai - sentiment

taskscheduleR: Scheduling R script via R and Rstudio

previous post - Analyse type of hospital Readmission

Scheduling R script via R.?

taskscheduleR - It allow users to automate R script on specific timepoints from R / Rstudio itself. No need of windows Task scheduling or Linux Cron to automate R script.

Installation of taskscheduleR

devtools::install_github("jwijffels/taskscheduleR")

If you want the RStudio add-in to work, also install miniUI and shiny

install.packages('miniUI') install.packages('shiny')

Open Rscheduler window

Click the Addins menu added in Rstudio

You can also open Rscheuler window in Browser by

hit below command in R terminal,
taskscheduleR:::taskschedulerAddin()

you will notify with "Listening on http://127.0.0.1:5499 " and the same can be viewed in browser

Related posts :
RPostgreSQL: Analyse PostgreSQL data using R

issues: Login in RStudio and Installing SWIRL Package

Rserve - TCP/IP server which allows other programs to use facilities of R

previous post - RPostgreSQL Data analytics on PostgreSQL data using R

Rserve - is a TCP/IP server which allows other programs to use facilities of R from various languages without the need to initialize R or link against R library. Every connection has a separate workspace and working directory. Client-side implementations are available for popular languages such as C/C++, PHP and Java. Rserve supports remote connection, authentication and file transfer.

Installing and initiate Rserve

> install.packages('Rserve')

> library(Rserve)

> Rserve()

Error

> Rserve()

Rserv started in daemon mode.
##> SOCK_ERROR: bind error #98(address already in use)

Solution

boss@passive:~$ ps faux | grep Rserve

will list the Rserv started daemon.

Related posts :
RPostgresql : how to pass dynamic parameter to dbGetQuery

Working With RHadoop

RPostgreSQL Data analytics on PostgreSQL data using R

previous post - XGBoost in R. Error in xgb.DMatrix can not open file

Data analytics on PostgreSQL data using R. Working with R and PostgreSQL for large scale data analytics

Installing RPostgreSQL

install.packages('RPostgreSQL')

compilation terminated. /usr/lib/R/etc/Makeconf:128: recipe for target 'RS-PQescape.o' failed make: *** [RS-PQescape.o] Error 1 ERROR: compilation failed for package 'RPostgreSQL' * removing '/home/boss/R/x86_64-pc-linux-gnu-library/3.1/RPostgreSQL'
RS-PostgreSQL.h:23:26: fatal error: libpq-fe.h: No such file or directory

Solution

RPostgreSQL require libpq-dev.

libpq-dev is a set of library functions that allow client programs (for our case its 'R') to pass queries to the PostgreSQL backend server and to receive the results of these queries.

So we have to install libpg-dev on OS not from R terminal

solai@server# sudo apt-get install libghc-postgresql-libpq-dev

then install RPostgreSQL inside R session
> install.packages('RPostgreSQL')

Related posts :
RPostgresql : how to pass dynamic parameter to dbGetQuery

Working With RHadoop

RHadoop - Running Hadoop MapReduce from R

previous post - RHadoop - rJava Installation Error

Running Hadoop Mapreduce From R. Working with R and Hadoop for large scale data analytics

Running Mapreduce job

m <- mapreduce(InputPath, input.format = 'csv', map = mapFunction, reduce = reduceFunction )

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1

Solution

Error seems map not find the file location,
I was trying to reading File from local but by backend configured with Local

rmr.options(backend = "local")

Reading File from Hadoop in R session

f = hdfs.line.reader("/HDFS/GdpData.csv");

Error in .jcheck(silent = TRUE) :
No running JVM detected. Maybe .jinit() would help.

Solution

Run .jinit() to initialize the JVM,

.jinit()

Related posts :
RHadoop Installation Issues

Working With RHadoop

RHadoop - rJava installtion Error and solution

previous post - Twitter hashflags for TN2016 election

Installaing RHadoop package. Working with R and Hadoop for large scale data analytics

Install rJava package from local system

install.packages("/home/bdlnn/Downloads/rJava_0.9-7.tar.gz", repos = NULL)

configure: error: Java interpreter '/usr/lib/jvm/default-java/jre/bin/java' does not work ERROR: configuration failed for package "rJava" * removing "/usr/local/lib/R/site-library/rJava"

Solution

Set JAVA_HOME for R

solai@vm1$ sudo R CMD javareconf JAVA_HOME='/home/bdlnn/Software/jdk1.7.0_79'

dont put '/' in end of JAVA_HOME Path

You can also try the alternate options

solai@vm1$ sudo apt-get install r-cran-rjava

inside R
Sys.setenv(JAVA_HOME='/opt/jdk1.7.0_79/jre')

Related posts :
RHadoop Installation Issues

Working With RHadoop

XGBoost in R. Error in xgb.iter.update SoftmaxMultiClassObj

previous post - Top 9 Rule : Code of conduct for Data Science professional

Xgboost is short for eXtreme Gradient Boosting package, XGBoost includes regression, classification and ranking.

Install XGBoost latest version from github

devtools::install_github('dmlc/xgboost',subdir='R-package')

Error

clf<-xgboost(
 data=data.matrix(train[,feature.names]),
        label     = train$Survived,
 booster   = "gblinear",
 nrounds   = 20,
 objective = "multi:softprob",
 num_class = 2,
        eval_metric = "merror"
            )  

Error in xgb.iter.update(bst$handle, dtrain, i - 1, obj) : SoftmaxMultiClassObj: label must be in [0, num_class), num_class=2 but found 2 in label

Solution

needed to convert the response variable from factor to numeric.

train$Survived<-as.numeric(levels(train$Survived))[train$Survived]

also install package "libx11-dev"
solai@vm1$ sudo apt-get install libx11-dev

then try again.

Next :

How to Install rattle package : Graphical User Interface for Data Mining in R

previous post - Apache Spark : steps to Start worker across cluster

step to install rattle package in R

Need to installl "libgtk2.0-dev" package in OS

bigdata@solai:~$ sudo apt-get install libgtk2.0-dev

then install "RGtk2" and "rattle" in R environment
install.packages("RGtk2")
install.packages("rattle")

load the library for the session by
library(rattle)

Next : Code of Conduct for data science Professional

RPostgresql : how to pass dynamic parameter to dbGetQuery statement

previous post - Working with Association in R

Working with RPostgreSQL package

How to pass dynamic / runtime parameter to dbGetQuery in RPostgrSQL ?

#use stri_paste to form a query and pass it into dbGetQuery icd = 'A09'
require(stringi)

qry <- stri_paste("SELECT * FROM visualisation.ipd_disease_datamart WHERE icd ='", icd, "'",collapse="")

rs1 <- dbGetQuery(con, qry)

Error

 
 Error in postgresqlNewConnection(drv, ...) : 
  RS-DBI driver: (cannot allocate a new connection -- 
maximum of 16 connections already opened)

library(RPostgreSQL)
drv <- dbDriver("PostgreSQL")

con <- dbConnect(drv, dbname="DBName", host="127.0.0.1",port=5432,user="yes",password="yes")

Solution
close the connection. max 16 connection can able to establish from R to PostgreSQL, if exceeds this limit, will throw error.

##list all the connections
dbListConnections(drv)
## Closes the connection
dbDisconnect(con)
## Frees all the resources on the driver
dbUnloadDriver(drv) #OR on.exit(dbUnloadDriver(drv), add = TRUE)

How to close/drop all the connection Postgresql session.?

We can terminate the PostgreSQL connection using "pg_terminate_backend" SQL command.
In my case I was open up 16 connection using RPostgreSQL unfortunately forget to release them.
So I ended up with Max. connection exceed limit.

SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE client_addr = '10.184.36.131' and pid > 20613 AND pid <> pg_backend_pid();

In above query, pg_stat_activity will return list of all the active connection.
I have terminating only the connection from R session which made from the (client_addr) IP 10.184.36.181

Next : Apache Spark : start Worker across cluster

Working with Association in R : arules and arulesViz package

previous post - working with RHadoop

working with Association in R using arules and arulesViz packages

when I try to visualize the top five rules,

plot(highLiftRules,method="graph",control=list(type="items"))

Error in as.double(y) : 
  cannot coerce type 'S4' to vector of type 'double'

Solution
load the library "arulesViz" into R session

library(arulesViz)

During the load the library I got error,

Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : namespace "lattice" 0.20-24 is already loaded, but >= 0.20.27 is required Error: package or namespace load failed for "arulesViz"

Solution
error says that i have outed package that need to be upgraded. to know the installed package list

inst = packageStatus()$inst

inst[inst$Status != "ok", c("Package", "Version", "Status")]
#will list out all the package with installed version( not current version)

old.packages()
#list out the installed version and current version of the package.

unloadNamespace("lattice")
#then restart the R session will solve the error.

detach_package("lattice", TRUE)
#will unload the package with out restarting R session

finallly got the output of the plot

Next : working with RPostgreSQL (R and PostgreSQL)

Working with RHadoop

previous post - RHadoop installation issues

working with RHadoop

Error

hdfs.ls("/tamil")

Error in .jcall("java/lang/Class",
 "Ljava/lang/Class;", "forName", cl,  : 

  No running JVM detected. Maybe .jinit() would help. 
Error in .jfindClass(as.character(class)) : 
  No running JVM detected. Maybe .jinit() would help.

Solution

hdfs.init()

hdfs.ls("/")

Error

hdfs.init()
sh: 1: /media/bdalab/bdalab/sw/hadoop-2.7.1/bin: Permission denied
Error in .jnew("org/apache/hadoop/conf/Configuration") : java.lang.ClassNotFoundException In addition: Warning message: running command '/media/bdalab/bdalab/sw/hadoop-2.7.1/bin classpath' had status 126

Solution

Sys.setenv(HADOOP_CMD='/hadoop-2.7.1/bin/hadoop')
Sys.setenv(JAVA_HOME='/jdk1.8.0_60/')
to know the environment varibale
Sys.getenv("HADOOP_CMD")
hdfs.init()

Next :

RHadoop integration isssues

previous post - 3D Data Visualization using R

installaing RHadoop package for working with R and Hadoop

Installing rjava package in R

install.packages("rJava_0.9-7.tar.gz", repos = NULL)

Error

configure: error: Java Development Kit (JDK) is missing or not registered in R Make sure R is configured with full Java support (including JDK). Run R CMD javareconf as root to add Java support to R. If you don't have root privileges, run R CMD javareconf -e to set all Java-related variables and then install rJava. ERROR: configuration failed for package 'rJava' * removing '/home/bdalab/R/x86_64-pc-linux-gnu-library/3.1/rJava'

Solution

install

sudo apt-get install r-cran-rjava

then try again. I have installed succefully in my system. still I got above error, then i checked java -version it was pointing out to openJDK instaed of Oracle java HotSpot. changed to HotSpot.

Error

library(rJava) 
##only in RStudio, In terminal its working fine.

Error : .onLoad failed in loadNamespace() 
for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object
 'x86_64-pc-linux-gnu-library/3.1/rJava/libs/rJava.so':
  libjvm.so: cannot open shared object file:
 No such file or directory
Error: loading failed
Execution halted
ERROR: loading failed

Solution

install

sudo apt-get install r-cran-rjava

Solution

locate libjvm.so and make shared link, sudo ln -s /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so /usr/lib/

Error

R CMD javareconf

/usr/lib/R/bin/javareconf: 405: 
/usr/lib/R/bin/javareconf: cannot create
 /usr/lib/R/etc/Makeconf.new: Permission denied

Solution

sudo -i R CMD javareconf

OR

R CMD javareconf JAVA=jdk1.8.0_60/jre/bin/java JAVA_HOME=jdk1.8.0_60/ JAVAC=jdk1.8.0_60/bin/javac JAR=jdk1.8.0_60/bin/jar JAVAH=jdk1.8.0_60/bin/javah

Next :

3D Data visualization using R - configure: error: X11 not found but required. missing required header GL/gl.h

previous post - Running SQL Query on Hadoop : Apache Hive Alternatives

3D Data Visualization using R : Using the rgl package, rglplot plots a graph in 3D. The plot can be zoomed, rotated, shifted, etc. but the coordinates of the vertices is fixed.

Installing RGL package in R

install.packages("rgl")

Error

configure: using libpng-config configure: using libpng dynamic linkage checking for X... no configure: error: X11 not found but required, configure aborted. ERROR: configuration failed for package ‘rgl’ * removing ‘/home/bdalab/R/x86_64-pc-linux-gnu-library/3.1/rgl’ Warning in install.packages :

Solution

install
sudo apt-get install xorg
sudo apt-get install libx11-dev
then try again.

Error

In my case again I got
configure: error: missing required header GL/gl.h

Solution

install
sudo apt-get install libglu1-mesa-dev
then try again.

Next : RHadoop

Sentiment Analysis : YennaiArinthal Tamil movie review

In this blog will show the sentiment analyis on thala 55, YennaiArinthal movie from twitter #YennaiArinthal

#After succeful authentication, Get the tweet by,

bdalab@bdalabsys:/$ ya = searchTwitter("#YennaiArinthal", n=500, lang="en")

#here I have taken 500 tweet and it was analysed at 12:15 on 05/02/2015

#After necessary data cleansing, use Naive Bayes classifier to into 7 emotion

anger, disgust, fear, joy, sadness, surprise, best_fit

#In the next step we will plot the obtained results (in data frame), using ggplot2

detailed step by step instruction for sentiment Analysis with R will be posted soon,
now my ultimate aim is to show the review graph.

issues : Login in RStudio, SWIRL Package installation and R 3.1.0 installation

Tried to access Rstudio via HTTP using default port 8787. http://localhost:8787.

Error
In my first login i was given root as a username and password. got below error

cannot login to rtudio server on Debian (RStudio initialization error) Unable to connect to service (RStudio initialization error)

Solution
Create new user called ruser.

root@solaiv[bin]# adduser ruser

Adding user `ruser' ...
Adding new group `ruser' (1002) ...
Adding new user `ruser' (1002) with group `ruser' ...
Creating home directory `/home/ruser' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully 

Refreshed the browser and login with newly created user credentials

Error
R version 3.1.0, while configure R

root@solaiv[R-3.1.0]# ./configure

configure: error: --with-readline=yes (default) and headers/libs are not available configure: error: --with-x=yes (default) and X11 headers/libs are not available

Solution
do not need R to be built with this library, you can simply set with-readline to "no":

root@solaiv[R-3.1.0]# ./configure --with-x=no --with-readline=no

Error
while installing "SWIRL" packages

> install.packages("swirl")

Cannot find curl-config
ERROR: configuration failed for package "RCurl"
* removing "/usr/local/lib/R/library/RCurl"
ERROR: dependency "RCurl" is not available for 
package "httr"

* removing "/usr/local/lib/R/library/httr"
ERROR: dependencies "httr", "RCurl" are not 
available for package "swirl"

* removing "/usr/local/lib/R/library/swirl"

Solution

> install.packages("swirl", dependencies=TRUE)

still getting the same error, install and re-try

root@solaiv[R-3.1.0]# apt-get install libcurl4-openssl-dev

root@solaiv[R-3.1.0]# apt-get install libxml2-dev

root@solaiv[R-3.1.0]# R

> install.packages("swirl", dependencies=TRUE)

Related posts

Install R on debian

Install R in Debian squeeze/BOSS OS

Install R Prog. on Debian Squeeze

R is a Free software environment for Data Analysis and Graphics

• Programming Language
• Data Visualization

By default, Debian squeeze comes up with R packages, you can verify by

root@solaiv[~]#apt-cache search ^r-.*

if you were not find anything add an appropriate entry in

root@solaiv[~]#vi /etc/apt/sources.list

the newest R release can be installed using a command sequence like

root@solaiv[~]#apt-get update
root@solaiv[~]#apt-get install r-base
root@solaiv[~]#apt-get install r-base-dev

to enter to R console

root@solaiv[~]#R
>

simple add example in R

> a <- 5;
> b <-10;
> c <- a+b;
> c
[1] 15

R packages may then be installed by the local user/admin from the CRAN source packages, typically from inside R using the working wit addition R packages

Install packages from CRAN

> install.packages("RJDBC")

Install NEW packeges from downloaded file

> install.packages("file.tar.gz")

loading packages into sessions

> library("strings")
> require("strings")

Related posts

Install RStudio
issues cannot login to RStudio server