3 Best Machine Learning Framework for design, experiment and deployment

previous post - Artificial Intelligence #AI for social media

ML / DL framework for deployment and experiment

*) Listed framework is opensource

*) It provides dashboard

*) Configure, organize, log and reproduce experiments

*) make prototyping experiments easier

Machine learning life cycle framework

List of opensource ML framework

*) MLFlow
    ** An open source platform for the complete machine learning lifecycle

    ** It currently offers three components: MLflow Tracking, MLflow Projects, MLflow Models

---

*) Sacred
    ** configure, organize, log and reproduce experiments

    ** easily run your experiment for different settings

---

*) FGLab
    ** machine learning dashboard, designed to make prototyping experiments easier.

    ** here are 3 ways to run FGLab: Installing locally, via Docker, or hosted on Heroku.

---

If anything can be add / missed out, mention in the comments section.

In next blog will discuss possible use-case from Education AREAS FOR AI INTERVENTION.

Next :

Big Data / Data Analytics Job

Data Architect   3-7+ years   

Salary : £70000 - £75000 per annum     location : Central London

Posted: 26/06/2018    Job Type : Permanent

Job Ref: 439101143    Skills & Apply ->

Data Scientist   3-7+ years   

Salary : £232 - £298 per day     location : London

Posted: 18/06/2018    Job Type : Temporary

Job Ref: 775109025    Skills & Apply ->

crowd sourced Tamil Movie review - Annadurai

Tweet Summary - #annadurai

total tweet mined - 1958

| Original tweet - 435

Top 5 mention with #annadurai

@vijayantony - 165

|

@mrsvijayantony - 66

|

@realradikaa - 53

@dianachampika - 33

|

@vijayantonyfilm - 28

Top 5 hastag associated #annadurai

#VijayAntony - 354

|

#ndrasena - 41

|

#Thiruttuppayale2 - 30

#DianaChampika - 11

|

#mersal - 10

popular tweet - #annadurai

@avardhaar ::After #Thalapathy #Vijay 's #Mersal Only #VijayAntony 's #Annadurai \nGST Song became Popular - Annadurai Heroine https://t.co/FJ6P98b3c3

@LMKMovieManiac ::#Annadurai 1st half - Has an interesting, attention-grabbing surprise element at the interval block! \nThe film is c https://t.co/kRkeGEgx4q

@sri50 ::#Annadurai is a mass commercial @vijayantony template film in which he impresses in a dual role as identical twins https://t.co/jrYhRQ4aOL

crowd sourced #annadurai - sentiment

Weka Hadoop Integration - weka read/write data from HDFS

previous post - Top 10 Deep Learning Tools

Weka Hadoop Integration using distributedWekaHadoop package

In weka, Tools --> PackageManager search "distributedWekaHadoop" and install the packages.

Now go back to your weka KnowledgeFlow, you can find HDFSLoader and HDFSSaver in DataLoder and DataSink portion.

by using which you can read / write data to / from HDFS

Related posts :

Big Data / Data Analytics Jobs

RPostgreSQL Data analytics on PostgreSQL data using R

previous post - XGBoost in R. Error in xgb.DMatrix can not open file

Data analytics on PostgreSQL data using R. Working with R and PostgreSQL for large scale data analytics

Installing RPostgreSQL

install.packages('RPostgreSQL')

compilation terminated. /usr/lib/R/etc/Makeconf:128: recipe for target 'RS-PQescape.o' failed make: *** [RS-PQescape.o] Error 1 ERROR: compilation failed for package 'RPostgreSQL' * removing '/home/boss/R/x86_64-pc-linux-gnu-library/3.1/RPostgreSQL'
RS-PostgreSQL.h:23:26: fatal error: libpq-fe.h: No such file or directory

Solution

RPostgreSQL require libpq-dev.

libpq-dev is a set of library functions that allow client programs (for our case its 'R') to pass queries to the PostgreSQL backend server and to receive the results of these queries.

So we have to install libpg-dev on OS not from R terminal

solai@server# sudo apt-get install libghc-postgresql-libpq-dev

then install RPostgreSQL inside R session
> install.packages('RPostgreSQL')

Related posts :
RPostgresql : how to pass dynamic parameter to dbGetQuery

Working With RHadoop

RHadoop - Running Hadoop MapReduce from R

previous post - RHadoop - rJava Installation Error

Running Hadoop Mapreduce From R. Working with R and Hadoop for large scale data analytics

Running Mapreduce job

m <- mapreduce(InputPath, input.format = 'csv', map = mapFunction, reduce = reduceFunction )

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1

Solution

Error seems map not find the file location,
I was trying to reading File from local but by backend configured with Local

rmr.options(backend = "local")

Reading File from Hadoop in R session

f = hdfs.line.reader("/HDFS/GdpData.csv");

Error in .jcheck(silent = TRUE) :
No running JVM detected. Maybe .jinit() would help.

Solution

Run .jinit() to initialize the JVM,

.jinit()

Related posts :
RHadoop Installation Issues

Working With RHadoop

Top 9 Rule : Code of conduct for Data Science professional

previous post - Rattle : GUI for R

Top 9 Rule : Code of conduct for Data Science professional

Rule 1 - know the Terminology

Data Scientist - Client Relationship
Rule 2 - Competence
Rule 3 - Scope of Data Science Professional Services Between Client and Data Scientist
Rule 4 - Communication with Clients
Rule 5 - Confidential Information
Rule 6 - Conflicts of Interest
Rule 7 - Duties to Prospective Client
Rule 8 - Data Science Evidence, Quality of Data and Quality of Evidence

Maintaining the Integrity of the Data Science Profession
Rule 9 - Misconduct

For more details report

Next : XGBoost in R. Error in xgb.iter.update SoftmaxMultiClassObj

RPostgresql : how to pass dynamic parameter to dbGetQuery statement

previous post - Working with Association in R

Working with RPostgreSQL package

How to pass dynamic / runtime parameter to dbGetQuery in RPostgrSQL ?

#use stri_paste to form a query and pass it into dbGetQuery icd = 'A09'
require(stringi)

qry <- stri_paste("SELECT * FROM visualisation.ipd_disease_datamart WHERE icd ='", icd, "'",collapse="")

rs1 <- dbGetQuery(con, qry)

Error

 
 Error in postgresqlNewConnection(drv, ...) : 
  RS-DBI driver: (cannot allocate a new connection -- 
maximum of 16 connections already opened)

library(RPostgreSQL)
drv <- dbDriver("PostgreSQL")

con <- dbConnect(drv, dbname="DBName", host="127.0.0.1",port=5432,user="yes",password="yes")

Solution
close the connection. max 16 connection can able to establish from R to PostgreSQL, if exceeds this limit, will throw error.

##list all the connections
dbListConnections(drv)
## Closes the connection
dbDisconnect(con)
## Frees all the resources on the driver
dbUnloadDriver(drv) #OR on.exit(dbUnloadDriver(drv), add = TRUE)

How to close/drop all the connection Postgresql session.?

We can terminate the PostgreSQL connection using "pg_terminate_backend" SQL command.
In my case I was open up 16 connection using RPostgreSQL unfortunately forget to release them.
So I ended up with Max. connection exceed limit.

SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE client_addr = '10.184.36.131' and pid > 20613 AND pid <> pg_backend_pid();

In above query, pg_stat_activity will return list of all the active connection.
I have terminating only the connection from R session which made from the (client_addr) IP 10.184.36.181

Next : Apache Spark : start Worker across cluster

Working with RHadoop

previous post - RHadoop installation issues

working with RHadoop

Error

hdfs.ls("/tamil")

Error in .jcall("java/lang/Class",
 "Ljava/lang/Class;", "forName", cl,  : 

  No running JVM detected. Maybe .jinit() would help. 
Error in .jfindClass(as.character(class)) : 
  No running JVM detected. Maybe .jinit() would help.

Solution

hdfs.init()

hdfs.ls("/")

Error

hdfs.init()
sh: 1: /media/bdalab/bdalab/sw/hadoop-2.7.1/bin: Permission denied
Error in .jnew("org/apache/hadoop/conf/Configuration") : java.lang.ClassNotFoundException In addition: Warning message: running command '/media/bdalab/bdalab/sw/hadoop-2.7.1/bin classpath' had status 126

Solution

Sys.setenv(HADOOP_CMD='/hadoop-2.7.1/bin/hadoop')
Sys.setenv(JAVA_HOME='/jdk1.8.0_60/')
to know the environment varibale
Sys.getenv("HADOOP_CMD")
hdfs.init()

Next :

RHadoop integration isssues

previous post - 3D Data Visualization using R

installaing RHadoop package for working with R and Hadoop

Installing rjava package in R

install.packages("rJava_0.9-7.tar.gz", repos = NULL)

Error

configure: error: Java Development Kit (JDK) is missing or not registered in R Make sure R is configured with full Java support (including JDK). Run R CMD javareconf as root to add Java support to R. If you don't have root privileges, run R CMD javareconf -e to set all Java-related variables and then install rJava. ERROR: configuration failed for package 'rJava' * removing '/home/bdalab/R/x86_64-pc-linux-gnu-library/3.1/rJava'

Solution

install

sudo apt-get install r-cran-rjava

then try again. I have installed succefully in my system. still I got above error, then i checked java -version it was pointing out to openJDK instaed of Oracle java HotSpot. changed to HotSpot.

Error

library(rJava) 
##only in RStudio, In terminal its working fine.

Error : .onLoad failed in loadNamespace() 
for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object
 'x86_64-pc-linux-gnu-library/3.1/rJava/libs/rJava.so':
  libjvm.so: cannot open shared object file:
 No such file or directory
Error: loading failed
Execution halted
ERROR: loading failed

Solution

install

sudo apt-get install r-cran-rjava

Solution

locate libjvm.so and make shared link, sudo ln -s /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so /usr/lib/

Error

R CMD javareconf

/usr/lib/R/bin/javareconf: 405: 
/usr/lib/R/bin/javareconf: cannot create
 /usr/lib/R/etc/Makeconf.new: Permission denied

Solution

sudo -i R CMD javareconf

OR

R CMD javareconf JAVA=jdk1.8.0_60/jre/bin/java JAVA_HOME=jdk1.8.0_60/ JAVAC=jdk1.8.0_60/bin/javac JAR=jdk1.8.0_60/bin/jar JAVAH=jdk1.8.0_60/bin/javah

Next :

Running SQL Query on Hadoop : Apache Hive Alternatives

previous post - Mahout 0.11.0 on Spark1.4.1

Hive is the SQL programmer friendly tool for running SQL query on Hadoop HDFS File system. While running query Hive will convert SQL like query into MapReduce.

Hive is not the only tool will do the same. This post will let give synopsis on open source alternative of Hive.

1) spark sql (previously Shark - Sql on Spark) - will be the best alternative of Hive over Spark. Spark SQL is Spark's module for working with structured data.

2) Cloudera Impala - like Hive but it uses its own execution daemons which we need to install every datanodes in Hadoop cluster. Impala do BI-style Queries on Hadoop.

3) Facebook Presto - like Impala need to install all datanodes. Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.

4) Apache Drill - Schema free SQL for Hadoop. It support multiple datastores HDFS, MongoDB and Hbase

Next : 3D visualization using R

Hive configuration at Zeppelin - error and way to solve

previous post - Configuring Hive with Hue (Hadoop UI)

I have successfully compliled latest version of Zeppelin from source. You can get Apache Zeppelin binary on request, give your mail id, I will provide link to Download.

hive interpreter not available in my Apache Zeppelin what should i do?

Install Hive and Set HIVE_HOME.

if you have installed Hive in your system, Zeppelin will automatically recognize and list the Hive interpreter. if you have installed Hive but no interpreter listed out in Zeppelin, check HIVE_HOME and set it properly

If you have not installed Hive, install Hive it will take automaticall by next restart

Error

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

Solution

Add export HADOOP_USER_CLASSPATH_FIRST=true at HIVE_HOME/conf/ hive-env.sh

Next :

Configuring Hive at HUE (Hadoop ui)

previous post - Hue installation error

I have successfully compliled latest version of Hue from source. when i was access by http://localhost:8000/ have seen warning Hive editor not configured.

Hive Editor - The application won't work without a running HiveServer2.

to start Hive

bdalab@solai:/opt$ $HIVE_HOME/bin/hive --service hiveserver2

in another terminal

bdalab@solai:/opt$ $HIVE_HOME/bin/beeline -u jdbc:hove2//localhost:10000

i have got permission denied error Error

Error: Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=APP, access=EXECUTE, inode="/tmp/hive/APP/704d9b8d-a9d5-4be6-a8f1-082e8eba9a0c":hdfs:supergroup:drwx------

Solution

then I have changed derby (javax.jdo.option.ConnectionUserName ) username From APP to hdfs in hive_default.xml under HIVE_HOME/conf

Next : Hive with Zeppelin

Hue installation Error

previous post - Hive dynamic partition issues

I have been installing recent version of HUE from source, During compilation i have got few issues. here is the step to resolve the issues.

Error

    Unable to get dependency information: Unable to read the metadata file for artifact 'com.sun.jersey:jersey-core:jar': Cannot find parent: net.java:jvnet-parent for project: com.sun.jersey:jersey-project:pom:1.9 for project com.sun.jersey:jersey-project:pom:1.9 com.sun.jersey:jersey-core:jar:1.9

....
Path to dependency: 1) com.cloudera.hue:hue-plugins:jar:3.8.1-SNAPSHOT
2) org.apache.hadoop:hadoop-client:jar:2.6.0-mr1-cdh5.5.0-SNAPSHOT
3) org.apache.hadoop:hadoop-hdfs:jar:2.6.0-cdh5.5.0-SNAPSHOT

Solution

update maven, by default mvn -version shows 2.x, but hue needs maven version 3 and above.
Download , extract and install Maven3 (if not already installed) and set maven in PATH

export PATH=$PATH:$MAVEN_HOME/bin

even i have updated the PATH variable with latest maven. mvn -version shows 2.x
now i manually updated the maven using update-alternatives.

  bdalab@solai:/opt$ sudo update-alternatives --install /usr/bin/mvn mvn $MAVEN_HOME/bin/mvn 1

  bdalab@solai:/opt$ sudo update-alternatives --config mvn

Now, select a number referring to the recent maven3 installation, from the list of choices

Error

    Failed to obtain user group information: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: hue (error 401)

Solution

Configure Hue as a proxy user for all other users and groups, meaning it may submit a request on behalf of any other user. Add below property to core-site.xml within the configuration tags:
<property> <name>hadoop.proxyuser.hue.hosts</name> <value>*</value>
</property>
<property> <name>hadoop.proxyuser.hue.groups</name> <value>*</value>
</property>

Next : Configuring Hive at Hue

4 best tools for Big Data visualization

Vizuvalization will play the major role in big data data analytics. Human role in
visuvalization are limited to
      Identify the visual patterns and anomalies
      Seeing pattern across groups.
Once you have done big data analytics using your favourite tools (Hadoop,
Spark or Machine learning), next to impress customer by dashbord/graphcs in order to making better business decision.

Big Data visualization tools From Apache

zeppelin

kylin

note : both are now part of apache incubator project

Other interesting tools

lumify

Gephi

I had experience with gephi. Now working with apache incubator Zeppelin and Kylin. Will update further with working model.

Configure UBER mode - MapReduce job for small dataset

previous post - what is Uber mode

How to configure uber job.?

    To enable uber jobs, need to set the following property in yarn-site.xml.
    mapreduce.job.ubertask.enable=true
    mapreduce.job.ubertask.maxmaps=9 (default 9)
    mapreduce.job.ubertask.maxreduces=0 (default 1)
    mapreduce.job.ubertask.maxbytes=4096

mapreduce.job.ubertask.maxbytes

    above value for 4MB, default value = bloksize. The total input size of a job must be less then or equal to this value for the job to be uberized.
   Ex. say if you have data set which is 5MB of size, but you have set 4MB for mapreduce.job.ubertask.maxbytes, then uber mode will not set.
    If you omit this, by default bloksize value assigned (12MB). If you are going to run a dataset of size 50MB will not set in uber mode.)

UBER mode in YARN Hadoop2 - Running MapReduce jobs in small dataset

You might have seen these lines while running MapReduce in Hadoop2. mapreduce.Job: Job job_1387204213494_0005 running in uber mode : false

what is UBER mode in Hadoop2?

    In normally mappers and reducers will run by ResourceManager (RM), RM will create separate container for mapper and reducer.
    uber configuration, will allow to run mapper and reducers in the same process as the ApplicationMaster (AM).

Uber jobs :

    Uber jobs are jobs that are executed within the MapReduce ApplicationMaster. Rather then communicate with RM to create the mapper and reducer containers.
    The AM runs the map and reduce tasks within its own process and avoided the overhead of launching and communicate with remote containers.

Why

    If you have a small no dataset, want to run MapReduce on small amount of data. Uber configuration will help you out, by reducing additional time that MapReduce normally spends mapper and reducers phase.

Can I configure/have a Uber for all MapReduce job.?

    As of now,
       map-only jobs
       jobs with one reducer are supported.

Next How to configure uber job.?

Easy way to recover the deleted files/dir in Hadoop hdfs

In some cause, accidently files or dir will be deleted,
Is there any way to recover to get back.??

   By default Hadoop will delete the files/dir forever. It has Trash feature, which is not enabled by default.

   By configuring #fs.trash.interval and #fs.trash.checkpoint.interval in Hadoop core-site.xml will move the deleted files/dir into .Trash folder.

   location of .Trash folder is in HDFS /user/$USER/.Trash

configuring core-site.xml

<property>
<name>fs.trash.interval</name>
<value>120</value> 
</property>

<property>
<name>fs.trash.checkpoint.interval</name>
<value>45</value>
</property>

   In above configuration, all the deleted files / dir will me moved to .Trash folder and keep the data for two hours.
   checkpoint intervel check will performed for every 45 min and deletes all the file/dir which more then 2 hours old from .Trash folder.

restart the hadoop

    once you modify the core-site.xml , stop and start the hadoop

   here is the example of remove dir command

hadoop@solai# hadoop fs -rmr /testTrash
15/04/05 01:10:14 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 120 minutes, Emptier interval = 45 minutes. Moved: 'hdfs://127.0.0.1:9000/testTrash' to trash at: hdfs://127.0.0.1:9000/user/bdalab/.Trash/Current

   you can clearly get the message say that deletd folder will be moved to /user/bdalab/.Trash/Current and will keep the data for 2 hours with check point interval 45 min.

list the deleted files/dir in .Trash folder using -ls

hadoop@solai# hadoop fs -ls hdfs://127.0.0.1:9000/user/bdalab/.Trash/Current/testTrash

you can view the content (by -cat) or move the files to original path.

hadoop@solai# hadoop fs -mv hdfs://127.0.0.1:9000/user/bdalab/.Trash/Current/testTrash /testTrash

Hadoop 2.5.x/2.6.x Multi Node cluster setup on Ubuntu / lubuntu

Setup Apache Flink on cluster mode - Ubuntu

Thus is continue of previous post installing Flink on Local , In this blog will see how to Setup Apache Flink on Cluster with Hadoop, once it's done will Execute / Run Flink job on the files which is stored in HDFS.

If you are new to Hadoop, find here to setting up Hadoop Cluster

#start Hadoop cluster- HDFS

bdalab@bdalabsys: HADOOP_HOME/$ ./sbin/start-dfs.sh

#start Hadoop cluster- YARN MR2

bdalab@bdalabsys: HADOOP_HOME/$ ./sbin/start-yarn.sh

make sure all the Hadoop daemons up and running

#Download the latest Flink (matching to your Hadoop version) and un-tar the file.

bdalab@bdalabsys:/$ tar -xvzf flink-0.8-incubating-SNAPSHOT-bin-hadoop2.tgz

#rename the folder

bdalab@bdalabsys:/$ mv flink-0.8-incubating-SNAPSHOT/ flink-0.8

#move the working dir into flink_home

bdalab@bdalabsys:/$ cd flink-0.8

#similar to the HDFS configuration, edit the file $FLINK_HOME/conf/slaves and
enter the IP/Host name of each worker node.

#Enable Password less ssh from master to all the slave's

bdalab@bdalabsys:flink-0.8/$ ssh-keygen -t rsa -P ""
bdalab@bdalabsys:flink-0.8/$ ssh-copy-id -i /home/bdalab/.ssh/id_dsa.pub bdalab@slave1

repaet the last step to as many slave mentioned in conf/slaves file.

#run flink on cluster mode

bdalab@bdalabsys:flink-0.8/$ ./bin/start-cluster.sh
....
Starting job manager
Starting task manager on host bdalabsys
.....

#JobManager will started by above command. check the status by

bdalab@bdalabsys:flink-0.8/$ jps
6740 Jps
6725 JobManager
6895 TaskManager

Flink cluster mode will work on both local / HDFS. If you want to process the
data from HDFS, make sure all the HDFS daemons are up&running .