sparkR.init問題 - Linux

By Charlotte
at 2017-03-31T16:18
at 2017-03-31T16:18
Table of Contents
[sparkR.init問題]
我sparkR初始化上一直出問題,
想請教一下前輩們
[目的]=========================================
用spakR進行平行化計算
[系統環境]=======================================
我在vm上模擬一台master和一台node
沒有裝hadoop直接就裝spark,
版本:
R version 3.2.3 (2015-12-10)
ubuntu:16.04
spark:2.1.0
sparkR:1.4.1
[sparkR安裝]====================================
如下是我的安裝過程(以下只安裝在master)
[Part1.下載套件rJava和devtools]
Step1.設訂R CMD javareconf
sudo JAVA_HOME=/usr/lib/jvm/jdk/ R CMD javareconf
Step2.更新ubuntu函式庫,以下載devtools]
sudo apt-get -y install libcurl4-gnutls-dev libcurl4-openssl-dev libssl-dev
Step3.於R-shell下載rJava和devtools套件
sudo R
install.packages("rJava")
install.packages("devtools")
[Part2.於R-shell用devtools下載sparkR套件、啟動]
devtools::install_github('apache/[email protected]', subdir='R/pkg')
簡單來說,我就只是安裝rJava套件和devtools
[初始化code]====================================
我是在Rstudio server上啟動sparkR的
Sys.setenv(SPARK_HOME='/home/hduser/spark/')
.libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'), .libPaths()))
Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages"
"com.databricks:spark-csv_2.10:1.0.3" "sparkr-shell"')
library(SparkR, lib.loc = "/home/hduser/spark/R/lib/")
library(devtools)
#初始化sparkr為sc
sc <- sparkR.init(master = "local", sparkHome = "/home/hduser/spark")
sqlContext <- sparkRSQL.init(sc)
[錯誤訊息]=======================================
在sc <- sparkR.init(master = "local", sparkHome = "/home/hduser/spark")
這行會出現訊息
17/03/31 02:30:17 ERROR RBackendHandler: createSparkContext on
org.apache.spark.api.r.RRDD failed
java.lang.IllegalArgumentException: Invalid type
.......
Error: returnStatus == 0 is not TRUE
附圖:
https://www.facebook.com/photo.php?fbid=1490142967725208&set=p.1490142967725208&type=3&theater
https://www.facebook.com/photo.php?fbid=1490143787725126&set=p.1490143787725126&type=3&theater
不知道怎麼了!!!QAQ
希望有貴人指點,感謝
--
我sparkR初始化上一直出問題,
想請教一下前輩們
[目的]=========================================
用spakR進行平行化計算
[系統環境]=======================================
我在vm上模擬一台master和一台node
沒有裝hadoop直接就裝spark,
版本:
R version 3.2.3 (2015-12-10)
ubuntu:16.04
spark:2.1.0
sparkR:1.4.1
[sparkR安裝]====================================
如下是我的安裝過程(以下只安裝在master)
[Part1.下載套件rJava和devtools]
Step1.設訂R CMD javareconf
sudo JAVA_HOME=/usr/lib/jvm/jdk/ R CMD javareconf
Step2.更新ubuntu函式庫,以下載devtools]
sudo apt-get -y install libcurl4-gnutls-dev libcurl4-openssl-dev libssl-dev
Step3.於R-shell下載rJava和devtools套件
sudo R
install.packages("rJava")
install.packages("devtools")
[Part2.於R-shell用devtools下載sparkR套件、啟動]
devtools::install_github('apache/[email protected]', subdir='R/pkg')
簡單來說,我就只是安裝rJava套件和devtools
[初始化code]====================================
我是在Rstudio server上啟動sparkR的
Sys.setenv(SPARK_HOME='/home/hduser/spark/')
.libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'), .libPaths()))
Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages"
"com.databricks:spark-csv_2.10:1.0.3" "sparkr-shell"')
library(SparkR, lib.loc = "/home/hduser/spark/R/lib/")
library(devtools)
#初始化sparkr為sc
sc <- sparkR.init(master = "local", sparkHome = "/home/hduser/spark")
sqlContext <- sparkRSQL.init(sc)
[錯誤訊息]=======================================
在sc <- sparkR.init(master = "local", sparkHome = "/home/hduser/spark")
這行會出現訊息
17/03/31 02:30:17 ERROR RBackendHandler: createSparkContext on
org.apache.spark.api.r.RRDD failed
java.lang.IllegalArgumentException: Invalid type
.......
Error: returnStatus == 0 is not TRUE
附圖:
https://www.facebook.com/photo.php?fbid=1490142967725208&set=p.1490142967725208&type=3&theater
https://www.facebook.com/photo.php?fbid=1490143787725126&set=p.1490143787725126&type=3&theater
不知道怎麼了!!!QAQ
希望有貴人指點,感謝
--
Tags:
Linux
All Comments
Related Posts
如何更新filezilla

By Necoo
at 2017-03-31T10:34
at 2017-03-31T10:34
懸賞1000P no working init found

By Charlotte
at 2017-03-30T18:41
at 2017-03-30T18:41
抓不到SDIO網卡

By Regina
at 2017-03-30T11:53
at 2017-03-30T11:53
軟體raid10開機問題

By Tracy
at 2017-03-30T05:18
at 2017-03-30T05:18
播放音樂沒聲音的問題

By Mia
at 2017-03-29T11:59
at 2017-03-29T11:59