ClassNotFoundException: org.apache.spark.SparkConf với tia lửa trên hive

Question

Tôi đang cố gắng sử dụng SPARK như động cơ thực thi hive, nhưng nhận được lỗi dưới đây. Spark 1.5.0 được cài đặt và tôi đang làm việc với phiên bản Hive 1.1.0 với phiên bản Hadoop 2.7.0.

hive_emp bảng được tạo dưới dạng bảng định dạng ORC trong tổ ong.

hive (Koushik)> insert into table hive_emp values (2,'Koushik',1); 
Query ID = hduser_20150921072727_feba8363-258d-4d0b-8976-662e404bca88 
Total jobs = 1 
Launching Job 1 out of 1 
In order to change the average load for a reducer (in bytes): 
    set hive.exec.reducers.bytes.per.reducer=<number> 
In order to limit the maximum number of reducers: 
    set hive.exec.reducers.max=<number> 
In order to set a constant number of reducers: 
    set mapreduce.job.reduces=<number> 
java.lang.NoClassDefFoundError: org/apache/spark/SparkConf 
    at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.generateSparkConf(HiveSparkClientFactory.java:140) 
    at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:56) 
    at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) 
    at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) 
    at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) 
    at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) 
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) 
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) 
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638) 
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397) 
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) 
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) 
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039) 
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207) 
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159) 
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370) 
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754) 
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) 
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:601) 
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356) 
    ... 25 more 
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. org/apache/spark/SparkConf 

Tôi cũng đặt đường dẫn tia lửa và công cụ thực thi trong vỏ hive.

hduser@ubuntu:~$ spark-shell 
    Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 1.5.0 
     /_/ 

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_21) 
Type in expressions to have them evaluated. 
Type :help for more information. 
Spark context available as sc. 
SQL context available as sqlContext. 

scala> exit; 
warning: there were 1 deprecation warning(s); re-run with -deprecation for details 
hduser@ubuntu:~$ hive 
SLF4J: Class path contains multiple SLF4J bindings. 
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 
SLF4J: Class path contains multiple SLF4J bindings. 
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 

Logging initialized using configuration in file:/usr/lib/hive/conf/hive-log4j.properties 
hive (default)> use Koushik; 
OK 
Time taken: 0.593 seconds 
hive (Koushik)> set spark.home=/usr/local/src/spark; 

Tôi cũng đã tạo ra một .hiverc như sau

hduser@ubuntu:/usr/lib/hive/conf$ cat .hiverc 
SET hive.cli.print.header=true; 
set hive.cli.print.current.db=true; 
set hive.auto.convert.join=true; 
SET hbase.scan.cacheblock=0; 
SET hbase.scan.cache=10000; 
SET hbase.client.scanner.cache=10000; 
SET hive.execution.engine=spark; 

chế độ DEBUG chi tiết lỗi đưa ra dưới đây:

hduser@ubuntu:~$ hive 
SLF4J: Class path contains multiple SLF4J bindings. 
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 
SLF4J: Class path contains multiple SLF4J bindings. 
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 

Logging initialized using configuration in file:/usr/lib/hive/conf/hive-log4j.properties 
hive (default)> use Koushik; 
OK 
Time taken: 0.625 seconds 
hive (Koushik)> set hive --hiveconf hive.root.logger=DEBUG 
       > ; 
hive (Koushik)> set hive.execution.engine=spark; 
hive (Koushik)> desc hive_emp; 
OK 
col_name data_type comment 
empid     int           
empnm     varchar(50)         
deptid     int           
Time taken: 0.173 seconds, Fetched: 3 row(s) 
hive (Koushik)> select * from hive_emp; 
OK 
hive_emp.empid hive_emp.empnm hive_emp.deptid 
Time taken: 1.689 seconds 
hive (Koushik)> insert into table hive_emp values (2,'Koushik',1); 
Query ID = hduser_20151015112525_c96a458b-34f8-42ac-ab11-52c32479a29a 
Total jobs = 1 
Launching Job 1 out of 1 
In order to change the average load for a reducer (in bytes): 
    set hive.exec.reducers.bytes.per.reducer=<number> 
In order to limit the maximum number of reducers: 
    set hive.exec.reducers.max=<number> 
In order to set a constant number of reducers: 
    set mapreduce.job.reduces=<number> 
java.lang.NoSuchMethodError: org.apache.spark.scheduler.LiveListenerBus.addListener(Lorg/apache/spark/scheduler/SparkListener;)V 
    at org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient.<init>(LocalHiveSparkClient.java:85) 
    at org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient.getInstance(LocalHiveSparkClient.java:69) 
    at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:56) 
    at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) 
    at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116) 
    at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113) 
    at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95) 
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) 
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) 
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638) 
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397) 
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) 
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) 
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039) 
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207) 
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159) 
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370) 
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754) 
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) 
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:601) 
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. org.apache.spark.scheduler.LiveListenerBus.addListener(Lorg/apache/spark/scheduler/SparkListener;)V 
hive (Koushik)> 

Tôi đã thực hiện chèn trên hai lần và cả hai lần nó không thành công . Vui lòng tìm hive.log được tạo hôm nay. hive.log

Answer 1

Lý do cho lỗi này là tổ ong không thể tìm thấy bình lắp ráp tia lửa.

xuất SPARK_HOME =/usr/local/src/spark hoặc thêm bình lắp ráp tia lửa trong thư mục lib hive. Vấn đề này sẽ được giải quyết.

Answer 2

Tôi cũng đang gặp phải sự cố tương tự trên my Ubuntu 14.4 VitualBox. Dưới đây là các bước tôi đã theo sau để khắc phục:

1. hive> set spark.home=/usr/local/spark;

2. hive> set spark.master=local;

3. hive> SET hive.execution.engine=spark;

4. Added spark-assembly jar tập tin như hình dưới đây:

   hive> ADD jar /usr/local/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar;