Exception in thread “main“ java.lang.NoClassDefFoundError: org/apache/spark/SparkConf

在本地调试运行spark程序时,报错Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/spark/SparkConf,这个错误就是程序在运行时找不到类Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Spar

飞天小老头

6178人浏览 · 2021-09-18 13:08:29

飞天小老头 · 2021-09-18 13:08:29 发布

在本地调试运行spark程序时,报错Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/spark/SparkConf,这个错误就是程序在运行时找不到类

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
	at cn.jin.spark.JavaLambdaWordCount.main(JavaLambdaWordCount.java:25)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 1 more

这是由于我在pom文件中加入了scope标签,导致程序在编译时忽略了加入scope标签的依赖,所以找不到依赖,只要把对应的scope标签注释掉就可了

<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-core_2.12</artifactId>
   <version>${spark.version}</version>
   <!-- 将下面这条注释掉就可以了 -->
   <!-- <scope>provided</scope> -->
</dependency>

<dependency>
   <groupId>org.scala-lang</groupId>
   <artifactId>scala-library</artifactId>
   <version>${scala.version}</version>
   <!-- <scope>provided</scope> -->
</dependency>

修改后运行成功

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/09/18 13:06:06 INFO SparkContext: Running Spark version 3.0.0
21/09/18 13:06:07 INFO ResourceUtils: ==============================================================
21/09/18 13:06:07 INFO ResourceUtils: Resources for spark.driver:

21/09/18 13:06:07 INFO ResourceUtils: ==============================================================
21/09/18 13:06:07 INFO SparkContext: Submitted application: JavaLambdaWordCount