Clever Geek Handbook

NoClassDefFound: Scala / xml / metadata

I am running a simple spark program in Java (IDE: Eclipse Luna, Maven).

My example program

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;

public class testSpark {
    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("Testing").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);

        System.out.println(sc.appName());
    }
}

But I am getting error when I run my example program

Exception on thread "main" java.lang.NoClassDefFoundError: scala / xml / MetaData at org.apache.spark.ui.jobs.JobsTab. (JobsTab.scala: 30) at org.apache.spark.ui.SparkUI.initialize (SparkUI.scala: 50) at org.apache.spark.ui.SparkUI. (SparkUI.scala: 61) at iScope.testSpark.main (testSpark.java:9)

Caused by: java.lang.ClassNotFoundException: scala.xml.MetaData at java.net.URLClassLoader.findClass (Unknown source) at java.lang.ClassLoader.loadClass (Unknown source)

My pom.xml file

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>18.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>1.2.1</version>
        </dependency>
+3


source to share


4 answers


SparkUI seems to use the scala package, try adding this dsependency to your pom file to put the scala.xml package on your classpath.



<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-xml</artifactId>
    <version>2.11.0-M4</version>
</dependency>
+7


source


Adding the following dependency fixed the problem for me.



<dependency>
    <groupId>org.scala-lang.modules</groupId>
    <artifactId>scala-xml_2.11</artifactId>
    <version>1.0.6</version>
</dependency>
+1


source


you need to add the class file or .jar file that contains this class to the java path.

0


source


As the exception stack trace clearly shows, the problem occurs when SparkUI tries to create the Jobs tab. To do this, the JobsTab ( org.apache.spark.ui.jobs.JobsTab

) class tries to create a page ( org.apache.spark.ui.jobs.JobPage

) and attach it to itself. If you look into the JobPage source code, you will notice that it makes heavy use of scala.xml ( Scala's XML standard library ), which you are probably missing.

As other contributors have already pointed out, adding the scala.xml library to your list of dependencies should fix the problem. At the time of this writing, the latest version is 1.2.0 for Scala 2.13 (you can check for updates in the Maven repository ), so:

Maven:
<dependency>
    <groupId>org.scala-lang.modules</groupId>
    <artifactId>scala-xml_2.13</artifactId>
    <version>1.2.0</version>
</dependency>

Gradle:
compile group: 'org.scala-lang.modules', name: 'scala-xml_2.13', version: '1.2.0'

SBT:
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.2.0"
0


source






More articles:


All Articles