Error starting sbt build: sbt release failed
I ran into the exact problem as described in the next article and the suggested answer does not help. sbt-assembly: Deduplication error encountered
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.transaction\orbits\javax.transaction-1.1.1.v201105210645.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.servlet\orbits\javax.servlet-3.0.0.v201112011016.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.mail.glassfish\orbits\javax.mail.glassfish-1.4.1.v201005082020.jar:META-INF/ECLIPSEF.RSA
[error] C:\Users\xxx\.ivy2\cache\org.eclipse.jetty.orbit\javax.activation\orbits\javax.activation-1.1.0.v201105071233.jar:META-INF/ECLIPSEF.RSA
[error] Total time: 14 s, completed Sep 9, 2014 5:21:01 PM
My build.sbt file contains
name := "Simple"
version := "0.1.0"
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.twitter4j" % "twitter4j-stream" % "3.0.3"
)
//libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.2"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.0.2"
libraryDependencies += "org.apache.spark" %% "spark-streaming-twitter" % "1.0.2"
libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "0.4.2"
libraryDependencies ++= Seq(
("org.apache.spark"%%"spark-core"%"1.0.2").
exclude("org.eclipse.jetty.orbit", "javax.servlet").
exclude("org.eclipse.jetty.orbit", "javax.transaction").
exclude("org.eclipse.jetty.orbit", "javax.mail").
exclude("org.eclipse.jetty.orbit", "javax.activation").
exclude("commons-beanutils", "commons-beanutils-core").
exclude("commons-collections", "commons-collections").
exclude("commons-collections", "commons-collections").
exclude("com.esotericsoftware.minlog", "minlog")
)
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
case PathList("javax", "transaction", xs @ _*) => MergeStrategy.first
case PathList("javax", "mail", xs @ _*) => MergeStrategy.first
case PathList("javax", "activation", xs @ _*) => MergeStrategy.first
case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x => old(x)
}
}
Any pointers on how to fix the above issue?
source to share
If you plan to run your program from Spark, I highly recommend adding all Spark dependencies as provided
such so that they are excluded from the build task.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.0.2" % "provided",
"org.apache.spark" %% "spark-streaming" % "1.0.2" % "provided",
"org.apache.spark" %% "spark-streaming-twitter" % "1.0.2" % "provided")
Otherwise you need to either remove those jar
from the classpath or add the corresponding lines in mergeStrategy
, in your case it would be
case PathList("META-INF", "ECLIPSEF.RSA") => MergeStrategy.first
If you still want to deal with Spark dependencies, the sbt-dependency-graph plugin should help. Also note that other Spark dependencies like spark-streaming
and spark-streaming-twitter
probably need a directive as well exclude
.
source to share
So to get away with the annoying "deduplicating" messages, I didn't bother with the exception, it didn't seem to work for me. I copied and pasted defaultMergeStrategy
from the sbt code and just changed the line where it says deduplicate
- first
. I also had to add a trick at the end to insist on first
too. To be honest, I have no idea what this means or why it is causing annoying messages. I don't have time to get Phd in sbt, I want my code to just build !! Thus, the merge will look like this:
mergeStrategy in assembly <<= (mergeStrategy in assembly) ((old) => {
case x if Assembly.isConfigFile(x) =>
MergeStrategy.concat
case PathList(ps @ _*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
MergeStrategy.rename
case PathList("META-INF", xs @ _*) =>
(xs map {_.toLowerCase}) match {
case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
MergeStrategy.discard
case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
MergeStrategy.discard
case "plexus" :: xs =>
MergeStrategy.discard
case "services" :: xs =>
MergeStrategy.filterDistinctLines
case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
MergeStrategy.filterDistinctLines
case _ => MergeStrategy.first // Changed deduplicate to first
}
case PathList(_*) => MergeStrategy.first // added this line
})
source to share