How do I create Spark 1.1.0 on Windows 8?
I am trying to build Apache Spark 1.1.0 on Windows 8.
I installed all prerequisites (except Hadoop) and ran sbt/sbt assembly
in the root directory. After downloading many files, I get an error after the line:
Set current project to root <in build file:C:/.../spark-0.9.0-incubating/>". The error is:
[error] Not a valid command: /
[error] /sbt
[error] ^
How do I create Spark on Windows?
NOTE Please see my comments on the version differences.
The error Not a valid command: /
comes from sbt which was executed and tried to execute the command /
(as the first character in the line /sbt
). This could mean that you have a sbt
shell script available in PATH
(possibly installed separately outside of the current working directory) or in the current working directory.
Just follow through sbt assembly
and it should fix Spark well.
According to the Spark homepage :
If you like building Spark from scratch, visit Spark Building with Maven .
which clearly states that the official Spark build tool is now Maven (unfortunately).
You can create a Spark package with the following command:
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
This worked for me.