Error: must point to main resource (JAR or Python file) - Spark scala
I'm trying to run a simple twitter sentiment analysis code that has worked fine so far, but I don't know what has changed, which is giving me this error. My command line has all the options I need, including --class --master --jars etc. The only thing I did differently was to run the command sudo apt-get install 7-jdk and update the java version. I'm running spark 1.3.1 so this Java update shouldn't be a problem ... I guess. Now, even when I run commands like sbt or sbt build, I get an error because the build is not a know command.
Here's my command line:
./bin/spark-submit --class Sentimenter --master local [4] --jars / home / ubuntu / spark / spark-example-master / target / scala-2.10 / Sentiment_Analysis-assembly-1.0.jar
And here is the result I get:
Error: Must specify the main resource (JAR or Python file) Run with --help for usage help or --verbose for debug output Using Spark default log4j profile: org / apache / spark / log4j-defaults.properties
Any suggestions would be great!
source to share
From spark-submit --help
:
Usage: spark-submit [options] <app jar | python file> [app arguments]
You need to add jar or python file on command line after parameters. So, in your example, you would need to do
./bin/spark-submit --class Sentimenter --master local[4] /home/ubuntu/spark/spark-example-master/target/scala-2.10/Sentiment_Analysis-assembly-1.0.jar
Please note that I removed the --jars
. You can do this to add additional jars (dependencies). If all you have is one can, then the main resource goes to the command line without an option --jars
.
source to share
This may sound ridiculous. but this is my situation.
I had a command line too long, so each parameter is split on different lines to be readable.
when there are too many empty spaces in the argument, I reported this error for sure. so I removed the unnecessary spaces and the error went away.
if you've followed the above suggestion and still have weird bugs, look at your extra whitespace.
source to share