How do I access the static resources in the jar (which correspond to the src / main / resources folder)?
I have a Spark Streaming application built with Maven (like a jar) and deployed with a spark-submit
script. The app project layout follows the standard catalog layouts:
myApp
src
main
scala
com.mycompany.package
MyApp.scala
DoSomething.scala
...
resources
aPerlScript.pl
...
test
scala
com.mycompany.package
MyAppTest.scala
...
target
...
pom.xml
In an object DoSomething.scala
, I have a method (let it call it doSomething()
) that tries to execute a Perl script - aPerlScript.pl
(from a folder resources
) - using scala.sys.process.Process
and passing two script arguments (the first is the absolute path to the binary used as input, the second is the path / name generated output file). I call then DoSomething.doSomething()
.
The problem is I was unable to access the script, not absolute paths, relative paths, getClass.getClassLoader.getResource, getClass.getResource, I specified the resources folder in mine pom.xml
. None of my attempts have succeeded. I don't know how to find the stuff I have put in src / main / resources.
Any help would be appreciated.
SIDE NOTES:
- I am using an external Process instead of Spark because at this point in my workflow, I have to process binaries as input and output.
- I am using Spark-streaming 1.1.0, Scala 2.10.4 and Java 7. I am creating a jar with "Maven" installation from Eclipse (Kepler)
- When I use the
getClass.getClassLoader.getResource
"standard" method to access resources, I find that the actual classpath is aspark-submit
script.
source to share
There are several solutions. The simplest is to use the Scala process framework:
import scala.sys.process._
object RunScript {
val arg = "some argument"
val stream = RunScript.getClass.getClassLoader.getResourceAsStream("aPerlScript.pl")
val ret: Int = (s"/usr/bin/perl - $arg" #< stream).!
}
In this case ret
- the return code for the process, and any process concluded - stdout
.
The second (longer) solution is to copy the file aPerlScript.pl
from the jar file to some temporary location and execute it from there. This piece of code should have most of what you need.
object RunScript {
// Set up copy destination from the Java temporary directory. This is /tmp on Linux
val destDir = System.getProperty("java.io.tmpdir") + "/"
// Get a stream to the script in the resources dir
val source = Channels.newChannel(RunScript.getClass.getClassLoader.getResourceAsStream("aPerlScript.pl"))
val fileOut = new File(destDir, "aPerlScript.pl")
val dest = new FileOutputStream(fileOut)
// Copy file to temporary directory
dest.getChannel.transferFrom(source, 0, Long.MaxValue)
source.close()
dest.close()
}
// Schedule the file for deletion for when the JVM quits
sys.addShutdownHook {
new File(destDir, "aPerlScript.pl").delete
}
// Now you can execute the script.
This approach allows you to link your own libraries in JAR files. Copying them allows the libraries to be loaded at runtime for whatever JNI hint you have planned.
source to share