How to run spring boot application on Spark cluster
We have created a java spark java spring boot application.
./spark-submit --class com.recordlinkage.coreTest.IntegratedRecordLinkageTest
--deploy-mode cluster
--master spark://UCSL-GKL-HDP-02:6066/home/hadoop/spark-2.1.0-bin-hadoop2.7/bin/AIRecordLinkage.jar
It works without error in eclipse on the system we developed before exporting it to jar. trying to run it in cluster mode using the offline submit feature we run into a problem. We suspect that spring variables like autowired are not in context, so the objects are not built. Therefore we get a nullpointer exception
Exception in thread "main" java.lang.NullPointerException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
We run it on the test class, here is the code below
@RunWith(SpringRunner.class)
@SpringBootTest
public class IntegratedRecordLinkageTest {
@Autowired
private LoadCSV loadCSV;
@Autowired
private Environment envirnoment;
@Autowired
private IntegratedRecordLinkage integratedRecordLinkage;
@Test
public void testLoadCSVFile() {
try{
integratedRecordLinkage.link();
}
catch(Exception e){
e.printStackTrace();
Assert.fail("Exception Occurred");
}
}
}
kindly tell us how to start spring boot app in cluster
source to share
It's not easy to run things "in real life" through a test. The Maven configuration model, used by many build tools besides Maven, assumes that code and configuration are in different places depending on scope - compilation, runtime, test, etc. Running the test at runtime doesn't make much sense to me. It's okay to try something unorthodox, but your current setup seems to be a lot harder than it's worth.
Instead, I'll run things the way you should:
@SpringBootApplication
@EnableBatchProcessing
public class AIRecordLinkage implements CommandLineRunner {
@Autowired
private LoadCSV loadCSV;
@Autowired
private Environment environment;
@Autowired
private IntegratedRecordLinkage integratedRecordLinkage;
public static void main(String[] args) {
SpringApplication.run(AIRecordLinkage.class, args);
}
@Override
public void run(String... args) throws Exception {
integratedRecordLinkage.link();
}
}
source to share