Sending spring boot disk to start
I am pretty new to spark and I am trying to try spark submit. I created an application in spring boot using mvn package
to create a jar. But when I try to submit to the jar spark-submit
, it cannot find the Main class . But the main class is present in the bank.
spark-submit --class com.dip.sparkapp.SparkappApplication --master local target/sparkapp-0.0.1-SNAPSHOT.jar
source to share
We ran into the same problem, in fact, on the same day you posted this. Our solutions were to use a shadow maven plugin to edit our assembly. We found that when packaging with the spring-boot-maven plugin, it nested our classes in BOOT-INF / classes that we didn't like. I will insert the relevant section so you can try it out in your application - good luck!
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${spring-boot-version}</version>
</dependency>
</dependencies>
<configuration>
<keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
<resource>META-INF/spring.factories</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>${start-class}</mainClass>
</transformer>
</transformers>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
source to share
The above plugin to create a shaded jar worked well for me. But the entityManager is not configurable when I deploy my spring loaded jar using. / spark-submit. I am using auto-configuration with @SpringBootApplication.
spring boot version: 2.1.2. RELEASE spark version: used dependency 2.3.2: spring-boot-starter-data-jpa
Please help if you've used the same in your Springboot application.
source to share