How do I set up my Maven project so that dependencies are provided after deployment?

I want to try a few things with Spark, but the problem is that our cluster does provide, for example Spark 1.6.0 is already locally, but on the other hand, I need to provide these dependencies on my local development machine in my maven repository.

This will be my current pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>masterthesis.code</groupId>
    <artifactId>wordvectors</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <stanford-corenlp.version>3.6.0</stanford-corenlp.version>
        <sista.version>3.3</sista.version>
        <gson.version>2.6.2</gson.version>
        <spark.version>1.6.0</spark.version>
    </properties>

    <dependencies>

        <!-- Apache Spark -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>


        <!-- Stanford CoreNLP -->
        <dependency>
            <groupId>edu.stanford.nlp</groupId>
            <artifactId>stanford-corenlp</artifactId>
            <version>3.6.0</version>
        </dependency>


        <dependency>
            <groupId>edu.stanford.nlp</groupId>
            <artifactId>stanford-corenlp</artifactId>
            <version>${stanford-corenlp.version}</version>
        </dependency>

        <dependency>
            <groupId>edu.stanford.nlp</groupId>
            <artifactId>stanford-corenlp</artifactId>
            <version>${stanford-corenlp.version}</version>
            <classifier>models-english</classifier>
        </dependency>

        <!-- SISTA -->
        <dependency>
            <groupId>edu.arizona.sista</groupId>
            <artifactId>processors</artifactId>
            <version>${sista.version}</version>
        </dependency>


        <!-- GSON -->
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>${gson.version}</version>
        </dependency>


    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

      

But how can I configure this so that the final deployed .jar does not include eg. spark-core_2.10

but instead load this dependency from the cluster?

0


source to share


1 answer


You need to set up your dependency inside the provided scope, for example:

 
<dependency>
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-core_2.10</artifactId>
     <version>1.6.0</version>
     <scope>provided</scope>
</dependency>

      



The data provisioning scope is very similar to compile

(default scope), but indicates that you expect the JDK or container to provide a dependency at runtime. For example, when building a web application for Java Enterprise Edition, you must set a dependency on the Servlet API and its associated Java EE APIs for scope because the web container exposes these classes. This scope is only available on the compile and test path and is not transitive.

+1


source







All Articles