Force Storm use jar jar dependency instead of defined classpath dependency
How do I tell Storm to use the dependency included in the fat jar on top of the one in the Storm classpath ?
Here are some details / details:
-
joda-time
is the addiction referred to in 2.0 included instorm classpath
and 2.7 included in the fat bank. - Working with topology using
mvn compile exec:java -Dstorm.topology=ClassName
seems to be very easy to use. - The topology view using
storm jar target/filename-jar-with-dependencies.jar ClassName
seems to be using 2.0 instead of 2.7. -
filename-jar-with-dependencies.jar
was created withmvn package
:-
joda-time 2.7
ispom.xml
in the dependencies section - the correct joda timestamps for 2.7 are found when I do
jar tvf target/filename-jar-with-dependencies.jar | grep joda
-
The reason I even notice this is I see the following warning / error when submitting the topology using the command storm
:
WARN com.amazonaws.services.s3.internal.S3MetadataResponseHandler - Unable to parse last modified date: Mon, 25 May 2015 13:23:29 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is required, but found version: 2.0
at com.amazonaws.util.DateUtils.handleException(DateUtils.java:156) ~[filename-jar-with-dependencies.jar:na]
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:204) ~[filename-jar-with-dependencies.jar:na]
at com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78) ~[filename-jar-with-dependencies.jar:na]
at com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115) ~[filename-jar-with-dependencies.jar:na]
at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52) [filename-jar-with-dependencies.jar:na]
at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30) [filename-jar-with-dependencies.jar:na]
at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1050) [filename-jar-with-dependencies.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:724) [filename-jar-with-dependencies.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) [filename-jar-with-dependencies.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) [filename-jar-with-dependencies.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3672) [filename-jar-with-dependencies.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1160) [filename-jar-with-dependencies.jar:na]
Caused by: java.lang.IllegalArgumentException: Invalid format: "Mon, 25 May 2015 13:23:29 GMT" is malformed at "GMT"
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747) ~[joda-time-2.0.jar:2.0]
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:202) ~[filename-jar-with-dependencies.jar:na]
... 15 common frames omitted
source to share
One workaround I found after reading this post is to replace the library (in my case joda-time
) in the Storm directory /lib
with a newer version, since all .jars in that directory are displayed by default storm classpath
.
I hope there is a better answer if two versions of a library need to coexist for one reason or another.
source to share
In my case with Maven 3.3.3 and "Wildfly 9.0.0-RC2" I need to add to pom.xml
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.8.1</version>
</dependency>
This solved my problem.
If you look at the source code for "aws-sdk-java", you will see the following check. https://github.com/aws/aws-sdk-java/blob/1.10.1/aws-java-sdk-core/src/main/java/com/amazonaws/util/DateUtils.java
/**
* Returns the original runtime exception iff the joda-time being used
* at runtime behaves as expected.
*
* @throws IllegalStateException if the joda-time being used at runtime
* doens't appear to be of the right version.
*/
private static <E extends RuntimeException> E handleException(E ex) {
if (JodaTime.hasExpectedBehavior())
return ex;
throw new IllegalStateException("Joda-time 2.2 or later version is required, but found version: " + JodaTime.getVersion(), ex);
}
source to share