Clever Geek Handbook

How do I handle "java.net.SocketException: Connection reset" in AWS S3 multithreaded file upload?

I have a ThreadPoolExecutorService to which I submit running jobs that upload large (1-2GB) files to the Amazon S3 file system using the JavaSMS SDK. Sometimes one of my worker threads reports a java.net.SocketException error with "Connection reset" and then dies.

AWS doesn't use checked exceptions, so I really can't catch the SocketException directly - it has to be wrapped somehow. My question is how should I handle this problem, so I can repeat any problematic downloads and improve the reliability of my program.

Will the API for adding multipage files be more robust?

Is there some kind of exception that I can reliably catch to enable replays?

Here's the stack trace. Com.example code. * My. Basically, a call to DataProcessorAWS makes a call putObject(String bucketName, String key, File file)

on an instance AmazonS3Client

that is shared between threads.

14/12/11 18:43:17 INFO http.AmazonHttpClient: Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
    at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
    at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
    at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:169)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:119)
    at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:102)
    at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153)
    at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
    at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
    at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
    at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
    at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:685)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3697)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1434)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1294)
    at com.example.DataProcessorAWS$HitWriter.close(DataProcessorAWS.java:156)
    at com.example.DataProcessorAWS$Processor.run(DataProcessorAWS.java:264)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
+3


source to share


4 answers


The javadoc says it putObject()

throws an AmazonClientException when this kind of error occurs. AmazonClientException has a method called isRetryable()

, you can try with that.



+2


source


For this you only need to use AmazonS3Client and not Trasfermanager to upload and download

U should configure AmazonS3Client with the following properties:

1. connectionTimeout=50000 in ms
2.maxConnections=500
3.socketTimeout=50000 in ms
4.maxErrorRetry=10

To download use



AmazonS3Client.putObject(bucketName, key, inputFile);

To download use



S3Object s3Object = AmazonS3Client.getObject(new GetObjectRequest(bucketName, key));`
InputStream downloadStream  = s3Object.getObjectContent();

and save this stream by reading bytes.

+2


source


'Connection reset' means the connection is connected. Close the socket or whatever higher-level construct you are using. Probably the server decided that the download was too high or overloaded or something like that. Whether it is possible to repeat the operation, only you can know.

+1


source


In my case, I am storing an object with some metadata. During prototyping, we added the "time created" system metadata and consistently resulted in a similar error as above in the question. And it cannot be repeated as it intends to modify the system metadata. But according to the API as below,

Throws:

  • SdkClientException - If the client encounters any errors while executing> the request or processing the response.
  • AmazonServiceException - If Amazon S3 encountered errors processing the> request.

I understand that the expected behavior should be,

  • throw one of the above exceptions, OR,
  • returns a 403, not an AmazonClientException, "Disconnected connection".

See the logs below when S3 debugging is enabled,

2017-07-11 17:12:45,163 DEBUG  [org.apache.http.wire] - http-outgoing-3 >> "**[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"**
2017-07-11 17:12:45,221 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "PUT /dev-ec-dev/neon/ECTE/CPER/2017-6/ECTE_dp0p_CPER_2017-06-05.h5 HTTP/1.1[\r][\n]"
2017-07-11 17:12:45,221 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "Host: s3.data.neonscience.org[\r][\n]"
2017-07-11 17:12:45,221 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "Authorization: AWS dev-ec-owner:dE6ouKtPpP9aftLRba4OVn6DH9M=[\r][\n]"
2017-07-11 17:12:45,222 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-site: CPER[\r][\n]"
2017-07-11 17:12:45,226 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-time-created: 2017-07-11T17:12:44.904[\r][\n]"

More logs with regular logging,

2017-07-11 17:12:45,247 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"
2017-07-11 17:12:45,248 ERROR  [org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob] - Error copyDataToS3: Not getting descriptive AmazonS3Exception.
Batch runner failed.com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
java.lang.RuntimeException: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
        at org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob.runTransition(ECTELevelZeroPrimeJob.java:347)
        at org.battelle.neon.is.batch.BatchRunner.lambda$null$1(BatchRunner.java:152)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
+1


source






More articles:


All Articles