"The specified key does not exist" S3 error for a real object in the bucket
In our application, we upload the uploaded user image to AWS Java SDK server files for some processing operations (resizing, cropping, etc.). Sometimes we get the following error:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status code: 404; Error code: NoSuchKey;
but this key and the object that was saved this way exist. I know the AWS developer guide expects this behavior:
However, the change information may not be immediately replicated through Amazon S3, and you may observe the following behaviors: A process writes a new object to Amazon S3 and immediately tries to read it. Until the change is fully rolled out, Amazon S3 may report that "the key does not exist."
but how can i handle this error in my code? I tried to wait a few milliseconds, I tried to reload that object - and all my attempts failed.
try
{
Download download = s3TransferManager
.download(new GetObjectRequest(bucketName, key), new File(tempUrl));
download.waitForCompletion();
}
catch (AmazonS3Exception amazonS3Exception)
{
Thread.sleep(1000);
//retry 3 time.... }
I would love to hear any advice on how to load an existing file in this case. Thank!
source to share
The answer really depends on the region you are using. From the S3 FAQ (here http://aws.amazon.com/s3/faqs/ ):
Q: What data consistency model does Amazon S3 use?
Amazon S3 buckets across all regions provide post-write consistency for PUT new objects and eventual consistency for RIP and DELETES overwrites. Amazon S3 buckets in the standard US region only provide post-read consistency after access through the Northern Virginia Endpoint (s3-external-1.amazonaws.com).
If you are using US Standard and do not specify an endpoint, you may encounter large latencies in extreme cases between the put and when the object is available (anecdotally, delays are observed, which are measured in hours). The sequence of actions is to execute put and then rotate and wait for the object.
The immediate solution is to use the Virginia endpoint (for each FAQ) in the case of a US standard, or to migrate from the US standard and use a different region (for example, US-West-2). All other regions are recorded after recording, so the object will become available after the input is complete.
source to share
If your image processing is a background job (asynchronous job), you can use S3 Event Notification . This way, whenever your image is just placed in the bucket, S3 can trigger an SNS / SQS notification or call an AWS Lambda function.
source to share