Move file to another s3Location in Java
I have the following information. I want to move a file from a specific s3Location to another s3Location using the Java S3 SDK. For example, if the file is in bucket/current
, I want to move it to bucket/old
.
Currently I can download the file as S3Object, turn that object into a file (java.io, since the S3 Java client, for reasons I don't understand, does not allow you to download the S3Object, the very object you download!) And download this file. I'm curious if there is a better approach to this.
Thank!
source to share
S3 does not directly implement the rename or move operation. Instead, the typical solution is to copy the object to a new location and then delete the original. You can do this with the help of AmazonS3#copyObject
and AmazonS3#deleteObject
for the AWS SDK for Java.
This is more efficient than the method described in your question of downloading the file locally and then re-downloading it under a new key. copyObject
internally uses the S3 server copy provided in the S3 REST API PUT Object - Copy . The copy runs on the S3 server side, so you don't have to pay I / O costs (and real money when going through the AWS server) compared to uploading / downloading the file locally.
Remember that this is very different from the rename operation as indicated on a typical local filesystem, for several reasons:
- It's not atomic. Most local filesystems provide an atomic rename operation, which is useful for building secure "commit" or "checkpoint" constructs to post the fact that a file has been written and is ready for use by some other process.
- It's not as fast as the local filesystem renames. For typical local file systems, renaming is a metadata operation that involves managing a small amount of information in inodes. With the copy / delete method that I described, all data should be copied, even if that copy is running server side on S3.
- Your application can be associated with unique edge cases caused by the Amazon S3 data consistency model .
source to share