Spring Package - Reading Files with Aws S3
I am trying to read files from AWS S3 and process it using Spring Batch:
Can Spring Itemreader handle this task? If yes, how to pass credentials to S3 client and configure my Spring xml to read a file or multiple files
<bean id="itemReader" class=""org.springframework.batch.item.file.FlatFileItemReader"">
<property name="resource" value=""${aws.file.name}"" />
</bean>
source to share
Update. To use Spring-cloud-AWS, you will still be using FlatFileItemReader, but now you don't need to create a custom extended resource.
Instead, you set up an aws context and pass your S3Client bean to it.
<aws-context:context-resource-loader amazon-s3="amazonS3Client"/>
The reader will be configured just like any other reader - the only thing unique here is that you will now auto-increment your ResourceLoader
@Autowired
private ResourceLoader resourceLoader;
and then install this resource loader:
@Bean
public FlatFileItemReader<Map<String, Object>> AwsItemReader() {
FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
reader.setLineMapper(new JsonLineMapper());
reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
return reader;
}
I would use FlatFileItemReader and the setting that should happen creates its own S3 resource object. Extend Spring AbstractResource to create your own AWS resource containing AmazonS3 client, web and file path information, and more.
For getInputStream use Java SDK:
S3Object object = s3Client.getObject(new GetObjectRequest(bucket, awsFilePath));
return object.getObjectContent();
Then for contentLength -
return s3Client.getObjectMetadata(bucket, awsFilePath).getContentLength();
and lastModified use
.getLastModified().getTime();
The resource you create will have an AmazonS3Client that contains all the information that your Spring-batch application needs to bind to S3. This is what it might look like with Java configuration.
reader.setResource(new AmazonS3Resource(amazonS3Client, amazonS3Bucket, inputFile));
source to share