Amazon S3 - ColdFusion fileExists breaks when file was deleted using s3cmd
I am running a ColdFusion 9 site that stores cached information on Amazon S3.
The ColdFusion application creates files and puts them in Amazon S3. Every N
hour the cache is cleared by a bash script that executes s3cmd del
because it's much more efficient than ColdFusion fileDelete
or directoryDelete
.
However, after the file has been deleted using s3cmd, ColdFusion will still list it as an existing file, although it cannot read its contents.
For a ColdFusion application, I provide S3 credentials on Application.cfc
, and they are the same authentication keys used by s3cmd, so I don't think this is a user rights issue.
Skip the process:
// Create an S3 directory with 3 files
fileWrite( myBucket & 'rabbits/bugs-bunny.txt', 'Hi there, I am Bugs Bunny' );
fileWrite( myBucket & 'rabbits/peter-rabbit.txt', 'Hi there, I am Peter Rabbit' );
fileWrite( myBucket & 'rabbits/roger-rabbit.txt', 'Hi there, I am Roger Rabbit' );
writeDump( var = directoryList(myBucket & 'rabbits/', 'true', 'name' ), label = 'Contents of the rabbits/ folder on S3' );
// Delete one of the files with ColdFusion fileDelete
fileDelete( myBucket & 'rabbits/roger-rabbit.txt' );
writeDump( var = directoryList(myBucket & 'rabbits/', 'true', 'name' ), label = 'Contents of the rabbits/ folder on S3' );
// Now, let delete a file using the command line:
[~]$ s3cmd del s3://myBucket/rabbits/peter-rabbit.txt
File s3://myBucket/rabbits/peter-rabbit.txt deleted
writeDump( var = directoryList(myBucket & 'rabbits/', 'true', 'name' ), label = 'Contents of the rabbits/ folder on S3' );
// So far, so good!
// BUT!... ColdFusion still thinks that peter-rabbit.txt exists, even
// though it cannot display its contents
writeOutput( 'Does bugs-bunny.txt exist?: ' & fileExists(myBucket & 'rabbits/bugs-bunny.txt') );
writeOutput( 'Then show me the content of bugs-bunny.txt: ' & fileRead(myBucket & 'rabbits/bugs-bunny.txt') );
writeOutput( 'Does peter-rabbit.txt exist?: ' & fileExists(myBucket & 'rabbits/peter-rabbit.txt') );
writeOutput( 'Then show me the content of peter-rabbit.txt: ' & fileRead(myBucket & 'rabbits/peter-rabbit.txt') );
// Error on fileRead(peter-rabbit.txt) !!!
source to share
I agree with @MarkAKruger's comment that the issue here is latency.
Given that ColdFusion cannot consistently determine if a file exists, but it constantly reads its actual contents (and does not constantly read them when they are not available), I came up with this solution
string function cacheFileRead(
required string cacheFileName
){
var strContent = '';
try{
strContent = FileRead( ARGUMENTS.cachefileName );
}catch(Any e){
strContent = '';
}
return strContent;
}
source to share
This answer suggests latency is your problem, as I said in the comments above.
I think I will keep an eye on the launch of s3cmd. If you run it via CFEXECUTE, store the timestamp in the application area or in a DB file or table. Then, by checking the file if the command has been run in the last N minutes (you need to experiment to see which makes sense), you will automatically re-read. After N minutes, you can reliably rely on your verification system.
If you are not using s3cmd from cfexecute, try creating a script that updates the timestamp in the application area and then adds a curl command to your s3cmd script, which hits your cf script - keeping the 2 processes in sync.
Another option is to constantly use fileExists () (not a good idea - very expensive) or keep track of what is cached or not cached in some other way that can be updated in real time - like a DB table. Then you need to clear the table from the s3cmd script (probably using the mysql command line).
I can think of something else for you. That's all I have now. :)
source to share