Guaranteed atomic folder transition
I have a script that runs every minute at XX: 00. The script iterates over all subfolders in a given directory and performs operations on the files inside;
folder=/path/to/directory #Starting directory
someerror=0 #Did we have an error?
#CD to directory. Does it even exist?
cd $folder
RETVAL=$?
[ $RETVAL -eq 0 ] && echo Success changing directory to $folder && mainfolderexist=1
[ $RETVAL -ne 0 ] && echo Failure changing directory to $folder && mainfolderexist=0
if [ $mainfolderexist -eq 1 ]; then
shopt -s nullglob
for dir in $folder/*/
do
thedirname=`basename $dir` #Get directory name
#cd to sub dir. Does it even exist?
cd $dir
RETVAL=$?
[ $RETVAL -eq 0 ] && echo Success changing directory to $dir && subfolderexist=1
[ $RETVAL -ne 0 ] && echo Failure changing directory to $dir && subfolderexist=0
if [ $subfolderexist -eq 1 ]; then
#perform some operation on all files in this directory
someApp -someArgs --name=$thedirname *
else #sub folder doesn't exist
someerror=1
break
fi
#next folder
done
else #main folder doesn't exist
someerror=1
fi
#REPEAT (only if no errors occured)
if [ $someerror -eq 0 ]; then
at now + 1 minutes << END
/bin/bash "$0" "$@"
END
fi
The way I use this is I upload directories to the server using SFTP to a type folder /home/incoming
and once the directory is fully downloaded, I will translate it to /path/to/directory
. Now this is the part that I'm worried about.
So far I've tried to only move directories between XX: XX: 02 and XX: XX: 50, but is that even necessary? I would like to automate the upload + move process without regard to the system time, so I am wondering:
- What if a directory is in the process of moving (
mv "somedir" "/path/to/directory/somedir"
) at XX: 00 and a script is being executed iterating over all directories? - What if the system loses power during the mv command? If the directory ends up halfway through or something similar, I'll have to write a script to check that before executing the above script.
source to share
If your source and destination paths are on the same filesystem then mv
is an atomic operation. Since this is not really about copying or otherwise moving files, your directories will never be in a "half move" state.
If, on the other hand, your source and destination paths are on different filesystems, then mv
it is actually a copy followed by a deletion throughout the tree, which can take a significant amount of time and, if interrupted, will leave things in a semi-full state.
source to share
I believe it mv
is an atomic operation (moving a directory just renames the directory, it doesn't move any files on disk). mv
makes sys-call rename
, which according to this is atomic (with some restrictions).
If the new path already exists, it will be replaced with an atom (subject to multiple conditions - see ERRORS below), so it doesn't make sense that another process trying to access the new path doesn't find it.
source to share
Remember that traffic is not atomic, even if you are on the same disk and partition, where one (or both) uses encryption as software. For example: 2 users on the same computer have directories in / home. If one of them uses an encrypted home directory, the move will not be atomic because it will need to encrypt / decrypt or decrypt / encrypt or encrypt or decrypt files before making the move
source to share