Guaranteed atomic folder transition

I have a script that runs every minute at XX: 00. The script iterates over all subfolders in a given directory and performs operations on the files inside;

folder=/path/to/directory #Starting directory
someerror=0 #Did we have an error?

#CD to directory. Does it even exist?
cd $folder
[ $RETVAL -eq 0 ] && echo Success changing directory to $folder && mainfolderexist=1
[ $RETVAL -ne 0 ] && echo Failure changing directory to $folder && mainfolderexist=0

if [ $mainfolderexist -eq 1 ]; then
    shopt -s nullglob
    for dir in $folder/*/
    thedirname=`basename $dir` #Get directory name
    #cd to sub dir. Does it even exist?
    cd $dir
    [ $RETVAL -eq 0 ] && echo Success changing directory to $dir && subfolderexist=1
    [ $RETVAL -ne 0 ] && echo Failure changing directory to $dir && subfolderexist=0
    if [ $subfolderexist -eq 1 ]; then
        #perform some operation on all files in this directory
        someApp -someArgs --name=$thedirname *
    else #sub folder doesn't exist

    #next folder
else #main folder doesn't exist

#REPEAT (only if no errors occured)
if [ $someerror -eq 0 ]; then
at now + 1 minutes << END
/bin/bash "$0" "$@"


The way I use this is I upload directories to the server using SFTP to a type folder /home/incoming

and once the directory is fully downloaded, I will translate it to /path/to/directory

. Now this is the part that I'm worried about.

So far I've tried to only move directories between XX: XX: 02 and XX: XX: 50, but is that even necessary? I would like to automate the upload + move process without regard to the system time, so I am wondering:

  • What if a directory is in the process of moving ( mv "somedir" "/path/to/directory/somedir"

    ) at XX: 00 and a script is being executed iterating over all directories?
  • What if the system loses power during the mv command? If the directory ends up halfway through or something similar, I'll have to write a script to check that before executing the above script.

source to share

3 answers

If your source and destination paths are on the same filesystem then mv

is an atomic operation. Since this is not really about copying or otherwise moving files, your directories will never be in a "half move" state.

If, on the other hand, your source and destination paths are on different filesystems, then mv

it is actually a copy followed by a deletion throughout the tree, which can take a significant amount of time and, if interrupted, will leave things in a semi-full state.



I believe it mv

is an atomic operation (moving a directory just renames the directory, it doesn't move any files on disk). mv

makes sys-call rename

, which according to this is atomic (with some restrictions).

If the new path already exists, it will be replaced with an atom (subject to multiple conditions - see ERRORS below), so it doesn't make sense that another process trying to access the new path doesn't find it.



Remember that traffic is not atomic, even if you are on the same disk and partition, where one (or both) uses encryption as software. For example: 2 users on the same computer have directories in / home. If one of them uses an encrypted home directory, the move will not be atomic because it will need to encrypt / decrypt or decrypt / encrypt or encrypt or decrypt files before making the move



All Articles