File locks and transactions

I am writing a very simple pipeline to handle uploaded files and I decided to save the pipeline state to the file system instead of the database. The pipeline itself is very simple because the files are processed in temporary locations and then atomically moved to a folder that indicates that a particular stage of the pipeline is complete. Once the file is moved to the folder for the next step, the file from the previous step is deleted. The conveyor is a single process and only one of them is active at a time, so there are no race conditions or accidents to worry about. In the worst case, we redo the work that was done earlier and waste disk space.

Problems arise from the very first step of the pipeline. The server that handles the file upload moves the files to a specific folder to make it available to clients, and at the same time creates a symbolic link in another folder to indicate to the pipeline process that there is work to be done. Since there is a race condition here, I use file locks to create and remove symbolic links. The pipeline process blocks the symbolic link, processes the file, removes the symbolic link, and releases the lock. The download process does the same, except that it creates a symbolic link. It deals with the race condition between deleting a symbolic link and creating it, but there is still a bug here.

The download process may fail after moving the file into place, but before the symbolic link is created, it is indicated that there is a file to process. I would like to know how best to handle this case. I can theoretically create the file at the start of the download process and then delete it on successful symlink creation to show the successful creation of the symlink, but this again leads to the same blocking problems as before because there are multiple download processes that are needed to coordinate with each other.

Is there an easier way to deal with the crashed boot process?