Is there an OS independent way to atomically overwrite a file?
If the file already exists, I want to overwrite it. If it doesn't exist, I want to create it and write to it. I would rather not use a third party library like lockfile (which seems to handle all types of locks).
My initial idea was:
- Write to a temporary file with a randomly generated large ID to avoid conflict.
- Rename temporary filename -> new path.
source to share
os.Rename
calls syscall.Rename
which for Linux / UNIX uses syscall rename (which is atomic *). On Windows, syscall.Rename
calls MoveFileW
that assuming the source and destination are on the same device (which can be organized) and the NTFS file system (which is often the case) is atomic *.
I would make sure the source and destination are on the same device, so Linux rename is not interrupted, and Windows rename is actually atomic. As Dave C mentions above, creating his temp file (usually using ioutil.TempFile
) in the same directory as the existing file is the path; this is how I do my atomic renames.
This works for me in my use case:
- The One Go process receives updates and renames files to replace the updates.
- Another Go process monitors file updates with fsnotify and re mmap the file if updated.
In the above use case, just using os.Rename
worked fine for me.
Further reading:
- Renamed () atomic? "Yes and no. Rename () is atomic unless the OS crashes ..."
- Is it possible to rename an atomic file (overwrite) in Windows?
* Note. I want to point out that when people talk about how a file system works with an atomic file system, from an application perspective, they usually mean that an operation is happening or not happening (which journaling can help) from a user perspective. If you are using atomic in the sense of an atomic memory operation, very few file system operations (outside of direct I / O [ O_DIRECT ] one block writes and reads with disk buffered disabled) can be considered truly atomic.
source to share