Upload and download SFTP files at the same time
A cronjob runs every 3 hours to download a file using SFTP. The planner is written in Perl and the module used is Net::SFTP::Foreign
.
Can I upload files Net::SFTP::Foreign
that are only partially downloaded using SFTP?
If so, do you need to check the modification date of the SFTP file to verify that the copy process is complete?
Suppose a new file is uploaded by someone in SFTP and it is uploaded / copied by a file. If the download is trying at the same time, do I need to code the ability to extract only part of the file?
source to share
It is not a question of the SFTP client you are using, which does not matter. This is how an SFTP server handles the situation.
Some SFTP servers may block the download, preventing you from accessing it while it is still downloading. But most SFTP servers, especially the regular OpenSSH SFTP server, will not block the file.
There is no general solution to this problem. Checking for timestamping or resizing might work for you, but it's hardly reliable.
There are some general solutions to the problem:
- Download the download file "done" after download is complete. Make your program wait for the "done" file to appear.
- You can have a dedicated "upload" folder and upload (atomically) a downloadable file into the "done" folder. Make your program only in the "done" folder.
- You have a file naming convention for uploaded files (".filepart") and the bootloader will (atomically) rename the file after upload to its final name. Make your program ignore ".filepart" files.
See (My) article Locking files on upload / download to a temporary filename , for example when implementing this approach. - A gross hack is to periodically check the file attributes (size and time) and consider the download complete if the attributes have not changed for a certain period of time.
See my answer to SFTP file blocking mechanism for details .
source to share
The easiest way to do this, when the download process is also under your control, is to download the files using temporary names (for example foo-20170809.tgz.temp
) and after the download is complete, rename (the method Net::SFTP::Foreign::put
supports atomic
, which does just that). Then on the download side, filter out the files with names that match the temporary files.
In any case, the methods Net::SFTP::Foreign
get
and rget
can be instructed to resume the transmission passing the option resume => 1
.
Also, if you have full SSH access to the SFTP server, you can check if some other process is still writing the file downloaded with fuser
or some similar tool (although note that even then the file may be incomplete if, for example, there is a network problem and the bootloader must reconnect before resuming transmission).
source to share