Get large files from FTP using python lib

I need to download several large files from an FTP server (> 30 GB per file). I am using ftplib from the standard python list, but there are some pitfalls: if I upload a large file, I can no longer use the connection if the file ends. I get an EOF Error after that, so the connection is closed (due to a timeout?) And for every subsequent file I get a 421 error.

From what I've read, there are two connections. The data and control feed where the data feed seems to be working correctly (I can download the file completely), but the control channels time out from time to time. I also read that ftplib (and other python ftp libs) are not suitable for large files and can only support files up to 1GB in size. There is a similar question here: How do I download a large file in python via ftp (with monitoring and reconnecting)? which is not quite the same, because my files are huge in comparison.

My current code looks like this:

import ftplib
import tempfile

ftp = ftplib.FTP_TLS()

ftp.connect(host=server, port=port)
ftp.login(user=user, passwd=password)

for file in ftp.nlst():
    fd, local_filename = tempfile.mkstemp()
    f = open(fd, "wb")
    ftp.retrbinary('RETR %s' % file, callback=f.write, blocksize=8192)


Are there any settings or other library I can use that supports huge files?


source to share

1 answer

If you are having problems with standard FTP, you can try using a different protocol specifically designed to handle such large files.

There are a number of suitable solutions . Rsync is likely to be a good start.



All Articles