PhP ftp_put function limitation?
I am working on a solution where I need to upload 33,000 or more csv files to a server.
My script works fine for 25,000 files, but then it sometimes breaks after downloading 26,000 files, or sometimes 27,000 files, etc. with the following PHP warning. ie
PHP Warning: ftp_put (): bind () failed: address is already in use (98)
I investigated this error over the Internet, but did not find a solution, so I post it here.
I checked directory no. file limit on an Ubuntu 14.0.4 LTS server with ext4 file system and can't find anything, which limits uploading 33,000 files to one directory.
I tried to upload the same 33,000 files to upload in two different directories i.e. 20,000 to DirA and left 13,000 in DirB but after 5,000 successful files transferred to DirB it stops and gives the above error.
Here is the basic code I am using to download files as a ie link
$dpath = "/var/www/data/";
$sourceDir = $dpath .'test1';
$remoteDir = 'test2';
$serverIp = "192.168.0.1";
$user = "dataexport";
$pass = "abc";
$connHandle = ftp_connect($serverIp);
if($connHandle)
{
$loginFlag = ftp_login($connHandle, $user, $pass) or die('couldnt connect to $serverIp');
if($loginFlag)
{
if(is_dir($sourceDir))
{
$dirHandle = opendir($sourceDir);
if($dirHandle)
{
while(($file = readdir($dirHandle)) !== false)
{
if($file != "." && $file != "..")
{
if(is_dir($sourceDir.'/'.$file))
{
// code for copying directory.. I am not using it.
}
else
{
$srcFilePath = $sourceDir.'/'.$file;
$desFilePath = $remoteDir.'/'.$file;
if(ftp_put($connHandle, $desFilePath, $srcFilePath, FTP_ASCII))
{
echo "Copied Successfully to path: $desFilePath \n";
}
else
{
echo "Copying failed \n"; die();
}
}
}
}
closedir($dirHandle);
}
}
else
{
echo "Not a valid directory <br />\n";
}
}
else
{
echo "Wrong Credentials for the server with IP = $loginFlag <br />\n";
}
}
else
{
echo "Unable to connect to server with IP = $serverIp <br />\n";
}
ftp_close($connHandle);
}
NOTE. I changed my real code to keep it simple and not tested, maybe it will require some testing. But I get an error on the next line of code after successfully migrating 25,000 files. those.
ftp_put ($ connHandle, $ desFilePath, $ srcFilePath, FTP_ASCII)
I tried to upload a file with "FTP_BINARY" option but didn't work.
Look for the right advice. Thanks in advance.
Greetings
source to share
The error message is pretty self explanatory:
bind() failed: Address already in use (98)
This leads to the fact that the number of files is out of the question, it also has nothing to do with the transfer mode.
The reason lies deep within the internals of the protocol FTP
. FTP
is an old protocol that uses a command connection and a new connection for every file transferred. Each connection is TCP
bound to a port ; the port number is an unsigned 16-bit integer. This results in a maximum access of 64 to ports.
The first 1024 port numbers (0..1023) are reserved for known protocols (80 for HTTP
, 25 for SMTP
, 22 for FTP
command connection, etc.). The range 1024..49151 contains "registered" ports, and the port numbers between 49152 and 65535 contain "dynamic" ports. They are available for the OS to bind short-lived connections to . There are 16 thousand port numbers available in the "dynamic" range.
Some operating systems use a larger dynamic port range. As stated on the Wikipedia page ):
Many Linux kernels use a port range from 32768 to 61000.
There are about 28 thousand ports available in this range. Note that while your script is running, there are other programs running on the computer; many of them create connections TCP
, and those connections use ports as well. This reduces the number of port numbers available to the application.
I don't know what's going on inside the PHP implementation FTP
, but it looks like it doesn't close the data connections it uses to transfer files, and after a few thousand files there is a whole stock of free ports that can be used by applications and the system call bind()
fails ...
Decision
Since there appears to be no way to customize the behavior of functions FTP
in PHP, you can try to close the connection FTP
and reopen it. Hopefully it PHP
releases all related resources, including the many ports used for file transfer.
I haven't tested this, I don't know if it works!
If that doesn't work, you can try using the functions provided . is a protocol designed to avoid many of the flaws in the old protocols (for example ). You can find a basic use case on the function's documentation page . ssh/sftp
PHP
SSH/SCP
FTP
ssh2_scp_send()
source to share
What is probably happening is that for each file transferred, the client is using a different local port on the VPS. After you download many thousands of files, all ports on the system are in use.
You can run netstat -anop
on the VPS to see the ports in use. You may need to adjust the value net.ipv4.ip_local_port_range
to increase the number of available ports and decrease the value net.ipv4.tcp_fin_timeout
to free up ports earlier.
If possible, you can use SFTP, which uses SSH and will not have the same port usage restrictions.
source to share