What should be the convenient memory limit for reading a PHP 1Gb file?
I have a 1Gb1 file that needs to be read. I choose php to filter and change some lines and then create another file2 with those changes. The code is good. If I read the 50M file it works well. The code generates file2 with any changes as expected. But when I try to run the 1Gb file, file2 is not created and I get an error from the browser like this:
The connection to localhost was interrupted.
Check your Internet connection
Check any cables and reboot any routers, modems, or other network devices you may be using.
Allow Chrome to access the network in your firewall or antivirus settings.
If it is already listed as a program allowed to access the network, try removing it from the list and adding it again.
If you use a proxy server...
Check your proxy settings or contact your network administrator to make sure the proxy server is working. If you don't believe you should be using a proxy server: Go to the Chrome menu > Settings > Show advanced settings... > Change proxy settings... > LAN Settings and deselect "Use a proxy server for your LAN".
If I go back and run the small file, it works fine again. I already sit in php memeory up to 2040M ini_set('memory_limit', '2048M')
but I don't know if enough or f is possible.
So how should a comfortable memory be for this problem?
NOTE. The server is apache, win7, i7 8cores 64 bit, 16G RAM. I think the code is not important, but someone asks to see it:
ini_set('memory_limit', '2048M');#Set the memory limit
$new_dbName="au_site";
$patern[0]="/di_site/";
$replace[0]=$new_dbName;
#xdebug_break();
$dirBase=dirname(__FILE__);
$dir2 = new DirectoryIterator(dirname(__FILE__));
#xdebug_break();
foreach ($dir2 as $fileinfo) {
if (!$fileinfo->isDot() && $fileinfo->isFile()) {
$str = $fileinfo->getFilename();
if (preg_match("/\.sql/i", $str)) {
#xdebug_break();
$i=1;
if(!($handle= fopen("$str", "r"))){
die("Cannot open the file");
}
else{
while (!feof($handle)) {
#xdebug_break();
$line=trim(fgets($handle), "\t\n\r\0\x0B");
$firstChar = substr($line, 0,1) ;
$ord = ord($firstChar);
if(ord($firstChar)<>45){
if (preg_match("/di_site/", $line)) {
xdebug_break();
$chn=preg_replace($patern, $replace, $line);
$line=$chn;
}
#echo $line."<br>";
$sql.=$line."\n";
}
}
xdebug_break();
$newDBsql=$dirBase."/".$new_dbName.".sql";
if(!$handle = fopen($newDBsql,"w")){
die("Can not open the file");
}
else{
fwrite($handle, $sql);
fclose($handle);
}
}
}
}
}
source to share
Rather than creating all the file content you intend to write (which takes a lot of memory), consider writing a filter stream instead.
A stream filter operates on one buffered read from the underlying stream, typically about 8KB of data. The following code example defines such a filter, it breaks each bucket into separate lines and calls your code to make changes to it.
<?php
class myfilter extends \php_user_filter
{
private $buffer; // internal buffer to create data buckets with
private $pattern = ['/di_site/'];
private $replace = ['au_site'];
function filter($in, $out, &$consumed, $closing)
{
while ($bucket = stream_bucket_make_writeable($in)) {
$parts = preg_split('/(\n|\r\n)/', $bucket->data, -1, PREG_SPLIT_DELIM_CAPTURE);
$buffer = '';
// each line spans two array elements
for ($i = 0, $n = count($parts); $i + 1 < $n; $i += 2) {
$line = $parts[$i] . $parts[$i + 1];
$buffer .= $this->treat_line($line);
$consumed += strlen($line);
}
stream_bucket_append($out, stream_bucket_new($this->stream, $buffer));
}
return PSFS_PASS_ON;
}
/** THIS IS YOUR CODE **/
function treat_line($line)
{
$line = trim($line, "\t\n\r\0\x0B");
$firstChar = substr($line, 0,1) ;
if (ord($firstChar)<>45) {
if (preg_match("/di_site/", $line)) {
$line = preg_replace($this->pattern, $this->replace, $line);
}
}
return $line . "\n";
}
function onCreate()
{
$this->buffer = fopen('php://memory', 'r+');
}
function onClose()
{
fclose($this->buffer);
}
}
stream_filter_register("myfilter", "myfilter");
// open input and attach filter
$in = fopen(__FILE__, 'r');
stream_filter_prepend($in, 'myfilter');
// open output stream and start copying
$out = fopen('php://stdout', 'w');
stream_copy_to_stream($in, $out);
fclose($out);
fclose($in);
source to share