What is the difference between laravel course and laravel chunk method?
I would like to know what is the difference between larvel chunk and larvel cursor. Which method is more appropriate to use? What would be the use cases for both of these? I know you have to use a cursor to save memory, but how does it work on the backend?
A detailed explanation with an example would be helpful because I searched stackoverflow and other sites, but I didn't find much information.
Here is a code snippet from laravel documentation.
Quantization results
Flight::chunk(200, function ($flights) {
foreach ($flights as $flight) {
//
}
});
Using cursors
foreach (Flight::where('foo', 'bar')->cursor() as $flight) {
//
}
source to share
Actually, this question may attract some cocky answer, however simple answer is here in the Laravel Docs
Just for your reference:
This is a piece:
This is the cursor:
Chunk fetches records from the database and loads them into memory, positioning the cursor on the last retrieved record to avoid conflict.
So the advantage here is that if you want to reformat a large record before sending them, or you want to perform an operation on nth number of records at a time, then this is useful. For example, if you create a sheet like out / excel, that is, you can count the records in the counters until they are completed, so that they are not all loaded into memory at once and thus exceed the memory limit.
Cursor is using PHP Generators, you can check the php generators page , however here is an interesting signature:
While I can't guarantee that I fully understand the Cursor concept, for Chunk, the chunk executes a request with each record size, retrieves it, and passes it to the closure for further processing of the records.
Hope this is helpful.
source to share
We have a comparison: chunk () and cursor ()
- cursor (): high speed
- chunk (): persistent memory usage
10,000 records :
+-------------+-----------+------------+
| | Time(sec) | Memory(MB) |
+-------------+-----------+------------+
| get() | 0.17 | 22 |
| chunk(100) | 0.38 | 10 |
| chunk(1000) | 0.17 | 12 |
| cursor() | 0.16 | 14 |
+-------------+-----------+------------+
100,000 records :
+--------------+------------+------------+
| | Time(sec) | Memory(MB) |
+--------------+------------+------------+
| get() | 0.8 | 132 |
| chunk(100) | 19.9 | 10 |
| chunk(1000) | 2.3 | 12 |
| chunk(10000) | 1.1 | 34 |
| cursor() | 0.5 | 45 |
+--------------+------------+------------+
- TestData: default migration users table in Laravel
- Homestead 0.5.0
- PHP 7.0.12
- MySQL 5.7.16
- Laravel 5.3.22
source to share
chunk
is based on pagination, it maintains the page number and loops for you.
For example, it DB::table('users')->select('*')->chunk(100, function($e) {})
will execute multiple queries until the result set is less than the block size ( 100
):
select * from `users` limit 100 offset 0;
select * from `users` limit 100 offset 100;
select * from `users` limit 100 offset 200;
select * from `users` limit 100 offset 300;
select * from `users` limit 100 offset 400;
...
cursor
based on PDOStatement::fetch
and generator.
$cursor = DB::table('users')->select('*')->cursor()
foreach ($cursor as $e) { }
will execute one request:
select * from `users`
But the driver doesn't get the result right away.
source to share
Cursor()
- only one request
- get result on call
PDOStatement::fetch()
- the default is a buffered query, which fetches all results at once.
- turned only the current line into an eloquent model
pros
- minimize memory overhead for an eloquent model
- easy to manipulate
Against
- Huge result leads to out of memory
- buffered or unbuffered compromise
Chunk()
- partial query for constrained and offset queries
- get result on call
PDOStatement::fetchAll
- periodically turned results into eloquent models
Arguments
- managed memory usage
Against
- periodically converting results to eloquent models can lead to memory overload
- Requests and memory usage will cost
TL; DR
I used to think cursor () would make queries every time and only store one row result in memory. So when I saw the @ mohammad-asghari comparison chart I was really confused. It should be some kind of buffer behind the scenes.
Tracking Laravel code like below
/**
* Run a select statement against the database and returns a generator.
*
* @param string $query
* @param array $bindings
* @param bool $useReadPdo
* @return \Generator
*/
public function cursor($query, $bindings = [], $useReadPdo = true)
{
$statement = $this->run($query, $bindings, function ($query, $bindings) use ($useReadPdo) {
if ($this->pretending()) {
return [];
}
// First we will create a statement for the query. Then, we will set the fetch
// mode and prepare the bindings for the query. Once that done we will be
// ready to execute the query against the database and return the cursor.
$statement = $this->prepared($this->getPdoForSelect($useReadPdo)
->prepare($query));
$this->bindValues(
$statement, $this->prepareBindings($bindings)
);
// Next, we'll execute the query against the database and return the statement
// so we can return the cursor. The cursor will use a PHP generator to give
// back one row at a time without using a bunch of memory to render them.
$statement->execute();
return $statement;
});
while ($record = $statement->fetch()) {
yield $record;
}
}
I realized that Laravel created this function by wrapping PDOStatement :: fetch () . And while looking for PDO fetch and MySQL buffer I found this document.
https://www.php.net/manual/en/mysqlinfo.concepts.buffering.php
Queries use buffered mode by default. This means that the query results are immediately pushed from MySQL Server to PHP and then stored in the memory of the PHP process.
thus, by doing PDOStatement :: execute (), we are actually fetching entire rows of results one at a time and storing them in memory , not just one row. Therefore, if the result is too large, it will result in an out of memory exception .
Although the document is shown, we can use $pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
to get rid of the buffered request. But the downside is to be careful.
Unbuffered MySQL queries execute the query and then return the resource while the data is still waiting on the MySQL server to retrieve. This uses less memory on the PHP side, but can increase server load. If the full result set has not been received from the server, no further requests can be sent over the same connection. Unbuffered queries can also be referred to as "use cases".
source to share