Laravel: Synchronisch queue system
I am trying to set up an API that uses a queuing system on another server to handle requests. Let me start what I am trying to accomplish without a queuing system (no permission to keep it simple): Using Postman, for example, to create a GET request for a URL https://example.com/products returns a JSON string like
[
{
"id": 1,
"name": "some name",
...
},
{
"id": 2,
"name": "some other name",
...
}.
...
]
The code in routes /api.php will look something like this:
<php
Route::get('/products', ProductController@index');
And the code in the app /Http/Controllers/ProductController.php:
<?php
namespace App\Http\Controllers;
class ProductController extends Controller
{
/**
* Return the products.
*
* @return \Illuminate\Http\Response
*/
public function index()
{
// Logic to get the products.
return $products->toJson();
}
}
What I would like to do is have all the business logic processed on another server that has multiple employees running. Below is my reasoning.
- Security: in case we are hacked, it will most likely be the client server, not the production server. Since the latter has all the business logic, a hacker will be able to receive incoming and outgoing requests in the worst case.
- Several employees. It will likely take a long time to receive the products, but other requests may need to take longer to process. The user making the request will have to expect the result in most situations. However, other callers should not wait for this. Therefore, another worker can accept this request and process the job.
This will be the workflow as I see it:
-
All free workers are constantly polled for jobs in the queue
- User makes a request
- The client server takes the request data and queues it
- The worker completes the task from the queue and processes it
- The worker returns the result to the client server
- The client server returns the result to the user
Below is a small picture to clarify the situation.
User 1
_ \
| \
\ \ 1.
\ \ request
\ \ -------------
result \ \ / \
5. \ \ | Worker |
\ _| | Server |
\ | --------- |
------------- | / \ |
/ \ | | Worker1 | |
| Client | / | | | | \
| Server 2. | / | \ / | \
| --------- | / | --------- | \
| / \ | / | --------- | \
| | Queue | | / | / \ | \ ---------
| | | | |_ | | Worker2 | | _| / \
| | Job A | | | | | | | DB |
| | Job B | | 3. <----- | \ / | -----> | Server |
| | | | _ | --------- | _ | |
| \ / | | | ... | | \ /
| --------- | \ | --------- | / ---------
\ / \ | / \ | / ---------
------------- \ | | WorkerN | | / / \
_ 4. ? \ | | | | / | Other |
| | \ / | | Servers |
/ / | --------- | | |
1. / / \ / \ /
request / / ------------- ---------
/ /
/ / result
/ / 5.
/ /
|_
User 2
In the Laravel documentation, I came across queues that I thought would be easy to execute. I started experimenting with Beanstalkd, but I guess any queue driver will do. The problem I stumbled upon is that the queue system is running asynchronously. As a consequence, the client server simply continues without waiting for the result. If I'm missing something, there seems to be no way to get the queue to work synchronously.
When I looked further into the Laravel documentation, I came across a broadcast. I'm not sure if I understand the concept of broadcasting 100%, but what I understand is that receiving seems to be happening in Javascript. I am a third party developer and would like to stay away from Javascript. For some reason it seems wrong to me to use javascript here, however I'm not sure if that feeling is warranted.
Looking further into the documentation I came across Redis. I was mostly intrigued by the functionality of Pub / Sub. I thought the client server could generate a unique value, send it with the request data to the queue, and subscribe to it. Once the worker is finished, it can publish the result with this unique value. I thought this might cover the missing part for step 4. I'm still not sure how this would work in code if this logic would work in the first place. I mostly stick with the part where the client has to listen to and receive data from Redis.
I may be something very simple. Be aware that I am relatively new to programming with PHP and the concept of programming on the world wide web. So if you find the logic is flawed or too far-fetched, please give me some pointers to other / better methods.
Additional FYI, I've heard of Gearman, which seems to be capable of running synchronously and asynchronously. However, I would like to stay out of this as my goal is to make the most of the tools Laravel provides. I'm still involved and not sure enough to use too many external plugins.
Edit . This is how I got there. What am I still missing? Or is what I ask (near) impossible?
Custom calls http://my.domain.com/worker?message=whoop
User should receive JSON response
{"message":"you said whoop"}
Please note that in the response header, the content type must be "application/json"
, not"text/html; charset=UTF-8"
This is what I have so far:
Two servers API server and WORKER server. The API server receives requests and queues them up (local Redis). Workers on the WORKER server process jobs on the API server. As soon as the worker has processed the job, the result of this job is broadcast to the API server. The API server listens for the submitted response and sends it to the user. This is done with Redis and socket.io. My problem is that at this point, in order to send the result, I am sending a click file with some Javascript that listens for the response. This results in a content type "text / html; charset = UTF-8", with an element that is updated after the result from the worker has been submitted. Is there a way, rather than returning a view, to make the return "wait" until the result has been passed?
API server: routes \ web.php:
<?php
Route::get('/worker', 'ApiController@workerHello');
API Server: Application \ Http \ Controllers \ ApiController.php
<?php
namespace App\Http\Controllers;
use App\Jobs\WorkerHello;
use Illuminate\Http\Request;
class ApiController extends Controller
{
/**
* Dispatch on queue and open the response page
*
* @return string
*/
public function workerHello(Request $request)
{
// Dispatch the request to the queue
$jobId = $this->dispatch(new WorkerHello($request->message));
return view('response', compact('jobId'));
}
}
API Server: Application \ Work \ WorkerHello.php
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Support\Facades\Log;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
class WorkerHello implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $message;
/**
* Create a new job instance.
*
* @param string $message the message
* @return void
*/
public function __construct($message = null)
{
$this->message = $message;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
}
}
WORKER-server: app \ Jobs \ WorkerHello.php
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use App\Events\WorkerResponse;
use Illuminate\Support\Facades\Log;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
class WorkerHello implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $message;
/**
* Create a new job instance.
*
* @param string $message the message
* @return void
*/
public function __construct($message = null)
{
$this->message = $message;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
// Do stuff
$message = json_encode(['message' => 'you said ' . $this->message]);
event(new WorkerResponse($this->job->getJobId(), $message));
}
}
WORKER-server: app \ Events \ WorkerResponse.php
<?php
namespace App\Events;
use Illuminate\Broadcasting\Channel;
use Illuminate\Queue\SerializesModels;
use Illuminate\Broadcasting\PrivateChannel;
use Illuminate\Broadcasting\PresenceChannel;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
class WorkerResponse implements ShouldBroadcast
{
use Dispatchable, InteractsWithSockets, SerializesModels;
protected $jobId;
public $message;
/**
* Create a new event instance.
*
* @return void
*/
public function __construct($jobId, $message)
{
$this->jobId = $jobId;
$this->message = $message;
}
/**
* Get the channels the event should broadcast on.
*
* @return Channel|array
*/
public function broadcastOn()
{
return new Channel('worker-response.' . $this->jobId);
}
}
Server API: socket.js (works with node)
var server = require('http').Server();
var io = require('socket.io')(server);
var Redis = require('ioredis');
var redis = new Redis();
redis.psubscribe('worker-response.*');
redis.on('pmessage', function(pattern, channel, message) {
message = JSON.parse(message);
io.emit(channel + ':' + message.event, channel, message.data);
});
server.listen(3000);
Server API: resources \ views \ response.blade.php
<!doctype html>
<html lang="{{ config('app.locale') }}">
<head>
</head>
<body>
<div id="app">
<p>@{{ message }}</p>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.4/vue.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.0.3/socket.io.js"></script>
<script>
var socket = io('http://192.168.10.10:3000');
new Vue({
el: '#app',
data: {
message: '',
},
mounted: function() {
socket.on('worker-response.{{ $jobId }}:App\\Events\\WorkerResponse', function (channel, data) {
this.message = data.message;
}.bind(this));
}
});
</script>
</body>
</html>
source to share
No one has answered this question yet
See similar questions:
or similar: