Temporary storage for data collection prior to shipment
I am working on a composer package for PHP applications. The goal is to send some data after requests, queue jobs, other actions that have been taken. My initial (and working) idea is to use register_shutdown_function
for this. There are a couple of problems with this approach, firstly, it increases the page response time, which means there is an overhead of calculating the request as well as sending data through my API. Another issue is that long-running processes such as work queues do not execute this method for a long time, so there can be significant gaps between data creation and sending and processing.
My thought is that I can use some kind of temporary storage to store data and offer a cronjob every minute. The only question I see with this approach is concurrency management on hight IO. Since many processes will write to the file every (n) ms, there is a problem reading the file and deleting lines that have already been sent.
Another option that I am trying to desperately avoid is using a customer database. This can cause performance problems.
What would be the preferred way to do this?
Edit: The package is essentially a monitoring agent.
source to share
There are a couple of problems with this approach, firstly, it increases the response time of the page, which means there is an overhead of computing the request as well as sending data through my API
I'm not sure if you can get around this and the extra cost would require additional work in the context of the web request. I feel like using a queue based / asynchronous system will minimize this for the client. If you choose a local filesystem or write a socket, you have additional overhead, but you can immediately go back to the client and not block the processing of this request.
Another problem is that long-running processes such as work queues do not execute this method for a long time, so there can be significant gaps between data creation and sending and processing.
Isn't that all ?: p To return to the client immediately and then shutdown asynchronously at some point in the future? Using a job queue allows you to separate and scale the worker pool and web server separately Your web servers can be quite thin because the hard work is put off by the workers.
My thought is that I could use temporary storage to store data and send crowns to it every minute.
I would advise you to take a look at the task queue against which you fold your own. This is pretty much solved, and there are many hugely popular open source projects to solve this problem (any of the MQs). Will running a job for a minute work cron for the client? How do you scale it? If the file has 1000 records, or you scale 10x and have 10,000, can you do all these calculations in less than a minute? What happens if the server dies? How do you recover? Concurrency interprocess? Will you need to manage locks for each process? Will you use a separate file for each process and every minute? To hang events? What happens if you want to spend less than 1 minute?
Durability guarantees
What guarantees do you offer your clients? If the request comes back, can the client make sure the job has been saved and will be completed in the future?
I would recommend choosing a work queue and your web server processes will write to it. This is an extremely popular problem with so many resources, how to scale it, and with clear guarantees of durability and performance.
source to share