Mechanisms for creating a long-term process
Is there an alternative mechanism for creating a continuous process other than starting an endless loop?
The general template looks like this:
while True:
# Check for some condition or waiting for some event
# Do some processing
time.sleep(0.01)
I'm especially interested in a scenario in which a process acts like a worker listening for an event (like waiting on a task queue).
What are the performance characteristics of alternative approaches?
source to share
Prior art "waiting and processing a job" has been done in several ways:
- Worker Processes or Threads (see
multiprocessing
andthreading
for some useful primitives) - Event-based handling (asyncio, twisted, and a few others). Your asynchronous I / O library raises an event when you receive data on STDIN or any other channel you choose.
- Single threaded with I / O buffer. Depending on your desired load characteristics, it might be wise for worker processes to just wait for the IO and process it when it does. No fancy lineup. Just let the kernel buffer the IO and the calling process block when it's full.
source to share
If you are expecting performance needs for many long running tasks (and scaling compared to other machines), a higher-level system like Celery might fit:
"Celery is a simple, flexible and reliable distributed system for processing a huge number of messages, while providing the operations with the tools necessary to maintain such a system. Its own work queue with a focus on real-time processing, as well as support for task scheduling. "
source to share