An architectural approach with IPC, Twisted or ZeroMQ?
I am using twisted to fetch messages from internet connected sensors to store them in db.
I want to check these messages without interfering with this process, because I need to compare each message with some baseline values in the db, if anyone matches, I need to trigger an alert for this and the idea does not block any process ...
My idea is creating a new check and alert process, but I need to save this message after the first process, it will send a message to the new process to check and alert if needed.
> I need an IPC for this and I was thinking of using ZeroMQ but also looking for an approach to working with IPC, I think if I use ZeroMQ, but maybe it will be self-proclaimed ...
What do you think of my approach? Maybe I'm completely wrong?
Any advice is appreciated. Thanks to
PD: this process will run on a dedicated server with an expected load of 6000 msg / hour 1Kb each
source to share
All of these approaches are possible. I can only speak in abstract terms because I don't know the exact outline of your application.
If you already have a working application but is not fast enough to handle the number of messages you throw at it, then identify the bottleneck. Two likely reasons for your hold are database access or warnings, as one of them is likely synchronous I / O.
How you handle this depends on your workload:
- If your message rate is high and consistent, you need to make sure your database can handle this rate. If your DB can't handle this, then no amount of non-blocking messages will help you! In this order:
- Try to set up your database.
- Try to put your database on a larger computer with more memory.
- Try to set up the database on multiple computers to distribute the workload. Once you know that your db can handle the message rate, you can deal with other bottlenecks using other forms of parallelism.
- If your message rate is huge, you can use a queue to handle bursts. In this order:
- Install a load balancer in front of the message processor cluster. All this balancer has to do is redistribute sensor messages to different machines for verification and warning. The advantage of this approach is that you probably won't need to change your existing application, just run it on other machines. This works best if your load balancer doesn't have to wait for a response, just send a message.
- If your communication needs are more complex or bi-directional, you can use a message bus (such as ZeroMQ) as the communication layer between message handlers, sender notifications, and database checkers. The idea is to increase parallelism by having non-blocking communication across the bus and each node on the bus doing only one thing. Then you can change the ratio of node types depending on how long each stage of message processing takes. (I. to make the depth of the queue equal throughout the entire message processing.)
source to share
When you receive a message, do two things:
- Check if it should raise an alert (and if possible, send an alert if possible)
- Insert it into the database
You don't need a message queue, multiple processes, IPC, or any of these things. For example:
def messageReceived(self, message):
self.checkForAlerts(message).addCallbacks(self.maybeAlert, log.err)
self.saveMessageToDatabase(message).addErrback(log.err)
source to share