Take the first message to arrive in one of two queues

I have an application that I would like to use for messages from queue A, but if queue A is empty, I would like it to consume messages from queue B instead of sitting around. I only want it to process 1 message at a time, and if the consumer is busy processing a message (from any queue), other pending messages (from any queue) must remain in the queue until the consumer is running again (this gives other consumers the ability to process A new message).

So, for example, if a message arrives at either queue A or queue B while the consumer is not running (waiting), the consumer chooses it. If a message arrives on any queue while the consumer is currently processing the message, nothing happens until the consumer has finished processing the current message. And if a message is available in both queue A and queue B, the consumer prefers to select the message from queue A.

Edit : Actually, I'd be happy to be informed that messages are available for use on both A and B, and implement my own logic to choose which one I really want to retrieve from the queue.

Subscribing to both queues in separate threads at once does not work because:

  • if message M arrives on queue A, the consumer must unsubscribe from queue B while it processes message M, and then re-subscribes as soon as message M is processed. Otherwise, if a message arrives on queue B, it can be received and sent at the same time - I don't want to process more than one message at a time.
  • if a message arrives in both queues at once (or a message is already available in both queues when the consumer starts or ends processing of the previous message), then it is possible that both messages will be raised before the consumer has the opportunity to unsubscribe from queue B. If this will happen, I will have to discard one of the messages, but if it happens often, the message will exceed the rollback count and move to the failure queue.

Ideally, I would like to generalize this to the case of N queues. I'm interested in solutions that work perfectly through JMS, but if I need to use the IBM MQ API to do this, then OK. Is there a usage pattern or library that can help achieve this? Are there alternative queuing approaches / techniques / technologies that allow this?

Pulling all messages from all queues and placing them in one queue is not a go; we should be able to flush individual queues ourselves, and different consumers can subscribe to different subsets of N queues.

+3


source to share


3 answers


I don't know if this will be a big change for your code, but have you thought about adding Spring Framework and especially Spring integration to your project? The problems you describe are already being handled there.

This issue will be solved by using two endpoints with users with events in two queues and a thread pool.

http://www.eaipatterns.com/EventDrivenConsumer.html

http://en.wikipedia.org/wiki/Thread_pool_pattern



[change]

If you want the message to remain in the queue until the thread has been released, you can follow the user poll pattern

http://www.eaipatterns.com/PollingConsumer.html

which can choose at will, from which the queue requests the message in the first place. Keep in mind that with this technology, if your system processes messages faster than they arrive, you will experience some redundant polling in the queues.

+2


source


If I were to write this application, I would use an asynchronous consumer. I would set up two consumers using the MQCB verb both on the same connection descriptor (hConn) so that only one of them will be controlled at the same time. Then launch users with MQCTL to make things change.

As long as you process the message in the callback function, you won't be triggered again with a message from any queue.

To answer your additional question - if two messages arrive at the same time, there is no way to ensure that feedback in queue A is always preferred. Obviously, if a message arrived in queue B a fraction of a second earlier than in queue A, then your requirements should process the message in queue B. So the question is why is this important if you are processing one message in queue B before as one message Queue A when you reach exactly the same time.



I see from your edit in the question that you would be happy to just be notified of incoming messages and then you could do it yourself. You can also use Async Consume in view mode, which means you could be called whenever a message enters the queue, and then you can issue an MQGET to the callback function to get the actual message with destructive-get.

Then you can have a callback function on queue B, always do a quick MQGET call without waiting on queue A before choosing to process your message to handle the preferred requirement. This would be a little inefficient, but definitely better than polling both queues, because it adds wait intervals that you definitely don't need. No matter how ineffective it is, it will depend on your answer to the question why it matters.

Queue The callback function can simply go to process the message.

+1


source


My first reaction to this question would be to redirect the queue processing to a windows service (assuming you are using this on a windows server). I don't think you can do this entirely inside MQ itself.

Set for each queue a trigger for the first queue and connect the running application to the Service with information about which queue it represents, and then exit. The service will process each queue according to whatever rules you've configured for it.

Changing the answer as per the questions in the comments:

  • By "within MQ itself" I mean that the describing functionality is probably not something that you can accomplish with any parameter on MQ components like queue manager, pipes, queues, etc. To do this, you will need to configure some kind of service application outside of MQ.

  • "Why does it matter if it's a service, a regular console application, or a GUI?" It might not matter, but the Windows service is always on, runs without user intervention, and can interact with other applications included in the application and without intervention. A console application or GUI can do some of these things, but they are not designed to do so and may not work consistently. On the other hand, if you have to interact with it, it is obvious that a GUI or console application will be the way for you. However, I don't know what you are trying to do, so my ideas may be completely off base for you.

  • "I'm not familiar with the launch - are you talking about that?" Yes. I have been working with Websphere MQ for many years and I have used launch in almost every application.

If you are working with a GUI then you might want to poll the various queues to see what they have and line them up in your GUI to work, and none of my suggestions would have anything to do with you. I took over an ownerless operation, so here we are.

+1


source







All Articles