Websphere MQ using JMS, closed connections stuck on MQ
I have a simple JMS application deployed on OC4J under an AIX server, in my application I listen to some queues and submit to other queues in Websphere MQ deployed under AS400 server.
The problem is that my connections to these queues get terminated / closed when it stays idle for a while with an error MQJMS1016
(this is not a problem) and when this happens I try to re-establish the connection, but it works, however the old connection is stuck in MQ and does not stop until it is manually completed.
The recovery code looks like this:
public void recover() {
cleanup();
init();
}
public void cleanup(){
if (session != null) {
try {
session .close();
} catch (JMSException e) {
}
}
if (connection != null) {
try {
connection.close();
} catch (JMSException e) {
}
}
}
public void init(){
// typical initialization of the connection, session and queue...
}
source to share
Since orphan connections (stuck connections on the MQ side) do not affect message processing (i.e. they do not consume messages), we left things as they were until the maximum connections allowed for MQ are reached.
The restore did not work anymore and once we reached this point the MQ administrator had to clean up the orphan connection manually, however the good news is that looking for this particular issue led to an issue reported on the IBM support site:
source to share
MQJMS1016 is an internal error and indicates that the connection loss is due to something wrong with the code or WMQ. Tuning channels will help, but you really need to address the issue of why the app is spewing orphan connections fast enough to exhaust all available channels.
The first thing I would like to do is check the WMQ versions and the WMQ client running. If this is a new development, make sure you are using the WM7 v7 client as v6 is the end of life from September 2011. The v7 client works with v6 QMgrs until you can update it. Once you get to the v7 and QMgr client, you have quite a few options for configuring and reconnecting channels.
The WMQ v7 client download is here: http://bit.ly/bXM0q3
Also note that the reconnection logic in the above code is not intertwined between attempts. If the client requests connection requests at a high rate, it can overload the WMQ listener and perform a very effective DOS attack. It is recommended that you sleep for a few seconds between attempts.
Finally, please PLEASE print related exceptions in JMSException catch blocks. If you have a problem with the JMS transport provider, the JMS Linked Exception will contain low level error information. In the case of WMQ, it contains a reason code such as 2035 MQRC_AUTHORIZATION_ERROR or 2033 MQRC_NO_MSG_AVAILABLE. Here's an example:
try {
.
. code that might throw a JMSException
.
} catch (JMSException je) {
System.err.println("caught "+je);
Exception e = je.getLinkedException();
if (e != null) {
System.err.println("linked exception: "+e);
} else {
System.err.println("No linked exception found.");
}
}
If you receive an error message at 2am, your WMQ administrator will thank you for the associated exceptions.
source to share