Clever Geek Handbook

Apache Camel Clustering: Multiple JVMs. Same CamelContext

I have one application that we are deploying to a cluster. A cluster can have 2 or 4 JVMs according to the environment. The application has the same CamelContext that we deploy to all JVMs. Hence, all JVMs have the same routes. For FTP routes this is good as it makes it competitive and only 1 JVM receives the files. However, using a timer operation to fetch from the DB, I see that all JVMs are reading the same recordset and doing the same work. I want if one route picks it up, other routes shouldn't try. I tried to play this game. However, couldn't find a better way to do this. Is there any default camel component that supports this? I have read clustering and load balancingSee Camel documentation, but it didn't help. JGroups and ZooKeeper are both cluster type. Any help would be appreciated.

+3


source to share


2 answers


First, the various Camel contexts deployed will act as standalone applications that don't know each other.

Behavior if you have many routes consuming the same sources depends on the component.

  • File / FTP and similar devices usually have file locking mechanisms to avoid multiple users trying to use the same file.
  • Message queue (JMS / AMQP / etc) has built in message handling, if you are not using themes - then each instance will get a copy.
  • For the database and other polluting components, you may need to initiate a poll using some kind of signal that is generated from only one node.


I assume the latter is your main question. This can be done in several ways. They are probably a little tricky to set up, but should get the job done.

  • Clustered Quartz Scheduler. Typically a database is required to be shared. Read here how to tune quartz in Camel . Then you need to set up quartz.properites and clustering properties (JDBC or whatever). This is documented here .
  • You can have one route (selected from your deployed Camel instances) using ZooKeeper rules and routes . You can then use that route and switch to another route in case of problems, or use this single route to fire triggers using a timer or quartz for polling, which can be propagated across all camel instances.
+5


source


I have one question here, suppose we are using quartz in cluster mode per file component.

Failure: Instance 1 (JVM1) reads the file and inserts data into the table. What if my job failed in the middle of the job? In this case, the Instance2 (JVM2) instance will display the same file and add it to the table. How do I tell the camel to read the raw data from the file, or else a duplicate record will be created in the table if there are no constraints in the table.



Note. File route configuration uses parallel processing true
<from uri = "file: xxx">


<camel: split streaming = "true" parallelProcessing = "true">

0


source






More articles:


All Articles