Route strings from file queue to persistent JMS queue: how to improve performance?
I need help tuning performance in a use case. In this usage example, the Camel route binds the status lines in the log file and sends each line as a message to the JMS queue. I applied the use case as follows:
package tests;
import java.io.File;
import java.net.URI;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.broker.BrokerFactory;
import org.apache.activemq.broker.BrokerService;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.sjms.SjmsComponent;
import org.apache.camel.main.Main;
public class LinesToQueue {
public static void main() throws Exception {
final File file = new File("data/log.txt");
final String uri = "tcp://127.0.0.1:61616";
final BrokerService jmsService = BrokerFactory.createBroker(new URI("broker:" + uri));
jmsService.start();
final SjmsComponent jmsComponent = new SjmsComponent();
jmsComponent.setConnectionFactory(new ActiveMQConnectionFactory(uri));
final Main main = new Main();
main.bind("jms", jmsComponent);
main.addRouteBuilder(new RouteBuilder() {
@Override
public void configure() throws Exception {
fromF("stream:file?fileName=%s&scanStream=true&scanStreamDelay=0", file.getAbsolutePath())
.routeId("LinesToQueue")
.to("jms:LogLines?synchronous=false");
}
});
main.enableHangupSupport();
main.run();
}
}
When I run this use case with a file already populated with 1,000,000 lines, the overall performance I get along the route is around 313 lines per second. This means it takes about 55 minutes to process the file.
As a sort of reference, I also created another use case. In this example use case, the Camel route binds the status lines in the log file and sends each line as a document to the Elasticsearch index. I applied the use case as follows:
package tests;
import java.io.File;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.main.Main;
public class LinesToIndex {
public static void main() throws Exception {
final File file = new File("data/log.txt");
final String uri = "local";
final Main main = new Main();
main.addRouteBuilder(new RouteBuilder() {
@Override
public void configure() throws Exception {
fromF("stream:file?fileName=%s&scanStream=true&scanStreamDelay=0", file.getAbsolutePath())
.routeId("LinesToIndex")
.bean(new LineConverter())
.toF("elasticsearch://%s?operation=INDEX&indexName=log&indexType=line", uri);
}
});
main.enableHangupSupport();
main.run();
}
}
When I run this use case with a file already populated with 1,000,000 lines, the overall performance I get along the route is around 8333 lines per second. This means it takes about 2 minutes to process the file.
I understand that there is a huge difference between a JMS queue and an Elasticsearch index, but how can the above JMS example be used?
Update # 1:
It seems like persistence in JMS service is the bottleneck in my first use case above. If I turn off JMS persistence, the route is 11111 lines per second. Which persistence store for the JMS service will give me the best performance?
source to share
a few things to consider ...
-
ActiveMQ producers connections are expensive , make sure you are using the factory pooled connection ...
-
Consider using a VM transport for an ActiveMQ instance process
-
consider using an external ActiveMQ broker over TCP (so that it doesn't compete for resources with your test)
-
tune / tune KahaDB or LevelDB to optimize persistent memory for your use.
source to share