Route strings from file queue to persistent JMS queue: how to improve performance?

I need help tuning performance in a use case. In this usage example, the Camel route binds the status lines in the log file and sends each line as a message to the JMS queue. I applied the use case as follows:

package tests;

import java.io.File;
import java.net.URI;

import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.broker.BrokerFactory;
import org.apache.activemq.broker.BrokerService;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.sjms.SjmsComponent;
import org.apache.camel.main.Main;

public class LinesToQueue {

    public static void main() throws Exception {

        final File file = new File("data/log.txt");
        final String uri = "tcp://127.0.0.1:61616";

        final BrokerService jmsService = BrokerFactory.createBroker(new URI("broker:" + uri));
        jmsService.start();

        final SjmsComponent jmsComponent = new SjmsComponent();
        jmsComponent.setConnectionFactory(new ActiveMQConnectionFactory(uri));

        final Main main = new Main();
        main.bind("jms", jmsComponent);
        main.addRouteBuilder(new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                fromF("stream:file?fileName=%s&scanStream=true&scanStreamDelay=0", file.getAbsolutePath())
                        .routeId("LinesToQueue")
                        .to("jms:LogLines?synchronous=false");
            }
        });

        main.enableHangupSupport();
        main.run();
    }

}

      

When I run this use case with a file already populated with 1,000,000 lines, the overall performance I get along the route is around 313 lines per second. This means it takes about 55 minutes to process the file.

As a sort of reference, I also created another use case. In this example use case, the Camel route binds the status lines in the log file and sends each line as a document to the Elasticsearch index. I applied the use case as follows:

package tests;

import java.io.File;

import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.main.Main;

public class LinesToIndex {

    public static void main() throws Exception {

        final File file = new File("data/log.txt");
        final String uri = "local";

        final Main main = new Main();
        main.addRouteBuilder(new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                fromF("stream:file?fileName=%s&scanStream=true&scanStreamDelay=0", file.getAbsolutePath())
                        .routeId("LinesToIndex")
                        .bean(new LineConverter())
                        .toF("elasticsearch://%s?operation=INDEX&indexName=log&indexType=line", uri);
            }
        });

        main.enableHangupSupport();
        main.run();
    }

}

      

When I run this use case with a file already populated with 1,000,000 lines, the overall performance I get along the route is around 8333 lines per second. This means it takes about 2 minutes to process the file.

I understand that there is a huge difference between a JMS queue and an Elasticsearch index, but how can the above JMS example be used?

Update # 1:
It seems like persistence in JMS service is the bottleneck in my first use case above. If I turn off JMS persistence, the route is 11111 lines per second. Which persistence store for the JMS service will give me the best performance?

+3


source to share


1 answer


a few things to consider ...



  • ActiveMQ producers connections are expensive , make sure you are using the factory pooled connection ...

  • Consider using a VM transport for an ActiveMQ instance process

  • consider using an external ActiveMQ broker over TCP (so that it doesn't compete for resources with your test)

  • tune / tune KahaDB or LevelDB to optimize persistent memory for your use.

+1


source







All Articles