Spring XD very poor performance when using redis as transport
I have deployed a distributed configuration to my local machine.
1 admin 1 container 1 zoo 1 hsql 1 redis
I have the following threads:
stream create --name activityLog --definition "tail --name=/home/bruno/randomName | log"
stream create --name activityLogCounterTap --definition "tap:stream:activityLog > json-to-tuple | filter --expression=payload.reportable.equals(true) | counter --name=activitylogcount"
stream create --name activityLogEventTypeCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=status --name=status"
stream create --name activityLogClientTypeCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=clientType --name=clientType"
stream create --name activityLogIndexCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=index --name=index"
stream create --name activityLogCustomerCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=customer --name=customer"
stream create --name activityLogChannelCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=channel --name=channel"
stream create --name activityLogStrategyOnlineCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=strategyOnline --name=strategyOnline"
stream create --name activityLogStrategyOfflineCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=strategyOffline --name=strategyOffline"
I want to deploy using the following deployment descriptor:
stream deploy --name activityLog --properties "module.*.count=0"
stream deploy --name activityLogCounterTap --properties "module.*.count=0"
stream deploy --name activityLogEventTypeCounterTap --properties "module.*.count=0"
stream deploy --name activityLogClientTypeCounterTap --properties "module.*.count=0"
stream deploy --name activityLogIndexCounterTap --properties "module.*.count=0"
stream deploy --name activityLogCustomerCounterTap --properties "module.*.count=0"
stream deploy --name activityLogChannelCounterTap --properties "module.*.count=0"
stream deploy --name activityLogStrategyOnlineCounterTap --properties "module.*.count=0"
stream deploy --name activityLogStrategyOfflineCounterTap --properties "module.*.count=0"
Messages are json objects.
The idea is to have an arbitration number for containers consuming messages from apache kafka, however a simple tail is used for the example.
I deployed one by one and I see that performance using redis as transport degrades a lot after deploying 4th thread. I observe with the visual vm that there are some race conditions in the input. * Redis: queue-inbound-channel-adapter1. If I use more threads (taps) it becomes unusable.
(I tried to add a thread view image from the Visual VM here, but I need 10 rep: s)
It seems to be blocking here:
"inbound.activityLog.0-redis:queue-inbound-channel-adapter1" prio=10 tid=0x00007fe3581a1800 nid=0x29c7 waiting on condition [0x00007fe3d28de000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000788b48d48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:524)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:40)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:151)
at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:177)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:50)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:290)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
If fewer than 4 threads are deployed, the wait condition never occurs:
"inbound.activityLog.0-redis:queue-inbound-channel-adapter1" prio=10 tid=0x00007ff5101f6800 nid=0x352c runnable [0x00007ff5794a4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:108)
at redis.clients.util.RedisInputStream.fill(RedisInputStream.java:109)
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:45)
at redis.clients.jedis.Protocol.process(Protocol.java:120)
at redis.clients.jedis.Protocol.read(Protocol.java:191)
at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:212)
at redis.clients.jedis.BinaryJedis.brpop(BinaryJedis.java:2068)
at org.springframework.data.redis.connection.jedis.JedisConnection.bRPop(JedisConnection.java:1514)
at org.springframework.data.redis.core.DefaultListOperations$12.inRedis(DefaultListOperations.java:154)
at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:50)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:151)
at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:177)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:50)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:290)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
-
My first question is why doesn't spring-xd use redis as transport is capable of handling more faucets. I tried with rabbitMq and it works.
-
My second question is why is it using redis in the first place, so I deployed to memory config. Again using memory config, but this time with rabbitMQ I can keep track of the messages being passed, which shouldn't happen I guess.
- edit -
So, regarding the first question, I figured out the maxTotal parameter in org.apache.commons.pool2.GenericObjectPool by default is 8. I went into debug mode and changed this value at runtime (no limit = -1) and it seems to be solves the problem. To configure it for deployment using this hack, I needed to create an extension (from the documentation) and override the following beans:
<bean id="messageBus" class="org.springframework.xd.dirt.integration.redis.RedisMessageBus">
<constructor-arg ref="redisConnectionFactory2" />
<constructor-arg ref="codec"/>
<property name="defaultBackOffInitialInterval" value="${xd.messagebus.redis.default.backOffInitialInterval}" />
<property name="defaultBackOffMaxInterval" value="${xd.messagebus.redis.default.backOffMaxInterval}" />
<property name="defaultBackOffMultiplier" value="${xd.messagebus.redis.default.backOffMultiplier}" />
<property name="defaultConcurrency" value="${xd.messagebus.redis.default.concurrency}" />
<property name="defaultMaxAttempts" value="${xd.messagebus.redis.default.maxAttempts}" />
</bean>
<bean id="counterRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisCounterRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="fieldValueCounterRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisFieldValueCounterRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="gaugeRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisGaugeRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="richGaugeRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisRichGaugeRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="aggregateCounterRepository"
class="org.springframework.xd.analytics.metrics.redis.RedisAggregateCounterRepository">
<constructor-arg ref="redisConnectionFactory2" />
</bean>
<bean id="redisConnectionFactory2" lazy-init="false"
class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="hostName" value="${spring.redis.host}" />
<property name="port" value="${spring.redis.port}" />
<property name="password" value="" />
<property name="poolConfig" ref="jedisPoolConfig"/>
</bean>
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxTotal" value="-1"/>
</bean>
I first tried to define a redisConnectionFactory bean that will override the default, but that didn't work.
I'm not sure if the solution is to make maxTotal configurable in servers.yml to increase this value or if there is some kind of bug where the pools are not being freed. Anyway, I'll open a jira, hope it helps someone without going through all the work I submitted :-)
source to share
The problem is that Spring Boot (which Spring XD is based on) defaults to the redis pool size for connections to 8 (see spring.redis.pool.max-active at https://docs.spring.io/spring- boot / docs / current / reference / html / common-application-properties.html ). This is why the size of the GenericObjectPool is 8 as stated in the question (Jedis uses GenericObjectPool).
Also, as shown in the second stack trace, Spring XD can accept a connection from the pool and block it until a message is received. With only 8 connections in the pool, this can soon lead to very poor performance if multiple threads / branches / queues, etc. Deployed in one XD container.
However, its easy to change the default pool size without the hack mentioned in the question - the max active setting is already configured on server.yml (since it only depends on your Spring boot configuration), for example
# Redis properties
spring:
redis:
port: 6379
host: 127.0.0.1
pool:
maxActive: 100
maxIdle: 100
# sentinel:
# master: mymaster
# nodes: 127.0.0.1:26379,127.0.0.1:26380,127.0.0.1:26381
This is currently not well documented in Spring XD, but defaults will be documented on server.yml in the next release, see https://jira.spring.io/browse/XD-3733
source to share