What happens when redis is overloaded?
If redis is overloaded, can I configure it to remove given requests? I have an application where data is updated in real time (10-15 times per second per item) for a large number of items. The values were outdated quickly and I don't need any consistency.
I would also like to compute the parallel sum of the values that are being recorded in real time. What's the best option here? LUA executed in redis? Small application located in the same box as redis using UNIX sockets?
source to share
When Redis gets overloaded, it will simply slow down its clients. For most commands, the protocol itself is synchronous.
Redis supports pipelining, but the client cannot discard traffic still in the pipeline but not yet acknowledged by the server. Redis itself does not queue incoming traffic, it does the TCP stack.
Thus, it is not possible to configure the Redis server to delete given requests. However, on the client side, you can implement the last line of values:
-
the queue is actually 2 cards indexed by your positions (only one value stored for each item). An application card will be used. The secondary card will be used by a specific thread. The contents of 2 cards can be replaced atomic.
-
a specific thread is blocked when the primary card is empty. When not, it changes the contents of the two maps, asynchronously sends the contents of the secondary map to Redis using aggressive pipelining commands and variable parameters. It also receives a response from Redis.
-
as long as the thread is running on the secondary card, the application can still populate the primary card. If Redis is too slow, the application will accumulate the latest values in the main map.
This strategy can be implemented in C using hiredis and an event loop of your choice.
However, this is not easy to implement, so I would first check if Redis performance is not sufficient for all traffic for my purpose. It's not uncommon to test Redis at over 500K op / s these days (using a single core). Nothing prevents you from sharding your data across multiple Redis instances if needed.
You will likely saturate the network links in front of the Redis server processor. Therefore, it is better to implement the last queue of values (if necessary) on the client side rather than on the server side.
As for calculating the sum, I would try to calculate and maintain it in real time. For example, the GETSET command can be used to set a new value when returning a previous one.
Instead of just setting your own values, you can do:
[old value] = GETSET item <new value>
INCRBY mysum [new value] - [old value]
The mysum key will contain the sum of your values for all elements at any time. With Redis 2.6, you can use Lua to define this calculation to keep the rounded results.
Running a large batch to calculate statistics on existing data (as I understand your "parallel" sum) is not really suitable for Redis. It is not meant to be a map / reduction computation like a computation.
source to share