Redis: reset counter every day
I am looking to reset the counter every day using Redis. I'm new to Redis, so I want to make sure I have a good understanding of how transactions and pipes work.
Does the following code ensure that I always get a unique (date, number) pair when working in a multiprocessor environment, or do I need to use Redis locking?
import datetime
import redis
r = redis.Redis(...)
def get_utc_date_now():
return datetime.datetime.utcnow().date()
def get_daily_counter(r, dt_key='dt', counter_key='counter'):
def incr_daily_number(pipe):
dt_now = get_utc_date_now().isoformat() # e.g.: "2014-10-18"
dt = pipe.get(dt_key)
pipe.multi()
if dt != dt_now:
pipe.set(dt_key, dt_now)
pipe.set(counter_key, 0)
pipe.get(dt_key)
pipe.incr(counter_key)
result = r.transaction(incr_daily_number, dt_key)
return result[-2:]
# Get the (dt, number) couple
# 2014-10-18, 1
# 2014-10-18, 2
# etc.
dt, number = get_daily_counter(r)
UPDATE
Try with LUA Script:
r = redis.Redis(...)
incr_with_reset_on_change_lua_script = """
local dt = redis.call('GET', KEYS[2])
if dt ~= ARGV[2] then
redis.call('MSET', KEYS[1], ARGV[1], KEYS[2], ARGV[2])
end
return redis.call('INCR', KEYS[1])
"""
# Incr KEYS1 but reset first if KEYS2 has changed.
incr_with_reset_on_change = r.register_script(incr_with_reset_on_change_lua_script)
counter_key = 'dcounterA'
watch_key = 'dcounterA_dt'
watch_value = get_utc_date_now().isoformat()
number = incr_with_reset_on_change(keys=[counter_key, watch_key], args=[reset_value, watch_value])
source to share
Consider two parallel transactions occurring at midnight. Both can execute get (dt_key), but the MULTI / EXEC block will execute first. This will reset the counter, set a new date, increase the counter. The second will also go into the MULTI / EXEC block, but because the 'dt' value has changed, execution will fail and incr_daily_number will be called. This time get (dt_key) will return a new date, so when the MULTI / EXEC block is executed, the counter will be incremented without any reset. Two transactions will return a new date with different counter values.
So, I believe that there is no race condition here and that the pairs (date, number) will be unique.
You could also implement this using a server-side Lua script (whose execution is always atomic). This is usually more convenient.
Note that there really is no such thing as blocking Redis. The locking mechanism available in the API is provided by the Python client, not the Redis server. If you look at its implementation, you will see that it is also based on SETNX + WATCH / MULTI / EXEC or Lua blocks.
source to share