How to efficiently join non-overlapping sets in redis?
I have a use case where I know that some of the sets I have implemented in my redis store do not overlap. Some of my sets are quite large, resulting in sunion
or sunionstore
taking quite a long time. Does redis provide any functionality for dealing with such unions?
Alternatively, if there is a way to add items to a set in Redis without checking for uniqueness before each insert, it might solve my problem.
source to share
Actually, there is no need for such a function due to the relative cost of the operations.
When you create Redis objects (such as sets or lists), the cost does not include data structure management (hash table or linked lists), since the amortized complexity of individual inserts is O (1). The cost is dominated by the distribution and initialization of all elements (ie, Specified objects or list objects). When you retrieve these objects, the allocation and formatting of the output buffer prevails in cost, rather than the access paths in the data structure.
Thus, bypassing the set uniqueness property does not lead to significant optimization.
To optimize the SUNION command if the sets do not overlap, it is best to replace it with a pipeline of multiple SMEMBERS commands to retrieve the individual sets (and create a client side join).
Optimizing SUNIONSTORE is actually not possible, as non-overlapping sets are the worst for performance. Performance is dominated by the number of resulting elements, so the fewer common elements, the longer the response time.
source to share