Clever Geek Handbook

Fast implementation of a single stream providing data, many streams consuming data

I have a lot of data that I want to distribute in many different streams. This data comes from a single stream. Consumer threads can safely access the container at the same time.

The data needs to be merged into a container, ever for delta seconds (50ms <delta <1), during which consuming threads should be blocked, but not blocked. Likewise, when a data producer wants to merge the data, it must wait for any read streams to finish (which should be fast), but no one should start reading as the update should happen as soon as possible.

I am working on linux (platform specific solution excellent / expected) and I take care of every millisecond. What locking mechanisms should I use or is there an even better model for this problem?

+2


source to share


3 answers


If there is only one producer thread and memory, this is not a consideration, you might want to consider using a merge and swap algorithm.



In it, the writer thread creates a copy of the data structure while the readers continue to use the original, merge into new changes, then swap the two structures in a mutex or critical section (or read / write lock). If your Unix platform supports blocking as atomic operation , you can exchange locks at maximum read throughput through their implementation.

+4


source


It looks like you need to use pthread read / write locks. They allow you to restrict access to one script or several readers. Look at pthread_rwlock_init to initialize the lock, pthread_rwlock_rdlock to acquire a read lock, and pthread_rwlock_wrlock to acquire a write lock.



+4


source


This appears to be useful for pthread read and write locks, as well as some thread safe queues. The producer string inserts the items into the queue. The working pool will pop the items from the queue and process the data. I'm not sure how the output will work, but you can also use a thread-safe queue ... perhaps a priority queue for automatic data merging, if that makes sense.

The blocked queue design is nothing more than an exclusive lock mutex std::queue

for storing data and a condition variable for waking up threads waiting in the queue. The method enqueue

acquires the lock, queues it, releases the lock, and signals the state. The method dequeue

grabs the mutex, waits for the condition using the mutex as a guard, and deletes any data that is there when it wakes up. This is a fairly standard line of producers and consumers.

Before translating your own solution, you can check out Boost.MPI and Boost.Thread . Both of them provide better C ++ interfaces within the main OS implementation. I've used Boost.Thread a lot , but it doesn't provide a nice messaging interface, but it does make the pthread work better.

If you are really into multiprocessing you can give Boost.MPI or perhaps Apache Qpid some serious consideration. I plan to look at Qpid and AMPQ for future projects as both provide nice message-based interfaces.

+3


source






More articles:


All Articles