How do I estimate throughput using JMH when work is done by different threads?

we are not sure how to test the part of our application where multiple threads (producers) are written to one ConcurrentHashMap. In addition, one consumer thread switches the map to empty if a certain size is reached. A card switch is necessary because producers' streams must be written to the card as quickly as possible, without additional overloads. The consumer thread also processes the map elements and stores them in a buffer. If at the end another thread sends the contents of the buffer to another application component.

It is not clear how to measure the time (and associated bandwidth) it takes for items to be written to the ConcurrentHashMap until they are ready to be sent (-> aggregated bandwidth). For now, we exclude shipping items to ease the process. We know that bandwidth is throttled by the size of the card. However, we want to measure it in order to compare it with the throughput of future implementations.

So far, we have explored the following scenarios:

Silo-Test
  This benchmark will measure every step in a single test. Benchmark A: Insert items into the map until size is reached (= batch size). Benchmark B: Measure how long it takes to store an item in a buffer.
Advantage : relatively easy to implement.
Disadvantage . Stream interactivity is lost and performance aggregation is not provided by adding tests A and B.

Flat-Test
  Compared to Silo-Test, all steps are performed in one test (single-threaded). The first X elements are inserted into the map, then the map is switched, the X elements are processed and stored in the buffer.
Advantage : relatively easy to implement.
Disadvantage . Thread interaction is lost and the reference can only be started by one thread (-> only one producer), because the current implementation does not support concurrent map switching and element processing.

Dynamic-Amount-Test (preferred test)
  The comparison method inserts one item into the map at a time. The consumer thread will run in the @Setup phase, and it will deliver items to the buffer during the test. At the check end, the collected cardinality is provided to the JMH to compute the result using calls to the reference method. We are not aware of any off-the-shelf functionality that allows us to provide a JMH buffer size at the end of the test. Most likely we need to change the JMH for this requirement (right?).
The advantage . Composite bandwidth is calculated by JMH
Disadvantage : No functionality outside the box, JMH modification can be dangerous (measurements are affecting, check plausibility is unclear)

Additional Notes: We also have to deal with non-stationary criteria in this scenario. We thought about using the JMH dispensing capability and different dough sizes.

Thanks in Advance for any constructive answer.

+3


source to share





All Articles