Is there a way to reduce I / O latency in linux tasks?
I am doing several parallel file processing processes (minimum 200 processes) where each process reads logs of various sizes (0-50mb) to collect real-time data from the logs. I run my jobs on 16 cores and an 8GB Linux machine.
But there is a huge lag between the processor process and the elapsed time of the files.
Could you please explain why there is such a b / w cpu lag and elapsed time with some examples? There is also a way to reduce the waiting time of these processes.
source to share
What distribution are you using? I'm asking because one way to approach this would be to use a different I / O scheduler, but your distribution may not support changing the scheduler. Certain I / O schedulers are best suited for different file systems and architectures, so I would recommend looking here to get an idea of the different schedulers. As for the inconsistency you are experiencing, I honestly have no idea this is happening.
Also, I'm assuming the log files are located on the SSD?
source to share