Asynchronous IO file in multithreaded application with variable index
In one of my programs, I have the following method:
def set_part_content(self, part_no, block_no, data):
with open(self.file_path, "rwb+") as f:
f.seek(part_no * self.constant1 + block_no * self.constant2)
f.write(data)
I did it this way:
- I need to write at a different index (the reason why here
f.seek
) - and this function is thread safe (thanks to the operator
with
)
My problem is that this function is called about 10 to 100K times and is obviously really very slow (it represents half the execution time of one of my most important functions) due to the open / close times.
Because of, f.seek
I cannot open the file directly in the function __init__
to work on it (if 2 threads use the function at the same time, it results in a bad index for one of these two, which is critical).
Is there any module / way that could speed up this feature?
source to share
I'm not familiar with python, so I'm not sure if the operator with
will make this function flow safe or not.
If so, you won't have 2 threads using this feature at the same time.
If it is not, your function is not thread safe. You need a lock to keep the threads safe. You can test the implementation of file locks with a Python instruction . Another option is to make this function async and let one thread handle all the writes.
source to share