Separate output from different streams
All my code is logging through say () to sys.stderr and error forwarding. On a multi-threaded server, I want each thread to be written to a separate log file. Can I accomplish this without rewriting all the code used by threads?
from threading import Thread
from time import sleep
def say(*args):
print(*args, file=sys.stderr)
def worker(num):
for _ in range(5):
say("worker", num, "working")
sleep(.1)
for num in range(4):
Thread(target=worker, args=(num,)).start()
The output is mixed, the goal is to redirect it to a differnt log file for each thread:
worker 0 working
worker 1 working
worker 2 working
worker 3 working
worker 0 working
worker 1 working
worker 3 working
worker 2 working
. . .
I understand that if I try to redirect stderr to a file inside a stream, the redirection will be used by all streams:
def worker(num):
sys.stderr = open('worker{}.log'.format(num), 'w')
for _ in range(5):
say("worker", num, "working")
Result as expected:
$ cat worker3.log
worker 1 working
worker 1 working
worker 1 working
worker 3 working
worker 3 working
worker 3 working
worker 3 working
worker 3 working
Updates
@Amber, I already have a unique id and you can use it as the thread name:
def say(*args, end='\n'):
print(currentThread().getName(), *args, file=sys.stderr, end=end)
sys.stderr.flush()
t_worker = Thread(name=str(num), target=worker, args=(num,))
I can't dynamically fetch a log file for each thread because say () is global - it needs to either put a semaphore in this global log file store or pass a log object for each individual function that logs progress.
source to share
You can use thread local storage (via a class threading.local
) to store a separate file object for each thread. Then the function say
could find the correct file:
local = threading.local()
def say(*args):
if not hasattr(local, "logfile"):
local.logfile = open("logfile{}".format(threading.get_ident()), "a")
print(*args, file=local.logfile)
I am using threading.get_ident
to get a promising unique value to generate a log file name. If there is a more logical way to name the files in your program, I would suggest using that instead. You might want to separate the file creation from the function say
. For example, you can make it part of your thread start code:
local = threading.local()
def say(*args):
print(*args, filename=local.logfile)
def worker1():
local.logfile = open("logfile_worker1", "a")
# do stuff here, including calling `say()` as necessary
source to share
You can call threading.get_ident()
in your function say()
to decide where to write the string. It's not ideal (since thread ID numbers are allowed to be reused after a thread ends and another is created), but if your threads are running for a long time, it might work for your purposes.
source to share