Defining a write method in an asynchronous client makes data transfer very slow
I wrote an asynchronous client using python asyncore and met some problems. I solved it with this:
Asyncore client on thread crashes entire program when sending data at once
But now I am facing some other problem.
My client program:
import asyncore, threading, socket
class Client(threading.Thread, asyncore.dispatcher):
def __init__(self, host, port):
threading.Thread.__init__(self)
self.daemon = True
self._thread_sockets = dict()
asyncore.dispatcher.__init__(self, map=self._thread_sockets)
self.host = host
self.port = port
self.output_buffer = []
self.start()
def send(self, msg):
self.output_buffer.append(msg)
def writable(self):
return len("".join(self.output_buffer)) > 0
def handle_write(self):
all_data = "".join(self.output_buffer)
bytes_sent = self.socket.send(all_data)
remaining_data = all_data[bytes_sent:]
self.output_buffer = [remaining_data]
def handle_close(self):
self.close()
def handle_error(self):
print("error")
def handle_read(self):
print(self.recv(10))
def run(self):
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((self.host, self.port))
asyncore.loop(map = self._thread_sockets)
mysocket = Client("127.0.0.1",8400)
while True:
a=str(input("input"))
mysocket.send("popo")
And my server program:
import socket
HOST="127.0.0.1"
PORT=8400
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print("socket created")
s.bind((HOST, PORT))
s.listen(1)
print("listen")
conn,addr = s.accept()
print("Accepted. Receiving")
while True:
data = conn.recv(20)
print("Received: ")
print(data)
data = input("Please input reply message:\n").encode("utf-8")
conn.send(data)
print("Data sended. Receiving")
My problem is sending data from client to server very slowly, about 20-30 seconds! But he could always send data successfully. And if I comment on the write method on the client, the submission process is very fast. Why is he acting like this? How do I fix this if I want to use the recording method? Thank!
I am running a server with python3 and a client with python 2. I am using ubuntu 14.04.
source to share
The loop asyncore
calls writable()
when it is ready to do something with the socket. If the method writable()
says that there is something to write, then it is called handle_write()
. By default it writable()
always returns True
, so in this case the busy loop is called handle_write()
and writable()
.
In the above implementation, the method writable()
is called immediately after starting the client loop. At this moment, there is nothing in the buffer, so it writable()
says that there is nothing to write.
The loop asyncore
calls select()
. The loop is now in a waiting state. It can only be woken up when some data is received by a socket or timeout event. After any of these events, the loop checks again writable()
.
The server does not send anything to the client and the client waits for a timeout. The default timeout
is 30 seconds, so you need to wait up to 30 seconds before something is sent. You can reduce the timeout during startup asyncore.loop()
:
asyncore.loop(map = self._thread_sockets, timeout = 0.5)
Another idea that might come up here is to check if it is empty in send()
, and if it is empty, submit it immediately. However, this is a bad idea. send()
called on the main thread, but the socket is being looped asyncore
on a different thread.
For the same reason, it makes sense to secure use output_buffer
for concurrent access from different threads. The lock object threading.Lock()
can be used here:
def __init__(self, host, port):
#...
self.lock = threading.Lock()
def send(self, msg):
self.lock.acquire()
try:
self.output_buffer.append(msg)
finally:
self.lock.release()
def writable(self):
is_writable = False;
self.lock.acquire()
try:
is_writable = len("".join(self.output_buffer)) > 0
finally:
self.lock.release()
return is_writable
def handle_write(self):
self.lock.acquire()
try:
all_data = "".join(self.output_buffer)
bytes_sent = self.socket.send(all_data)
remaining_data = all_data[bytes_sent:]
self.output_buffer = [remaining_data]
finally:
self.lock.release()
There is no mechanism to protect threads from being woken up asyncore
by another thread. So the only solution is to shorten the cycle timeout, although too little timeout increases the CPU usage.
source to share