Python Socket Receive Large Amount of Data
python, sockets
Solution
TCP/IP is a stream-based protocol, not a message-based protocol. There's no guarantee that every `send()` call by one peer results in a single `recv()` call by the other peer receiving the exact data sent—it might receive the data piece-meal, split across multiple `recv()` calls, due to packet fragmentation.
You need to define your own message-based protocol on top of TCP in order to differentiate message boundaries. Then, to read a message, you continue to call `recv()` until you've read an entire message or an error occurs.
One simple way of sending a message is to prefix each message with its length. Then to read a message, you first read the length, then you read that many bytes. Here's how you might do that:
def send_msg(sock, msg):
# Prefix each message with a 4-byte length (network byte order)
msg = struct.pack('>I', len(msg)) + msg
sock.sendall(msg)
def recv_msg(sock):
# Read message length and unpack it into an integer
raw_msglen = recvall(sock, 4)
if not raw_msglen:
return None
msglen = struct.unpack('>I', raw_msglen)[0]
# Read the message data
return recvall(sock, msglen)
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = bytearray()
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data.extend(packet)
return data
Then you can use the `send_msg` and `recv_msg` functions to send and receive whole messages, and they won't have any problems with packets being split or coalesced on the network level.
Problem
When I try to receive larger amounts of data it gets cut off and I have to press enter to get the rest of the data. At first I was able to increase it a little bit but it still won't receive all of it. As you can see I have increased the buffer on the conn.recv() but it still doesn't get all of the data. It cuts it off at a certain point. I have to press enter on my raw_input in order to receive the rest of the data. Is there anyway I can get all of the data at once? Here's the code. ``` port = 7777 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind(('0.0.0.0', port)) sock.listen(1) print ("Listening on port: "+str(port)) while 1: conn, sock_addr = sock.accept() print "accepted connection from", sock_addr while 1: command = raw_input('shell> ') conn.send(command) data = conn.recv(8000) if not data: break print data, conn.close() ```