C# - Some packets are dropped while reading from the TCP connection
.net, c#, sockets, tcp
Solution
There are two things that catch my eye; the first is your use of `DataAvailable`. Using this is virtually never the right thing to do. The main time this is useful is for choosing between sync and async methods. It does not tell you whether more data is inbound, for example, and could give a "false positive" (because you are using for something that it doesn't mean) causing your loop to exit too soon. `DataAvailable` only tells you whether data is currently available in the local buffer, and that is all it tells you.
The second thing that interests me is whether `data` is binary or text. The fact that you are using `StreamReader` suggests text, but then... why would you re-encode it back to `byte[]`? If it is arbitrary binary, then you cannot process it as text - that will not work. By the time you've fetched it via `StreamReader` you've already corrupted the contents. If it is a text-based protocol, don't re-encode it: use a queue of strings (or similar).
On an unrelated note... if the queue is truly concurrent, you probably don't need to synchronize access.
Problem
I am using C# to read continues stream of data(ITCH Data i.e Forex Prices) over a TCP connection, but after running the application for longer durations sometimes the application drops a packet and the information is lost. Below is the code snippet which i am using to read the data: ``` private void ReaderThreadStarter() { StreamReader streamReader = new StreamReader(this._networkStream); while (!_stopping) { try { if (this._networkStream.DataAvailable) { while ((line = streamReader.ReadLine()) != null) { lock (_queue.ConcurrentQueue) { byte[] data = System.Text.Encoding.ASCII.GetBytes(line); Log.Info("Data Added in Queue: " + Encoding.ASCII.GetString(data, 0, data.Length)); _queue.WriteToQueue(data); } } } } catch (Exception exception) { Log.Error(exception); } finally { SetStopped(); } } } ``` What the above piece of code does is that it reads the data from the TCP connection and write it into a concurrent queue, and another thread then uses the data in the queue for processing. So basically a simply producer-consumer problem. The producer-consumer part seems to work fine as what I write in the queue gets used by the consumer. One option was to use the sniffer and confirm that the application was dropping packets but I am working in an environment where i can't use the sniffers. The reason I believe that there are packet drops is because for some of my forex orders I never get cancellations and my prices go off and the data providers tell me that the pricing is correct at there end. I am also logging the data which I read from the TCP port before saving in the queue, so from the logs I assume that the data gets lost in reading from the connection. Can someone tell me what I might be doing wrong here or what could be the reasons for dropping the packets. Below is the code snippet of my consumer code: ``` public void ReadQueue() { try { while (true) { { byte[] data = _queue.ReadFromQueue(); Parse(data); } } } catch (Exception exception) { Log.Error(exception); } } public byte[] ReadFromQueue() { try { byte[] data; lock (this) // Enter synchronization block { ConcurrentQueue.TryDequeue(out data); } return data; } catch (Exception exception) { Log.Error(exception); return null; } } ```