Poorly Ordered Data Through Web Sites
I am in the business of teaching delivery and display of data in real time across websites . Real-time in this context simply means that data is read from the sensor (imaging camera) every .03 up to 1 second . Each data point consists of a time and a value (t, v) that are encoded as doubles (although time is always an integer in this case, I am not suggesting it will be).
Server side uses Alchemy Websockets ( C # ) as I found it very easy to understand / change for my purposes.
I am using websockets examples found here and here , as well as examples included in Alchemy.
I am using HighCharts to display data in real time , but I also have a print in a div for debugging purposes (independent example so they don't interfere with each other).
Most of them already work very well, but there is a clear problem that occurs when I send data too fast (to be clear, sending point data every second or two results in a nice graph that seems to have no problem - the problems get more pronounced the faster I call the send send function on the alchemy server).
The data appears to be in the wrong order , leading to an interesting "skewed" effect.
I'm going to start looking at the ordering of the package contained in the server side buffer (the server is prompted to send a certain number of "historical" points when a new user connects, and this already works - this leads to a pronounced problem like the one shown above), as well as the order of receiving the client side looking through time stamps.
The error is inconsistent in that every time I reload the page, it results in a different "malformed" dataset . This leads me to suspect that the websockets communication is responding or has something to do with the alchemy server.
I'll attach the complete code if necessary, but it's pretty messy right now, so I'm looking more for troubleshooting ideas.
I realized that this is not the expected behavior for a websocket as it is built on TCP.
Any suggestions / ideas for viewing things?
Thank!
Edit: I ran another test to check how many data points were out of order each time I refreshed the page. The numbers are as follows:
1 2 3 25 6 5 10 11 96 2 8
Very inconsistent (never 0). Of course it's interesting!
This result was achieved by eliminating the chart component and using only Web sites and an array to store the data.
Update:
I figured I'd start analyzing the order of things that go into the system and it looks like it accidentally gets inaccurate points using an identical dataset. I have implemented an "insert" function that takes into account unpackaged packages. The result (plus a slight theme change) looks pretty good!
The question remains: Is the website expected to deliver the information out of order? Or is there something wrong with my server side implementation (or Alchemy). I will explore further when I have time.
DECISION!
I understood that! After a lot of testing, I realized that my Connection object (which is responsible for looking at the dataset for new data and submitting it as appropriate, given how the connection is set up) was implemented using a Timer object. This was what I took from the example (usually I just use the Thread object for most asynchronous operations).
As the Timer object accelerates, it starts to asynchronously run previous calls to the Tick function with it. This means that very rarely, one call to the Tick function will be faster than another (due to the delay in the Alchemy Send function). This causes minor downtime issues.
I switched the implementation of the communication loop from the Timer object to the Thread object, thereby ensuring synchronization, and the packets disappeared from the order!
source to share
Websockets uses TCP, and TCP ensures that data is delivered in order.
I would hope the browser fires websocket message events as well. In my tests it looked like this.
Using this simple Node app for testing:
var sio = require('socket.io')
, http = require('http')
, express = require('express')
, app = express()
, srv = http.createServer(app)
, io = sio.listen(srv)
, fs = require('fs')
, idx = fs.readFileSync('index.html').toString()
;
app.get('/', function(req, res) {
res.send(idx);
});
var i = 0;
setInterval(function() {
io.sockets.emit('count', i++);
}, 1);
srv.listen(888);
It just sends websocket messages as fast as it can with a number that increments each message. Customer:
<script src="/socket.io/socket.io.js"></script>
<script>
var last = 0;
var socket = io.connect('/');
socket.on('count', function(d) {
if (d-1 != last) console.warn('out of order!', last, d);
last = d;
});
</script>
Issues a console warning if it receives a message containing a number that is not more than the previous message.
In Chrome and Firefox, I saw zero out of line messages.
I have also tried blocking for a while on the event received by message ( for (var i = 0; i < 1000000; i++) { }
) to simulate a job that will force messages to queue. Message events are still fired ok.
In other words, it is something else in your setup . Chances are the Alchemy server actually sends messages in a different order.
source to share
I don't know when the issue was posted. I also have a similar problem. I am using Alchemy Client to send small data, then no problem. There are many examples for chat. But when I send a file larger than 4KB (not exactly) the problem occurs. I am trying to find what happened. I wrote a program sent from 0-7000 by the Alchemy client, and received from UserContext.DataFrame(onreceive)
, it will happen that it DataFrame.ToString
will get an extra "\0\0\0\0"
at about position 508. Then after that position the data ordering will be wrong. I used 2.2.1 from nuget. And I read version 2.0 on GiHub. The original code doesn't work. So for the old and no reference value.
source to share