Time series data in JSON
I need to simulate 1,000,000+ data points in JSON. I am thinking of two ways to do this:
a) Array of objects:
[{time:123456789,value:1432423},{time:123456790,value:1432424},....]
or
b) Nested arrays
[[123456789,1432423],[123456790,1432424],....]
Naively comparing the two approaches, it seems that the latter is faster because it uses fewer characters, but is less descriptive. Is it really faster than that? Which one would you choose and why?
Is there a third approach?
source to share
{time:[123456789,123456790,...], value:[1432423,1432424,...]}
why?
- Iterating over a primitive array is faster.
- comparable to the size of "JSON" with b) but you won't lose column information
this npm might be of interest: https://github.com/michaelwittig/fliptable
source to share
If your time series data is modeling some kind of continuous function, especially at regular time intervals, there might be a much more efficient delta compression representation, even if you're still using JSON:
[
{time:10001,value:12345},
{time:10002,value:12354},
{time:10003,value:12354},
{time:10010,value:12352}
]
It can be represented as:
[[10001,1,1,7],[12345,9,,-2]]
Which is 4 times shorter.
The original can be restored using:
[{time:a[0][0],value:a[1][0]},{time:a[0][0] + a[0][1]||1, value: a[1][0] + a[1][1]||0 ...
source to share
To add another example (idea: "time is the key"):
ts1 = {123456789: 1432423, 123456790: 1432424}
You can even imagine:
ts2 = {"2017-01-01": {x: 2, y: 3}, "2017-02-01": {x: 1, y: 5}}
Quite compact in notation.
If you want to get the keys, use Object.keys
:
Object.keys(ts2) // ["2017-01-01", "2017-02-01"]
Then you can either get the values, iterate with these keys, or use a more experimental one Object.values
:
Object.values(ts2) // [{x: 2, y: 3}, {x: 1, y: 5}
In terms of speed: A quick test with 10.000.000 elements in the array worked here:
obj3 = {};
for(var i=0; i < 10000000; i++) {obj3[i] = Math.random()};
console.time("values() test");
Object.values(obj3);
console.timeEnd("values() test");
console.time("keys() test");
Object.keys(obj3);
console.timeEnd("keys() test");
Results on my machine (Chrome, 3.2Ghz Xeon):
- values (): 181.77978515625ms
- keys () test: 1230.604736328125ms
source to share