Passing data from file to rendered page in Sails.js
My application needs to be read in a large dataset and passed to the client for control with D3.js. The problem is that on large datasets, it can take a while to read / load the contents of the file. I want to solve this with threads. However, I'm not sure how to do this in the context of the Sails framework.
What I want to do is read the content of the file and pipe it to the rendered page. However, I cannot figure out how to skip it if I use something like res.view('somePage', { data: thePipedData });
.
I currently have something like this:
var datastream = fs.createReadStream(path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv'));
datastream.pipe(res);
...
return res.view('analytics', { title: 'Analytics', data: ??? });
What's the best way to approach this?
source to share
Based on your example, it seems that the best way would be to create a separate data-only endpoint and enable it on the client using a regular tag <script>
.
MyDataController.js
getData: function(req, res) {
/* Some code here to determine datatype and dataset based on params */
// Wrap the data in a Javascript string
res.write("var theData = '");
// Open a read stream for the file
var datastream = fs.createReadStream(
path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv')
);
// Pipe the file to the response. Set {end: false} so that res isn't closed
// when the file stream ends, allowing us to continue writing to it.
datastream.pipe(res, {end: false});
// When the file is done streaming, finish the Javascript string
datastream.on('end', function() {
res.end("';");
});
}
MyView.ejs
<script language="javascript" src="/mydata/getdata?datatype=<%=datatype%>&etc.."></script>
MyViewController.js
res.view('analytics', {datatype: 'someDataType', etc...});
A small variation on this strategy would be to use a JSONP-style approach; instead of wrapping the data in a variable in a data controller action, you should wrap it in a function. Then you can call the endpoint via AJAX to get the data. You can take advantage of fast page loading anyway as the large dataset is loaded separately, but with the JSONP change, you can also easily show the loading progress bar while waiting for data.
source to share