Convert JSON to stream node with map or template
I am relatively new to Javascript and Node and I enjoy learning, but my lack of awareness of Javascript design patterns makes me wary of reinventing the wheel, I would like to know from the community if what I want to do is already present in one form or another, I not looking for specific code for the example below, just a nudge in the right direction and what I should be looking for.
Basically I want to create my own personal IFTTT / Zapier to connect data from one API to another.
I am using Node module request
to GET
data from one API and then POST
to another.
request
supports streaming to do things like this:
request.get('http://example.com/api')
.pipe(request.put('http://example.com/api2'));
In between these two requests, I would like to pass JSON through transform, cherry-pick the key / value pairs I need and change the keys to whatever the destination API expects.
request.get('http://example.com/api')
.pipe(apiToApi2Map)
.pipe(request.put('http://example.com/api2'));
Here's a sample JSON from the original API: http://pastebin.com/iKYTJCYk
And this is what I would like to send forward: http://pastebin.com/133RhSJT
The converted JSON in this case takes the keys from the value of each attribute "attribute" of each object and the value from the key "value" of each object.
So my questions are:
-
Is there a framework, library, or module that will make the conversion process easier?
-
Flow, how should I approach this? This seems like an elegant way to do it, since I created some Javascript wrapper functions using
request
for easy access to API methods, I just need to figure out the middle step. -
Can you create "templates" or "maps" for these transformations? Let's say I want to change the source or target API, it would be nice to create a new file that maps the source code for the target key / value.
Hope the community can help and I'm open to any suggestions! :) This is an open source project I'm working on, so if anyone wants to get involved, just get in touch.
source to share
Yes, you are definitely on the right track. There are two streaming libraries that I would point you to through
, which makes it easier to define your own streams and JSONStream
that helps convert a binary stream (such as what you get from request.get
) to a stream of parsed JSON documents. Here's an example using both of them to get you started:
var through = require('through');
var request = require('request');
var JSONStream = require('JSONStream');
var _ = require('underscore');
// Our function(doc) here will get called to handle each
// incoming document int he attributes array of the JSON stream
var transformer = through(function(doc) {
var steps = _.findWhere(doc.items, {
label: "Steps"
});
var activeMinutes = _.findWhere(doc.items, {
label: "Active minutes"
});
var stepsGoal = _.findWhere(doc.items, {
label: "Steps goal"
});
// Push the transformed document into the outgoing stream
this.queue({
steps: steps.value,
activeMinutes: activeMinutes.value,
stepsGoal: stepsGoal.value
});
});
request
.get('http://example.com/api')
// The attributes.* here will split the JSON stream into chunks
// where each chunk is an element of the array
.pipe(JSONStream.parse('attributes.*'))
.pipe(transformer)
.pipe(request.put('http://example.com/api2'));
source to share
As Andrey noticed through the flow of events or events, however, I did something even more user-friendly, scramjet . It works the same way as it does through, but the API is almost identical to arrays, so you can easily use the map and filter methods.
Code for your example:
request.get('http://example.com/api')
.pipe(JSONStream.parse('attributes.items.*'))
.pipe(new scramjet.DataStream())
.filter((item) => item.attibute) // filter out ones without attribute
.reduce((acc, item) => {
acc[item.attribute] = item.value;
return acc;
.then((result) => request.put('http://example.com/api2', result))
;
I think this is a little easier to use, however in this example you are accumulating data into an object, so if the JSON is actually much longer than that, you can include it in the JSONStream again.
source to share