Optimize a huge JSON response
I am working on a large client side application. The server language is Java. Inside Frontend, I have heavily vanilla JavaScript, but AngularJS as an MVC framework.
Problem
Turning to big data analysis, one REST api response at a time is between 1.5MB to 3MB. Working with this data to build the DOM is a pain.
- It takes about 5-10 seconds to load JSON at first.
- Then I create the user interface (DOM)
- After creating the DOM based on user interaction with the data - I have to send / delete the server back with the same JSON with updated values.
Suggest what parameters I need to optimize for responsiveness to pages "
- A few things I mean:
- split the JSON into 1000 chunks at a time as soon as the DOM is loaded, then fetch the data quietly and update the UI.
- GZIP JSON on the server and decode back on the client.
Give me your specific workarounds!
JSON example could be:
var data = [
{
prop:val,
prop2: {},
prop3:[
id: val,
prop4: { {}, {}, {}, {}},
prop5: [ [], [], [] ]
]
},
{},
{},
{}
]
Some use cases
- The DATA size can be 10,000 objects, nested at least six or seven levels.
- It is necessary to build a grid (table), rows are approximately the same length as objects and columns, at least 100 columns.
- All data cells have a custom context menu, have nested headers, all columns are sorted, rows are sorted, and these sorted orders go to the server as soon as users change this. But I have a second threshold.
Simplest example: http://shekhardesigner.github.io/OuchGrid/
source to share
some of my tips:
- First, flip the response data from the server to reduce the size of the json object.
- parallel rendering then ui with chunk
- Don't make deep observations in the data if the data is too large, for example not ngRepeat for too much data, if two-way binding is optional. which will make your application very slow.
source to share
"It only takes 5-10 seconds to load the JSON. Then I create the UI (DOM)."
-
is it not possible to do these 2 async steps? for example load dom and wait for ajax callback?
-
I'm not sure if there is a way as I am missing the details, but perhaps you want to rethink the whole process to load "smaller objects" when you need them.
-
think about compressing the object / string somehow
These are the first 3 ways I can think of how to optimize right now. You can add these suggestions depending on your convenience.
I hope this helps - feel free to add reviews
source to share
"It takes about 5-10 seconds to load the JSON first. Then I create the interface (DOM)."
I recommend that you load the UI and async data, but prevent the user from taking certain actions until the appropriate data has been loaded.
Once the data is loaded into your variable (s) / service (s), use foreground pagination to minimize the load on the browser. Javascript can store a lot of data, but the DOM will try to render a lot of HTML.
source to share
For one of my assignments, we used our own solution. Most of the data would be a collection or an array, so we implemented a simple algorithm to remove all redundant property names and only one set of property names and a collection of the value object. We saw a decent size reduction and probably at the next level some libraries could be used for further decompression. Again, this only works if we are working on structured data, a different algorithm may be required if the array objects have a different structure.
Other algorithms, as I would recommend, look at natural compression techniques such as removing all whitespace from JSON while decreasing field names.
Alternatively, you can look at a specification such as Protocol Buffer, which will probably give a much smaller download size. Have a look at https://github.com/dcodeIO/ProtoBuf.js
If processing your data takes time and gets stuck on screen rendering, you can try web workers (for recent browsers) offloading the processing logic and the main thread / event loop will be available to display the UI or respond to user actions.
source to share