How to save the content of an AJAX request with PhantomJS
I am trying to record a constant update of data on a webpage. In google chrome developer tools, i can see my input is received from an AJAX request.
When I click on the "got" text file, I can see the data I want in Google Chrome. I would like to use PhantomJS to receive AJAX responses and then store those responses in files.
So far, I have a program that opens the url of the web page I'm interested in and can print a general overview of the network traffic that I receive, but I don't know how I can save the actual files as they come in. How would I do it?
The code so far:
var page = require('webpage').create();
var url = "www.site_of_interest.com";
page.onResourceRequested = function(request) {
console.log('Request ' + JSON.stringify(request, undefined, 4));
};
page.onResourceReceived = function(response) {
console.log('Receive ' + JSON.stringify(response, undefined, 4));
};
page.open(url);
source to share
This is currently not possible with PhantomJS. It doesn't expose the content of the request / response in these callbacks. Possible workarounds:
- If the AJAX requests can be played back (multiple requests for the same URL give the same response every time), you can make your own AJAX request in the handler
onResourceReceived
and save the response to a file usingfs
. - AJAX responses for the same url means that certain content is changing on the page. You can write your own code to validate the DOM for these changes and infer what an AJAX request might be. It doesn't have to be DOM. Perhaps the data is available in some JavaScript variable from the page context, or it is stored in localStorage.
It is also possible to write a custom implementationXMLHttpRequest
as a proxy that stores responses so that they can be grabbed. It must be entered before running any JavaScript page. This way the handlerpage.onInitialized
works best.
I wrote a post about these workarounds for CasperJS, but they can be easily converted for use with simple PhantomJS: A: How can I trap and process data from XHR responses using casperjs? ...
source to share