Request node module not providing html
I am using the request nodes module to get the html for a website as shown below:
var request = require('request');
request("http://www.thenewschool.org/", function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log("body>>>>>>>>>>");
} else {
console.log("error>>>>>>>>>"+error);
console.log("response statusCode>>>>>>>>>"+response.statusCode);
console.log("response body>>>>>>>>>"+response.body);
}
})
and it gives me this outlet
error → → → →> null
response statusCode → → → →> 403
response body → → → →> Sorry, this request was blocked due to an invalid user agent.
This is passed for most of the cases, but fails in this case, can someone help me fix this.
+3
source to share
2 answers
Well, you are getting HTTP error code 403: Access is Forbidden.
this probably means that your request was "profiled" as "we don't want you here":
- This could be because your IP was tagged
- or because you are missing a header that will make your request look like a real browser. Most likely the user-agent header specified by the response body
+4
source to share
You just need to pass user-agent
in headers (because it requires a url) like:
var options = {
headers: {'user-agent': 'node.js'}
}
request("http://www.thenewschool.org/", options, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log("body>>>>>>>>>>" + body);
} else {
console.log("error>>>>>>>>>"+error);
console.log("response statusCode>>>>>>>>>"+response.statusCode);
console.log("response body>>>>>>>>>"+response.body);
}
})
+8
source to share