Html to rvest verses html
As the following code shows, html
the package rvest
uses the package htmlParse
from XML
.
html
function (x, ..., encoding = NULL)
{
parse(x, XML::htmlParse, ..., encoding = encoding)
}
<environment: namespace:rvest>
htmlParse
function (file, ignoreBlanks = TRUE, handlers = NULL, replaceEntities = FALSE,
asText = FALSE, trim = TRUE, validate = FALSE, getDTD = TRUE,
isURL = FALSE, asTree = FALSE, addAttributeNamespaces = FALSE,
useInternalNodes = TRUE, isSchema = FALSE, fullNamespaceInfo = FALSE,
encoding = character(), useDotNames = length(grep("^\\.",
names(handlers))) > 0, xinclude = TRUE, addFinalizer = TRUE,
error = htmlErrorHandler, isHTML = TRUE, options = integer(),
parentFirst = FALSE)
.....
So for the following url:
myurl<-"http://www.nepalstock.com.np/"
parse_XML<-htmlParse(myurl) #runs without error
parse_rvest<-html(myurl) # throws out the Internal Sever error
Error in parse.response(r, parser, encoding = encoding) :
server error: (500) Internal Server Error
Any idea?
source to share
Reset default user agent from base query httr::GET
, then it works:
library(httr)
library(rvest)
parse_rvest <- html(myurl, add_headers("User-Agent" = "myagent"))
or
parse_rvest <- html(myurl, user_agent("myagent"))
Note that for debugging purposes, you can add verbose()
to html(...)
.
Add to
Using the new combination rvest
/ xml2
/ curl
, it should look like this:
library(xml2)
library(rvest)
library(curl)
parse_rvest <- curl(myurl, handle = new_handle("useragent" = "myua")) %>%
read_html()
source to share