Clever Geek Handbook

Javascript based dynamic content using htmlUnit

I am stuck with JavaScript based dynamic content creation using HtmlUnit. I am expecting to receive (Signin, Registration html content) from the page. With the following code, I am only getting static content.

I am new to HtmlUnit. Any help would be much appreciated.

String strURL = "https://www.checkmytrip.com" ;
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF);
java.util.logging.Logger.getLogger("org.apache.http").setLevel(java.util.logging.Level.OFF);

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_31);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
webClient.waitForBackgroundJavaScript(60 * 1000);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());

HtmlPage myPage = ((HtmlPage) webClient.getPage(strURL));

String theContent = myPage.getWebResponse().getContentAsString();
System.out.println(theContent);
+3


source to share


1 answer


Two points:



  • You need to wait for ForBackgroundJavaScript () after you get the page as shown here

  • You should use myPage.asText () or .asXml () instead, because getWebResponse () returns the original content without executing JavaScript.

    String strURL = "https://www.checkmytrip.com" ;
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF);
java.util.logging.Logger.getLogger("org.apache.http").setLevel(java.util.logging.Level.OFF);

try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_31)) {
    webClient.setAjaxController(new NicelyResynchronizingAjaxController());

    HtmlPage myPage = ((HtmlPage) webClient.getPage(strURL));
    webClient.waitForBackgroundJavaScript(10 * 1000);

    String theContent = myPage.asXml();
    System.out.println(theContent);
}
+4


source






More articles:


All Articles