HtmlAgilityPack HtmlWeb.Load returning empty Document

c#, html, html-agility-pack, web-crawler

Solution

It seems this website requires cookies to be enabled. So creating a cookie container for your web request should solve the issue:

var url = "http://www.prettygreen.com/";
var htmlWeb = new HtmlWeb();
htmlWeb.PreRequest += request =>
    {
        request.CookieContainer = new System.Net.CookieContainer();
        return true;
    };
var htmlDoc = htmlWeb.Load(url);
var outerHtml = htmlDoc.DocumentNode.OuterHtml;
Assert.AreNotEqual("", outerHtml);

Problem

I have been using HtmlAgilityPack for the last 2 months in a Web Crawler Application with no issues loading a webpage. Now when I try to load a this particular webpage, the document OuterHtml is empty, so this test fails ``` var url = "http://www.prettygreen.com/"; var htmlWeb = new HtmlWeb(); var htmlDoc = htmlWeb.Load(url); var outerHtml = htmlDoc.DocumentNode.OuterHtml; Assert.AreNotEqual("", pageHtml); ``` I can load another page from the site with no problems, such as setting ``` url = "http://www.prettygreen.com/news/"; ``` In the past I once had an issue with encodings, I played around with htmlWeb.OverrideEncoding and htmlWeb.AutoDetectEncoding with no luck. I have no idea what could be the issue here with this webpage.

Original source