Why is BeautifulSoup not finding a specific table class?
beautifulsoup, python, web-scraping
Solution
The page uses broken HTML, and different parsers will try to repair it differently. Install the `lxml` parser, it parses that page better:
>>> BeautifulSoup(html, 'html.parser').find("div",{"id":"cntPos"}).find("table",{"class":"cntTb"}).tbody.find_all("tr")[1].find("td",{"class":"cntBoxGreyLnk"}) is None
True
>>> BeautifulSoup(html, 'lxml').find("div",{"id":"cntPos"}).find("table",{"class":"cntTb"}).tbody.find_all("tr")[1].find("td",{"class":"cntBoxGreyLnk"}) is None
False
This doesn't mean that `lxml` will handle all broken HTML better than the other parser options. Also look at `html5lib`, a pure-Python implementation of the WHATWG HTML spec and thus more closely follows how current browser implementations handle broken HTML.
Problem
I am using Beautiful Soup to try and scrape the Commodities table off of Oil-Price.net. I can find the first div, table, table body, and the rows of the table body. But there is a column in one of the rows that I can't find using Beautiful soup. When I tell python to print all the tables in that particular row, it doesn't show the one I want. This is my code: ``` from urllib2 import urlopen from bs4 import BeautifulSoup html = urlopen('http://oil-price.net').read() soup = BeautifulSoup(html) div = soup.find("div",{"id":"cntPos"}) table1 = div.find("table",{"class":"cntTb"}) tb1_body = table1.find("tbody") tb1_rows = tb1_body.find_all("tr") tb1_row = tb1_rows[1] td = tb1_row.find("td",{"class":"cntBoxGreyLnk"}) print td ``` All it prints is None. I even try to print each of the rows to see if I can find the column manually and nothing. ``It will show others. But not the one I want.