BeautifulSoup get all the values of a particular column
beautifulsoup, html, html-parsing, python, python-2.7
Solution
from BeautifulSoup import BeautifulSoup
soup = BeautifulSoup(open("file.html",'r').read())
cols = [header.string for header in soup.find('thead').findAll('th')]
col_idx = cols.index('Username')
col_values = [td[col_idx].string
for td in [tr.findAll('td')
for tr in soup.find('tbody').findAll('tr')]]
print(col_values)
results in:
[u'(none)', u'n/a', u'(blank)', u'admin']
Problem
I am using BeautifulSoup to parse html. So far I have the following code: ``` url = "http://routerpasswords.com" data = {"findpass":"1", "router":"Belkin", "findpassword":"Find Password"} post_data = urllib.urlencode(data) req = urllib2.urlopen(url, post_data) html_str = req.read() parser = new BeautifulSoup(html_str) table = parser.find("table") ``` Is there a way to get a list of all of the cels under `column`? Here is an example: If I had this table: ``` <table cellpadding="0" cellspacing="0" width="100%"> <thead> <tr> <th>Manufacturer</th> <th>Model</th> <th width="80">Protocol</th> <th width="80">Username</th> <th width="80">Password</th> </tr> </thead> <tbody> <tr> <td><b>BELKIN</b></td> <td>F5D6130</td> <td>SNMP</td> <td>(none)</td> <td>MiniAP</td> </tr> <tr> <td><b>BELKIN</b></td> <td>F5D7150<i> Rev. FB</i></td> <td>MULTI</td> <td>n/a</td> <td>admin</td> </tr> <tr> <td><b>BELKIN</b></td> <td>F5D8233-4</td> <td>HTTP</td> <td>(blank)</td> <td>(blank)</td> </tr> <tr> <td><b>BELKIN</b></td> <td>F5D7231</td> <td>HTTP</td> <td>admin</td> <td>(blank)</td> </tr> </tbody> </table> ``` How could I get a list of all of the items in the `Username` column? I would prefer them to be strings as well.