Parsing html data into python list for manipulation

beautifulsoup, html, html-parsing, python, regex

Solution

It's not a good practice to use regex for parsing html. Use `BeautifulSoup` parser: find the cell with `rowTitle` class and `EPS (Basic)` text in it, then iterate over next siblings with `valueCell` class:

from urllib import urlopen
from BeautifulSoup import BeautifulSoup

url = 'http://www.marketwatch.com/investing/stock/goog/financials'
text_soup = BeautifulSoup(urlopen(url).read()) #read in

titles = text_soup.findAll('td', {'class': 'rowTitle'})
for title in titles:
    if 'EPS (Basic)' in title.text:
        print [td.text for td in title.findNextSiblings(attrs={'class': 'valueCell'}) if td.text]

prints:

['13.46', '20.62', '26.69', '30.17', '32.81']

Hope that helps.

Problem

I am trying to read in html websites and extract their data. For example, I would like to read in the EPS (earnings per share) for the past 5 years of companies. Basically, I can read it in and can use either BeautifulSoup or html2text to create a huge text block. I then want to search the file -- I have been using re.search -- but can't seem to get it to work properly. Here is the line I am trying to access: EPS (Basic)\n13.4620.6226.6930.1732.81\n\n So I would like to create a list called EPS = [13.46, 20.62, 26.69, 30.17, 32.81]. Thanks for any help. ``` from stripogram import html2text from urllib import urlopen import re from BeautifulSoup import BeautifulSoup ticker_symbol = 'goog' url = 'http://www.marketwatch.com/investing/stock/' full_url = url + ticker_symbol + '/financials' #build url text_soup = BeautifulSoup(urlopen(full_url).read()) #read in text_parts = text_soup.findAll(text=True) text = ''.join(text_parts) eps = re.search("EPS\s+(\d+)", text) if eps is not None: print eps.group(1) ```

Original source