Decomposing HTML to link text and target

beautifulsoup, html, python, regex

Solution

Use Beautiful Soup. Doing it yourself is harder than it looks, you'll be better off using a tried and tested module.

EDIT:

I think you want:

soup = BeautifulSoup.BeautifulSoup(urllib.urlopen(url).read())

By the way, it's a bad idea to try opening the URL there, as if it goes wrong it could get ugly.

EDIT 2:

This should show you all the links in a page:

import urlparse, urllib
from BeautifulSoup import BeautifulSoup

url = "http://www.example.com/index.html"
source = urllib.urlopen(url).read()

soup = BeautifulSoup(source)

for item in soup.fetchall('a'):
    try:
        link =  urlparse.urlparse(item['href'].lower())
    except:
        # Not a valid link
        pass
    else:
        print link

Problem

Given an HTML link like ``` <a href="urltxt" class="someclass" close="true">texttxt</a> ``` how can I isolate the url and the text? Updates I'm using Beautiful Soup, and am unable to figure out how to do that. I did ``` soup = BeautifulSoup.BeautifulSoup(urllib.urlopen(url)) links = soup.findAll('a') for link in links: print "link content:", link.content," and attr:",link.attrs ``` i get ``` *link content: None and attr: [(u'href', u'_redirectGeneric.asp?genericURL=/root /support.asp')]* ... ... ``` Why am i missing the content? edit: elaborated on 'stuck' as advised :)

Original source