how to get raw html text of a given url using python

Question

I'm using html2text in python to get raw text (tags included) of a HTML page by taking any URL but I'm getting an error. My code - ``` import html2text import urllib2 proxy = urllib2.ProxyHandler({'http': 'http://:@:'}) auth = urllib2.HTTPBasicAuthHandler() opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler) urllib2.install_opener(opener) html = urllib2.urlopen("http://www.ndtv.com/india-news/this-stunt-for-a-facebook-like-got-the-hyderabad-youth-arrested-740851").read() print html2text.html2text(html) ``` The error - ``` Traceback (most recent call last): File "t.py", line 8, in html = urllib2.urlopen("http://www.ndtv.com/india-news/this-stunt-for-a-facebook-like-got-the-hyderabad-youth-arrested-740851").read() File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 404, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 42…

Accepted Answer

If you don't require SSL, this script in `Python 2.7.x` should work:

```
import urllib
url = "http://stackoverflow.com"
f = urllib.urlopen(url)
print f.read()
```

and in `Python 3.x` use `urllib.request` instead of `urllib`

Because `urllib2` for Python 2, in Python 3 it was merged into `urllib`.

`http://` is required.

EDIT: In 2020, you should use the 3rd party module `requests`. `requests` can be installed with `pip`.

```
import requests
print(requests.get("http://stackoverflow.com").text)
```

how to get raw html text of a given url using python

Solution

Problem