Make a list of all the files on a website

Question

I am working on a program that searches for something on the internet using `xgoogle`, then finds all the files in the websites of the results. I am having trouble with finding all the files in a website. I found a question that was similar, but I couldn't get it to work. Here is the code I've been using. ``` from bs4 import BeautifulSoup import requests def find_files(): url = "http://www.python.org" soup = BeautifulSoup(requests.get(url).text) for a in soup.find('div', {'class': 'catlist'}).find_all('a'): yield url + a['href'] ``` The code doesn't run when I call it. I have put print statements in the function, but nothing happens. What should I do to fix it? How could this function return a list of all the files in the website?

Accepted Answer

Try this to get you started..

```
from bs4 import BeautifulSoup
import requests

def find_files():
    url = "http://www.python.org"
    soup = BeautifulSoup(requests.get(url).text)

hrefs = []

for a in soup.find_all('a'):
        hrefs.append(a['href'])

return hrefs

list_of_links = find_files()

## show what you've found:
for link in list_of_links:
    print link
```

As you will see, you can't just add the `url` to all the results, as some of them are for other sites, so some of the resulting URLs would not exist... you should be taking a decision on all of the hits you get.

Also, please check on sites' policies regarding webpage scraping before running this

If you wanted to do this as a generator, the following might be of use:

```
from bs4 import BeautifulSoup
import requests

def find_files(url):

soup = BeautifulSoup(requests.get(url).text)

for a in soup.find_all('a'):
        yield a['href']

for link in find_files("http://www.python.org"):
    print link
```

note - I've moved your `url` to make this code more reusable.

Make a list of all the files on a website

Solution

Problem

Related problems