Crawl links with rel="nofollow" using wget

bash, grep, wget

Solution

info from here:

http://skeena.net/kb/wget%20ignore%20robots.txt

try:

wget -erobots=off http://your.site.here

Problem

I have a site (http://a-site.com) with many links like that. How can I use wget to crawl and grep this type of links to a file? ``` <a href="/user/333333/follow_user" class="btn" rel="nofollow">Follow</a> ``` I tried this but this command won't get me the links with nofollow. ``` $ wget --no-verbose -r -l1 http://a-site.com 2>&1 ```

Original source