`open_http': 403 Forbidden (OpenURI::HTTPError) for the string "Steve_Jobs" but not for any other string
http-error, open-uri, ruby
Solution
I think this happens for locked down entries like "Steve Jobs", "Al-Gore" etc. This is specified in the same book that you are referring to:
For some pages – such as Al Gore's locked-down entry – Wikipedia will not respond to a web request if a User-Agent isn't specified. The "User-Agent" typically refers to your browser, and you can see this by inspecting the headers you send for any page request in your browser. By providing a "User-Agent" key-value pair, (I basically use "Ruby" and it seems to work), we can pass it as a hash (I use the constant HEADERS_HASH in the example) as the second argument of the method call.
It is specified later at http://ruby.bastardsbook.com/chapters/web-crawling/
Problem
I was going through the Ruby tutorials provided at http://ruby.bastardsbook.com/ and I encountered the following code: ``` require "open-uri" remote_base_url = "http://en.wikipedia.org/wiki" r1 = "Steve_Wozniak" r2 = "Steve_Jobs" f1 = "my_copy_of-" + r1 + ".html" f2 = "my_copy_of-" + r2 + ".html" # read the first url remote_full_url = remote_base_url + "/" + r1 rpage = open(remote_full_url).read # write the first file to disk file = open(f1, "w") file.write(rpage) file.close # read the first url remote_full_url = remote_base_url + "/" + r2 rpage = open(remote_full_url).read # write the second file to disk file = open(f2, "w") file.write(rpage) file.close # open a new file: compiled_file = open("apple-guys.html", "w") # reopen the first and second files again k1 = open(f1, "r") k2 = open(f2, "r") compiled_file.write(k1.read) compiled_file.write(k2.read) k1.close k2.close compiled_file.close ``` The code fails with the following trace: ``` /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in `open_http': 403 Forbidden (OpenURI::HTTPError) from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in `catch' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in `open' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in `open' from /Users/arkidmitra/tweetfetch/samecode.rb:11 ``` My problem is not that the code fails but that whenever I change r2 to anything other than Steve_Jobs, it works. What is happening here?