File download via Post form
download, forms, post, python
Solution
You should look into using the 'requests' library: http://docs.python-requests.org/en/latest/
I believe this would be accomplished like this using the requests library:
data={'data[]':'downloadable_file1.tar'}
req = requests.post('webpage/data.php', data)
with open('file.data', 'wb') as file:
file.write(req.content)
Problem
There is a website that has a list of files that I want to download. To ease the process I tried to write a script to do it for me. (Even though I can select multiple options at the same time, clicking Submit only downloads the first file) The webpages url, webpage/list.php, is different from the action in the form. I'm not sure about what the policy is on posting the actual url here. ``` <form action="webpage/data.php" method="post"> <table align="center"> <tbody> <tr><td> <select name="data[]" size="8" multiple="multiple"> <option value="downloadable_file1.tar">Downloadable file1</option> <option value="downloadable_file2.tar">Downloadable file2</option> <option value="downloadable_file3.tar">Downloadable file3</option> </select> </td></tr> </tbody> </table> <input type="submit"> </form> ``` The script i have is this: ``` import urllib import urllib2 import shutil req = urllib2.Request('webpage/list.php') values = { 'data[]': 'downloadable_file1.tar'} req.add_data(urllib.urlencode(values)) resp = urllib2.urlopen(req) myfile = open('downloadable_file1.tar', 'wb') shutil.copyfileobj(resp.fp, myfile) myfile.close() ``` When running the script the server does not seem to acknowledge the request and just serves me the same webpage with the file list. Is there an option I am missing? Might there be som redirect problem? This is the info I get when using Chrome: ``` Request URL:webpage/data.php Request Method:POST Status Code:200 OK Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3 Accept-Encoding:gzip,deflate,sdch Accept-Language:sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4 Cache-Control:max-age=0 Connection:keep-alive Content-Length:26 Content-Type:application/x-www-form-urlencoded Host:webpage Origin:webpage Referer:webpage/list.php User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22 data[]:downloadable_file1.tar Accept-Ranges:bytes Cache-control:private Connection:Keep-Alive Content-Disposition:attachment; filename="downloadable_file1.tar.gz" Content-Length:1043436 Content-Transfer-Encoding:binary Content-Type:application/x-gzip Date:Tue, 26 Mar 2013 20:18:58 GMT Expires:Mon, 26 Jul 1997 05:00:00 GMT Keep-Alive:timeout=5, max=100 Pragma:private Server:Apache/2.2.9 (FreeBSD) mod_ssl/2.2.9 OpenSSL/0.9.7e-p1 DAV/2 PHP/5.2.6 with Suhosin-Patch X-Powered-By:PHP/5.2.6 ```