How to run a POST request programmatically in python with a GUI ? (spynner, webkit...)

python, screen-scraping, spynner, user-interface, webkit

Solution

You can do it like this with Spynner:

import spynner
from PyQt4.QtCore import QUrl
from PyQt4.QtNetwork import QNetworkRequest, QNetworkAccessManager

url = "http://localhost:8080/niklas/test.php"
data = "foo=bar"
headers = { "Content-Type": "application/x-www-form-urlencoded" }

req = QNetworkRequest(QUrl(url))
for k, v in headers.items():
    req.setRawHeader(k, v)

browser = spynner.Browser()
browser.webframe.load(req, QNetworkAccessManager.PostOperation, data)
browser._wait_load()

print browser.html

Problem

I have a web site with flash forms that I need to scrape. Instead of filling the `flash` forms, I would like to `POST` some keys/values to the URL that doesn't support `GET` requests. I use spynner to interact with the site, and `spynner` can have a `GUI`, but my search on google, stackoverflow, spynner github and in the spynner module are unsuccessful. If `spynner` can't do a `POST` request, maybe `gtk` or `qt` + `webkit` can do that ? Any real life code sample will be really appreciated.

Original source