How to access netgear router web interface
python, selenium, web-scraping
Solution
Okay, I found the solution and it was way easier than I thought. I did try John1024's suggestion and was able to download the proper webpage from the router using wget. However I didn't like the fact that wget saved the result to a file, which I would then have to open and parse.
I ended up going back to the original reboot_router.py script I had attempted to modify unsuccessfully the first time. My problem was I was trying to make it too complicated. This is the final script I ended up using:
import urllib2
user = 'admin'
pwd = 'notmyrealpassword'
host = '192.168.1.1'
url = 'http://' + host + '/traffic_meter_2nd.htm'
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, host, user, pwd)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
response = opener.open(url)
stuff = response.read()
response.close()
print stuff
This prints out the entire traffic meter webpage from my router, with its proper values loaded. I can then take this and parse the values out of it. The nice thing about this is it has no external dependencies like selenium, wget or other libraries that needs to be installed. Clean is good.
Thank you, everyone, for your suggestions. I wouldn't have gotten to this answer without them.
Problem
What I am trying to do is access the traffic meter data on my local netgear router. It's easy enough to login to it and click on the link, but ideally I would like a little app that sits down in the system tray (windows) that I can check whenever I want to see what my network traffic is. I'm using python to try to access the router's web page, but I've run into some snags. I originally tried modified a script that would reboot the router (found here https://github.com/ncw/router-rebooter/blob/master/router_rebooter.py) but it just serves up the raw html and I need it after the onload javascript functions have run. This type of thing is described in many posts about web scraping and people suggested using selenium. I tried selenium and have run into two problems. First, it actually opens the browser window, which is not what I want. Second, it skips the stuff I put in to pass the HTTP authentication and pops up the login window anyway. Here is the code: ``` from selenium import webdriver baseAddress = '192.168.1.1' baseURL = 'http://%(user)s:%(pwd)s@%(host)s/traffic_meter.htm' username = 'admin' pwd = 'thisisnotmyrealpassword' url = baseURL % { 'user': username, 'pwd': pwd, 'host': baseAddress } profile = webdriver.FirefoxProfile() profile.set_preference('network.http.phishy-userpass-length', 255) driver = webdriver.Firefox(firefox_profile=profile) driver.get(url) ``` So, my question is, what is the best way to accomplish what I want without having it launch a visible web browser window? Update: Okay, I tried sircapsalot's suggestion and modified the script to this: ``` from selenium import webdriver from contextlib import closing url = 'http://admin:notmyrealpassword@192.168.1.1/start.htm' with closing(webdriver.Remote(desired_capabilities = webdriver.DesiredCapabilities.HTMLUNIT)) as driver: driver.get(url) print(driver.page_source) ``` This fixes the web browser being loaded, but it failed the authentication. Any suggestions?