Sending GET Request with fragment identifier in Python
get, html, python, url
Solution
The anchor (`p20_quantity_desc`) means nothing to the server. There's a bit of Javascript on the page that's changing the sort order of the results based on this anchor, but this is client-side. Requests/urllib will see the same page response with or without the anchor.
Try disabling Javascript on the page and you'll see what I mean.
What you want to do instead is make a request to the API endpoint that the page is using. Here's an example:
http://steamcommunity.com/market/search/render/?query=&start=0&count=10&search_descriptions=0&sort_column=quantity&sort_dir=asc&appid=730
Note the `sort_column` argument? This is the value you can change to decide the ordering of the results. Use a library like lxml to parse the `results_html` field and presto, you're done.
Problem
I'm trying to send a GET request using the requests module including a fragment identifier. I have the following code: ``` url = 'http://steamcommunity.com/market/search?appid=730#p20_quantity_desc' page = requests.get(url, headers=headers) ``` However I always end up getting the base page (http://steamcommunity.com/market/search?appid=730) instead of the page with the fragment identifier (the #p20_quanitity_description doesn't seem to be sent). urllib2 also isn't working for me with the code: ``` req = urllib2.Request(url, headers={ 'User-Agent': 'Mozilla/5.0' }) page = urllib2.urlopen(req).read().decode('UTF-8', 'ignore') ``` How would I go about sending a GET request and including the #p20_quanitity_description in the URL?