Scraping all mobiles of Flipkart.com

beautifulsoup, python, web-scraping

Solution

You can play around with the get parameters. The regular URL is:

http://www.flipkart.com/mobiles/pr?p[]=sort%3Dprice_asc&sid=tyy%2C4io&layout=grid

Once you hit the 'more results' button (or scroll down) the next page is loaded using AJAX with the following url:

http://www.flipkart.com/mobiles/pr?p%5B%5D=sort%3Dprice_asc&sid=tyy%2C4io&layout=grid&start=41&ajax=true

The url consists of the following parts:

- path: http://www.flipkart.com/mobiles/pr

- querystring:

- p[]: sort=price_asc

- sid: tyy,4io

- layout: grid

- start: 41

- ajax: true

If you want all phones, just increase the 'start' argument. Something like this:

item_count = 600
for i in range(0, item_count, 40):
    link = "http://www.flipkart.com/mobiles/pr?p%5B%5D=sort%3Dprice_asc&sid=tyy%2C4io&layout=grid&ajax=true&start=%d" % (i+1)

    // Do something with the link
    print link

Enjoy, Wout

Problem

I am trying to scrape all the mobiles from www.flipkart.com. Now, what I have thought of doing is that I can scrape all mobiles from here. ``` http://www.flipkart.com/mobiles/pr?p[]=sort%3Dprice_asc&sid=tyy%2C4io&layout=grid ``` Now, the problem is that, in this website I have to press 'show more results' to see more results. But, how can I do this using code? I am using BeautifulSoup package in python. My code till now: ``` import bs4 import re import urllib2 import sys link = 'http://www.flipkart.com/mobiles/pr?p[]=sort%3Dprice_asc&sid=tyy%2C4io&layout=grid' response = urllib2.urlopen(link) thePage = response.read() soup = bs4.BeautifulSoup(thePage) allMobiles = soup.find('div', attrs={'id': 'products'}) ``` I only get the first page in the output? How can I access the other pages?

Original source