Nutch 2.x not crawling websites like flipkart and jabong
hbase, hdfs, nutch, web-crawler
Solution
The regex-urlfilter blocks urls that have querystring parameters:
skip URLs containing certain characters as probable queries, etc.
-[?*!@=]
Modify that file so that urls with querystring parameters are crawled:
skip URLs containing certain characters as probable queries, etc.
-[*!@]
Nutch probably lacks the support for crawling Ajax page. See this
you can probably look at https://issues.apache.org/jira/browse/NUTCH-1323
Problem
I've done some experiments with nutch to crawl websites which were not having any ajax calls and I got all the data. I executed following steps to get the data. - user@localhost:~/sample/nutch/runtime/local/bin$ ./nutch inject /path/to/the/seed.txt - $: ./nutch generate -batchId 321 - $: ./nutch fetch 321 - $: ./nutch parse 321 - $: ./nutch updatedb I have hbase as the storage which stores files on hdfs. If I execute these 5 steps it gives me all the data if the url is http://www.naaptol.com/brands/nokia/mobile-phones.html but if I change it to http://www.flipkart.com/mens-footwear/shoes/sports-shoes/pr?sid=osp,cil,nit,1cu&otracker=hp_nmenu_sub_men_0_Sports%20Shoes it gives me nothing My nutch-site.xml file: ``` <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>storage.data.store.class</name> <value>org.apache.gora.hbase.store.HBaseStore</value> <description>Default class for storing data</description> </property> <property> <name>http.agent.name</name> <value>com.datametica.agent</value> <description>this is just an agent name</description> </property> <property> <name>http.robots.agents</name> <value>datametica_robot</value> <description>this is just a robot</description> </property> <property> <name>plugin.folders</name> <value>/home/sachin/source_codes/svn/nutch/nutch_2.x/build/plugins</value> </property> </configuration> ```