scrapy- how to stop Redirect (302)

Question

I'm trying to crawl a url using Scrapy. But it redirects me to page that doesn't exist. ``` Redirecting (302) to from ``` The problem is http://www.shop.inonit.in/Products/Inonit-Home-Decor--Knick-Knacks-Cushions/Shor-Sharaba/Andaz-Apna-Apna-Cushion-Cover/pid-1275197.aspx exists, but http://www.shop.inonit.in/mobile/Products/Inonit-Home-Decor--Knick-Knacks-Cushions/Shor-Sharaba/Andaz-Apna-Apna-Cushion-Cover/1275197 doesn't, so the crawler cant find this. I've crawled many other websites as well but didn't have this problem anywhere else. Is there a way I can stop this redirect? Any help would be much appreciated. Thanks. Update: This is my spider class ``` class Inon_Spider(BaseSpider): name = 'Inon' allowed…

Accepted Answer

yes you can do this simply by adding meta values like

```
meta={'dont_redirect': True}
```

also you can stop redirected for a particular response code like

```
meta={'dont_redirect': True,"handle_httpstatus_list": [302]}
```

it will stop redirecting only 302 response codes. you can add as many http status code you want to avoid redirecting them.

example

```
yield Request('some url',
    meta = {
        'dont_redirect': True,
        'handle_httpstatus_list': [302]
    },
    callback= self.some_call_back)
```

scrapy- how to stop Redirect (302)

Solution

Problem