How to callback on 301 redirect without crawling in scrapy?

I am scraping a search result page where in some cases a 301 redirect will be triggered. In that case I do not want to crawl that page, but I need to call a different callback function, passing the redirect URL string to it.

I belive it should be possible to do it along the rules, but could not figure out how to:

rules = (
        Rule(
            LinkExtractor(              
                allow=['^https://www\.testrule*$'],                               
            ), 
            follow=False,
            callback= 'parse_item'
        ),
)

Is this possible with scrapy 2.4.x



Read more here: https://stackoverflow.com/questions/65013097/how-to-callback-on-301-redirect-without-crawling-in-scrapy

Content Attribution

This content was originally published by Merlin at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: