I am scraping a search result page where in some cases a 301 redirect will be triggered. In that case I do not want to crawl that page, but I need to call a different callback function, passing the redirect URL string to it.
I belive it should be possible to do it along the rules, but could not figure out how to:
rules = ( Rule( LinkExtractor( allow=['^https://www\.testrule*$'], ), follow=False, callback= 'parse_item' ), )
Is this possible with scrapy 2.4.x