Scrapy - Spider Crawls Duplicate Urls

I'm crawling a search results page and scrape title and link information from the same page. As its a Search page, I have the links to the next pages as well, which I have specifie

Solution 1:

You are not given a code example of your Spider, but, possible, you pass argument dont_filter = True, when calling Request method. Try to specify explicitly Request(dont_filter=False). This points out to Spider that he don't must repeat identical requests.