python - scrapy url去重

问题描述

请问scrapy是url自动去重的吗？比如下面这段代码，为什么运行时start_urls里面的重复url会重复爬取了？

class TestSpider(scrapy.Spider): name = 'test' allowed_domains = ['baidu.com'] start_urls = [’http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9’, ’http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9’, ’http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9’,] def parse(self, response):for sel in response.xpath(’//p[@class='grid-list grid-list-spot']/ul/li’): item = TestspiderItem() item[’title’] = sel.xpath(’p[@class='list']/a/text()’)[0].extract() item[’link’] = sel.xpath(’p[@class='list']/a/@href’)[0].extract() yield item

问题解答

建一个Url管理器，就不会重复抓取了

知道了，改成这样就可以了。

def start_requests(self):

yield scrapy.Request(’http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9’, self.parse)yield scrapy.Request(’http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9’, self.parse)yield scrapy.Request(’http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9’, self.parse)