python - Scrapy存在内存泄漏的问题。

问题描述

再编写爬虫的时候，总是跑了一段时间（不会超过12个小时）就会被OOM掉。很是无奈！！！根据官方的文档， 使用这个prefs()但是实在找不出问题的所在。

Live ReferencesHtmlResponse 42 oldest: 753s agoMySuteSpider1 oldest: 2964s agoRequest 32412 oldest: 2920s agoSelector 42 oldest: 751s agoTripItem 37 oldest: 751s ago

爬虫的处理是获取所有页面的a标签的链接：

#获取域名的后缀def get_domain_suffix(domain): if ’com’ in tldextract.extract(domain).suffix:return True return False#拼接域名。只存主域名def save_domain(domain): domain_name = tldextract.extract(domain).domain suffix_name = tldextract.extract(domain).suffix return domain_name + ’.’ + suffix_name#获取域名ipdef get_domain_ip(domain): try:ip = socket.gethostbyname(domain)return ip except:return ’114.114.114.114’# 获取域名所在的国家def get_domain_ct_iso(ip): GEO = geoip2.database.Reader(’/var/test/geodb/GeoLite2-City.mmdb’) r = GEO.city(ip) return r.country.iso_codeclass MyDomainSpider(scrapy.Spider): name = ’my_domain’ start_urls = [’http://xxx.com ] def parse_items(self, response):item = TripItem()for url in response.xpath(’//a/@href’).extract(): if url.startswith(’http’): domain = urlparse.urlparse(url).netloc if get_domain_tw(domain) or get_domain_ct_iso(get_domain_ip(domain)) == ’US’:item[’domain’] = save_domain(domain)item[’ip’] = get_domain_ip(domain)item[’datetime’] = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')yield item def parse(self, response):for url in response.xpath(’//a/@href’).extract(): if url.startswith(’http’):domain = urlparse.urlparse(url).netlocif get_domain_tw(domain) or get_domain_ct_iso(get_domain_ip(domain)) == ’US’: yield scrapy.Request(url, callback=self.parse_items)

请指教一下谢谢

问题解答

yield item 是不是得落地，存文件或者db，不然一直存内存了