python - 关于Scrapy中TwistedPipline报错

问题描述

> python3.5 / pycharm

crawlspider 返回的 item 是绝对没有问题的，接上TwistedPipline就报错 请哪位老师帮忙分析下报错代码

# TwistedPiplineclass MysqlTwistedPipline(object): def __init__(self, dbpool):self.dbpool = dbpool @classmethod def from_settings(cls, settings):dbparms = dict( host = settings['MYSQL_HOST'], db = settings['MYSQL_DBNAME'], user = settings['MYSQL_USER'], passwd = settings['MYSQL_PASSWORD'], charset=’utf8’, cursorclass=MySQLdb.cursors.DictCursor, use_unicode=True,)dbpool = adbapi.ConnectionPool('MySQLdb', **dbparms)return cls(dbpool) def process_item(self, item, spider):query = self.dbpool.runInteraction(self.do_insert, item)query.addErrback(self.handle_error, item, spider) def handle_error(self, failure, item, spider):print (failure) def do_insert(self, cursor, item):insert_sql, params = item.insert_values()print (insert_sql, params)cursor.execute(insert_sql, params)return item

Error code

2017-05-01 00:06:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/jobs/2108656.html> (referer: https://www.lagou.com/zhaopin/Python/)2017-05-01 00:06:16 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.lagou.com/jobs/2108656.html>None[Failure instance: Traceback: <class ’TypeError’>: connect() argument 3 must be str, not NoneD:Python35libthreading.py:914:_bootstrap_innerD:Python35libthreading.py:862:runD:Python35libsite-packagestwisted_threads_threadworker.py:46:workD:Python35libsite-packagestwisted_threads_team.py:190:doWork--- <exception caught here> ---D:Python35libsite-packagestwistedpythonthreadpool.py:250:inContextD:Python35libsite-packagestwistedpythonthreadpool.py:266:<lambda>D:Python35libsite-packagestwistedpythoncontext.py:122:callWithContextD:Python35libsite-packagestwistedpythoncontext.py:85:callWithContextD:Python35libsite-packagestwistedenterpriseadbapi.py:464:_runInteractionD:Python35libsite-packagestwistedenterpriseadbapi.py:36:__init__D:Python35libsite-packagestwistedenterpriseadbapi.py:76:reconnectD:Python35libsite-packagestwistedenterpriseadbapi.py:431:connectD:Python35libsite-packagesMySQLdb__init__.py:86:ConnectD:Python35libsite-packagesMySQLdbconnections.py:204:__init__]

问题解答

从提示看出,你的配置(settings)的时候有1个参数是None,而不是字符串