scrapy调用httpproxy

middleware.py中构建proxy中间件

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] =  crawler.settings.get('HTTPPROXY')
        pass

settings.py中启用proxymiddleware 和设定proxy

DOWNLOADER_MIDDLEWARES = {
    'mingyan.middlewares.ProxyMiddleware': 1,
}
HTTPPROXY="http://127.0.0.1:80"