[PySpider]抓取架构

---

Master

pyspider00

data_db(存储抓取数据)

• phantomjs
• scheduler
• processor
• --phantomjs-proxy='localhost:25555' fetcher
• result_worker
• webui

Slave_01

pyspider01

taskdb,projectdb,resultdb(存储主要任务和脚本数据,比较重要)

• phantomjs
• processor
• --phantomjs-proxy="localhost:25555" fetcher

Slave_02

pyspider02

• phantomjs
• processor
• --phantomjs-proxy="localhost:25555" fetcher