scrapy(三) logging

2021-06-09  本文已影响0人  万事万物

前言

使用 scrapy crawl XXX 启动爬虫的时候会产生很多启动日志信息，如下：

$ scrapy crawl quotes
2021-06-06 21:42:56 [scrapy.utils.log] INFO: Scrapy 2.5.0 started (bot: tutorial)
2021-06-06 21:42:56 [scrapy.utils.log] INFO: Versions: lxml 4.6.3.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1k  25 Mar 2021), cryptography 3.4.7, Platform Windows-10-10.0.19043-SP0
2021-06-06 21:42:56 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-06-06 21:42:56 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tutorial',
 'NEWSPIDER_MODULE': 'tutorial.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['tutorial.spiders']}
2021-06-06 21:42:56 [scrapy.extensions.telnet] INFO: Telnet Password: bc25f76010d8e84c
2021-06-06 21:42:56 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2021-06-06 21:42:56 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-06-06 21:42:56 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-06-06 21:42:56 [scrapy.middleware] INFO: Enabled item pipelines:
['tutorial.pipelines.TutorialPipeline']

对于我们来说，我们并不关心这些日志（只要能正常启动），我们往往只关心爬取的数据。如：

tein', 'tags': [{'href': '/tag/change/page/1/', 'text': 'change'}, {'href': '/tag/deep-thoughts/page/1/', 'text': 'deep-thoughts'}, {'href': '/tag/thinking/page/1/', 'text': 'thinking'}, {'href': '/tag/world/page/1/', 'text': 'world'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/change/page/1/', 'text': 'change'}, {'href': '/tag/deep-thoughts/page/1/', 'text': 'deep-thoughts'}, {'href': '/tag/thinking/page/1/', 'text': 'thinking'}, {'href': '/tag/world/page/1/', 'text': 'world'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'by_text': 'J.K. Rowling', 'by_href': '/author/J-K-Rowling', 'tags': [{'href': '/tag/abilities/page/1/', 'text': 'abilities'}, {'href': '/tag/choices/page/1/', 'text': 'choices'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'by_text': 'J.K. Rowling', 'by_href': '/author/J-K-Rowling', 'tags': [{'href': '/tag/abilities/page/1/', 'text': 'abilities'}, 
{'href': '/tag/choices/page/1/', 'text': 'choices'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/live/page/1/', 'text': 'live'}, {'href': '/tag/miracle/page/1/', 'text': 'miracle'}, {'href': '/tag/miracles/page/1/', 'text': 'miracles'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/live/page/1/', 'text': 'live'}, {'href': '/tag/miracle/page/1/', 'text': 'miracle'}, {'href': '/tag/miracles/page/1/', 'text': 'miracles'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'by_text': 'Jane Austen', 'by_href': '/author/Jane-Austen', 'tags': [{'href': '/tag/aliteracy/page/1/', 'text': 'aliteracy'}, {'href': '/tag/books/page/1/', 'text': 'books'}, {'href': '/tag/classic/page/1/', 'text': 'classic'}, {'href': '/tag/humor/page/1/', 'text': 'humor'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'by_text': 'Jane Austen', 'by_href': '/author/Jane-Austen', 'tags': [{'href': '/tag/aliteracy/page/1/', 'text': 'aliteracy'}, {'href': '/tag/books/page/1/', 'text': 'books'}, {'href': '/tag/classic/page/1/', 'text': 'classic'}, {'href': '/tag/humor/page/1/', 'text': 'humor'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'by_text': 'Marilyn Monroe', 'by_href': '/author/Marilyn-Monroe', 'tags': [{'href': '/tag/be-yourself/page/1/', 'text': 'be-yourself'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'by_text': 'Marilyn Monroe', 'by_href': '/author/Marilyn-Monroe', 'tags': [{'href': '/tag/be-yourself/page/1/', 'text': 'be-yourself'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': '“Try not to become a man of success. Rather become a man of value.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/adulthood/page/1/', 'text': 'adulthood'}, {'href': '/tag/success/page/1/', 'text': 'success'}, {'href': '/tag/value/page/1/', 'text': 'value'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“Try not to become a man of success. Rather become a man of value.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/adulthood/page/1/', 'text': 'adulthood'}, {'href': '/tag/success/page/1/', 'text': 'success'}, {'href': '/tag/value/page/1/', 'text': 'value'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': '“It is better to be hated for what you are than to be loved for what you are not.”', 'by_text': 'André Gide', 'by_href': '/author/Andre-Gide', 'tags': [{'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/love/page/1/', 'text': 'love'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“It is better to be hated for what you are than to be loved for what you are not.”', 'by_text': 'André Gide', 'by_href': '/author/Andre-Gide', 'tags': [{'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/love/page/1/', 'text': 'love'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': "“I have not failed. I've just found 10,000 ways that won't work.”", 'by_text': 'Thomas A. Edison', 'by_href': '/author/Thomas-A-Edison', 'tags': [{'href': '/tag/edison/page/1/', 'text': 'edison'}, {'href': '/tag/failure/page/1/', 'text': 'failure'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/paraphrased/page/1/', 'text': 'paraphrased'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': "“I have not failed. I've just found 10,000 ways that won't work.”", 'by_text': 'Thomas A. Edison', 'by_href': '/author/Thomas-A-Edison', 'tags': [{'href': '/tag/edison/page/1/', 'text': 'edison'}, {'href': '/tag/failure/page/1/', 'text': 'failure'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/paraphrased/page/1/', 'text': 'paraphrased'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”", 'by_text': 'Eleanor Roosevelt', 'by_href': '/author/Eleanor-Roosevelt', 'tags': [{'href': '/tag/misattributed-eleanor-roosevelt/page/1/', 'text': 'misattributed-eleanor-roosevelt'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------
2021-06-06 21:42:57 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”", 'by_text': 'Eleanor Roosevelt', 'by_href': '/author/Eleanor-Roosevelt', 'tags': [{'href': '/tag/misattributed-eleanor-roosevelt/page/1/', 'text': 'misattributed-eleanor-roosevelt'}]}
2021-06-06 21:42:57 [root] WARNING: {'text': '“A day without sunshine is like, you know, night.”', 'by_text': 'Steve Martin', 'by_href': '/author/Steve-Martin', 'tags': [{'href': '/tag/humor/page/1/', 'text': 'humor'}, {'href': '/tag/obvious/page/1/', 'text': 'obvious'}, {'href': '/tag/simile/page/1/', 'text': 'simile'}]}
2021-06-06 21:42:57 [root] WARNING: --------------------

LOG_LEVEL

在scrapy框架中，我们可以在 settings.py 设置日志级别的方式过滤一些无关重要的日志。
只需要在 settings.py 中指定 LOG_LEVEL 就可以配置日志级别。

注意：默认settings.py没有LOG_LEVEL，直接写就行了

LOG_LEVEL="WARNING"

LOG_LEVEL共五个日志等级

  1. CRITICAL - 严重错误(critical)
  2. ERROR - 一般错误(regular errors)
  3. WARNING - 警告信息(warning messages)
  4. INFO - 一般信息(informational messages)
  5. DEBUG - 调试信息(debugging messages)

我们重新启动爬虫，输出的只有我们的业务信息了。

$ scrapy crawl quotes
2021-06-06 22:06:59 [root] WARNING: {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/change/page/1/', 'text': 'change'}, {'href': '/tag/deep-thoughts/page/1/', 'text': 'deep-thoughts'}, {'href': '/tag/thinking/page/1/', 'text': 'thinking'}, {'href': '/tag/world/page/1/', 'text': 'world'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'by_text': 'J.K. Rowling', 'by_href': '/author/J-K-Rowling', 'tags': [{'href': '/tag/abilities/page/1/', 'text': 'abilities'}, {'href': '/tag/choices/page/1/', 'text': 'choices'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/live/page/1/', 'text': 'live'}, {'href': '/tag/miracle/page/1/', 'text': 'miracle'}, {'href': '/tag/miracles/page/1/', 'text': 'miracles'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'by_text': 'Jane Austen', 'by_href': '/author/Jane-Austen', 'tags': [{'href': '/tag/aliteracy/page/1/', 'text': 'aliteracy'}, {'href': '/tag/books/page/1/', 'text': 'books'}, {'href': '/tag/classic/page/1/', 'text': 'classic'}, {'href': '/tag/humor/page/1/', 'text': 'humor'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'by_text': 'Marilyn Monroe', 'by_href': '/author/Marilyn-Monroe', 'tags': [{'href': '/tag/be-yourself/page/1/', 'text': 'be-yourself'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': '“Try not to become a man of success. Rather become a man of value.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/adulthood/page/1/', 'text': 'adulthood'}, {'href': '/tag/success/page/1/', 'text': 'success'}, {'href': '/tag/value/page/1/', 'text': 'value'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': '“It is better to be hated for what you are than to be loved for what you are not.”', 'by_text': 'André Gide', 'by_href': '/author/Andre-Gide', 'tags': [{'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/love/page/1/', 'text': 'love'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': "“I have not failed. I've just found 10,000 ways that won't work.”", 'by_text': 'Thomas A. Edison', 'by_href': '/author/Thomas-A-Edison', 'tags': [{'href': '/tag/edison/page/1/', 'text': 'edison'}, {'href': '/tag/failure/page/1/', 'text': 'failure'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/paraphrased/page/1/', 'text': 'paraphrased'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”", 'by_text': 'Eleanor Roosevelt', 'by_href': '/author/Eleanor-Roosevelt', 'tags': [{'href': '/tag/misattributed-eleanor-roosevelt/page/1/', 'text': 'misattributed-eleanor-roosevelt'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------
2021-06-06 22:06:59 [root] WARNING: {'text': '“A day without sunshine is like, you know, night.”', 'by_text': 'Steve Martin', 'by_href': '/author/Steve-Martin', 'tags': [{'href': '/tag/humor/page/1/', 'text': 'humor'}, {'href': '/tag/obvious/page/1/', 'text': 'obvious'}, {'href': '/tag/simile/page/1/', 'text': 'simile'}]}
2021-06-06 22:06:59 [root] WARNING: --------------------

LOG_FILE

除此之外，也可以将日志写入文件。
在 settings.py 中直接指定LOG_FILE即可

LOG_FILE="./tutorial.log" # ./表示保存到当前项目的目录下

这样我们结果将直接写到文件中，而不是输出到控制台

2021-06-06 22:04:48 [root] WARNING: {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/change/page/1/', 'text': 'change'}, {'href': '/tag/deep-thoughts/page/1/', 'text': 'deep-thoughts'}, {'href': '/tag/thinking/page/1/', 'text': 'thinking'}, {'href': '/tag/world/page/1/', 'text': 'world'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'by_text': 'J.K. Rowling', 'by_href': '/author/J-K-Rowling', 'tags': [{'href': '/tag/abilities/page/1/', 'text': 'abilities'}, {'href': '/tag/choices/page/1/', 'text': 'choices'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/live/page/1/', 'text': 'live'}, {'href': '/tag/miracle/page/1/', 'text': 'miracle'}, {'href': '/tag/miracles/page/1/', 'text': 'miracles'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'by_text': 'Jane Austen', 'by_href': '/author/Jane-Austen', 'tags': [{'href': '/tag/aliteracy/page/1/', 'text': 'aliteracy'}, {'href': '/tag/books/page/1/', 'text': 'books'}, {'href': '/tag/classic/page/1/', 'text': 'classic'}, {'href': '/tag/humor/page/1/', 'text': 'humor'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'by_text': 'Marilyn Monroe', 'by_href': '/author/Marilyn-Monroe', 'tags': [{'href': '/tag/be-yourself/page/1/', 'text': 'be-yourself'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': '“Try not to become a man of success. Rather become a man of value.”', 'by_text': 'Albert Einstein', 'by_href': '/author/Albert-Einstein', 'tags': [{'href': '/tag/adulthood/page/1/', 'text': 'adulthood'}, {'href': '/tag/success/page/1/', 'text': 'success'}, {'href': '/tag/value/page/1/', 'text': 'value'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': '“It is better to be hated for what you are than to be loved for what you are not.”', 'by_text': 'André Gide', 'by_href': '/author/Andre-Gide', 'tags': [{'href': '/tag/life/page/1/', 'text': 'life'}, {'href': '/tag/love/page/1/', 'text': 'love'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': "“I have not failed. I've just found 10,000 ways that won't work.”", 'by_text': 'Thomas A. Edison', 'by_href': '/author/Thomas-A-Edison', 'tags': [{'href': '/tag/edison/page/1/', 'text': 'edison'}, {'href': '/tag/failure/page/1/', 'text': 'failure'}, {'href': '/tag/inspirational/page/1/', 'text': 'inspirational'}, {'href': '/tag/paraphrased/page/1/', 'text': 'paraphrased'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”", 'by_text': 'Eleanor Roosevelt', 'by_href': '/author/Eleanor-Roosevelt', 'tags': [{'href': '/tag/misattributed-eleanor-roosevelt/page/1/', 'text': 'misattributed-eleanor-roosevelt'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------
2021-06-06 22:04:48 [root] WARNING: {'text': '“A day without sunshine is like, you know, night.”', 'by_text': 'Steve Martin', 'by_href': '/author/Steve-Martin', 'tags': [{'href': '/tag/humor/page/1/', 'text': 'humor'}, {'href': '/tag/obvious/page/1/', 'text': 'obvious'}, {'href': '/tag/simile/page/1/', 'text': 'simile'}]}
2021-06-06 22:04:48 [root] WARNING: --------------------

logging

在开发中我们也需要使用日志，总不能总将信息使用print 打印到控制台吧，在python中，有专门的模块——logging。
使用方式，首先需要导入到项目中

from itemadapter import ItemAdapter
import logging

class TutorialPipeline:
    def process_item(self, item, spider):
        # 这里采用logger 将数据写入日志中
        logging.warning(item)
        logging.warning("-"*20)
        return item

更多使用方式：https://zhuanlan.zhihu.com/p/149765686?from_voters_page=true

上一篇下一篇

猜你喜欢

热点阅读