Spider spider is not interrupted using CloseSpider extension

I came up with a solution that helped the wig answer as well as my own research. It does have some kind of unexplained behavior that I will cover (comments appreciated).

In my spider file myspider_spider.py

I have (edited for brevity):

import scrapy
from scrapy.spiders import XMLFeedSpider
from scrapy.exceptions import CloseSpider
from myspiders.items import MySpiderItem

class MySpiderSpider(XMLFeedSpider):
    name = "myspiders"
    allowed_domains = {"www.mysource.com"}
    start_urls = [
        "https://www.mysource.com/source.xml"
        ]
    iterator = 'iternodes'
    itertag = 'item'
    item_count = 0

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(settings)

    def __init__(self, settings):
        self.settings = settings

    def parse_node(self, response, node):
        if(self.settings['CLOSESPIDER_ITEMCOUNT'] and int(self.settings['CLOSESPIDER_ITEMCOUNT']) == self.item_count):
            raise CloseSpider('CLOSESPIDER_ITEMCOUNT limit reached - ' + str(self.settings['CLOSESPIDER_ITEMCOUNT']))
        else:
            self.item_count += 1
        id = node.xpath('id/text()').extract()
        title = node.xpath('title/text()').extract()
        item = MySpiderItem()
        item['id'] = id
        item['title'] = title

        return item

      

        
        
        
      

    

This works - if I set CLOSESPIDER_ITEMCOUNT

to 10, it ends up after 10 elements have been processed (so it seems to be ignoring in this regard CONCURRENT_REQUESTS

- this was unexpected).

I commented this out in mine settings.py

:

#EXTENSIONS = {
#   'scrapy.extensions.closespider.CloseSpider': 500,
#}

      

        
        
        
      

    

So, it just uses the exception CloseSpider

. However, the log shows the following:

2017-06-16 10:04:15 [scrapy.core.engine] INFO: Closing spider (closespider_itemcount)
2017-06-16 10:04:15 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (10 items) in: file:///tmp/myspiders/export.jsonl
2017-06-16 10:04:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 600,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 8599860,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'closespider_itemcount',
 'finish_time': datetime.datetime(2017, 6, 16, 9, 4, 15, 615501),
 'item_scraped_count': 10,
 'log_count/DEBUG': 8,
 'log_count/INFO': 8,
 'response_received_count': 2,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 6, 16, 9, 3, 47, 966791)}
2017-06-16 10:04:15 [scrapy.core.engine] INFO: Spider closed (closespider_itemcount)

      

        
        
        
      

    

The main thing is to highlight the first line INFO

and finish_reason

- the message displayed in INFO

is not the one I am setting when throwing the exception CloseSpider

. This implies an expansion CloseSpider

that stops the spider, but I know it isn't? Very confusing.