Scrapy spiders multiprocessing in parallel processes

There are a few similar questions here that I have read on Stack Overflow. Unfortunately, I lost all of them because my browsing history was deleted unexpectedly.

All of the above questions did not help me. Either some of them used CELERY or some of them SCRAPYD and I want to use the MULTIPROCESSISNG library. Also, the official Scrapy documentation shows you how to run multiple spiders in ONE PROCESS, not in MULTIPLE PROCESSES.

None of them could help me and so I decided to ask this question.

After several tries, I came up with this code .

My conclusion -:

Enter a product to search for: apple
2015-06-27 14:34:15 [scrapy] INFO: Scrapy 1.0.0 started (bot: scrapybot)
2015-06-27 14:34:15 [scrapy] INFO: Scrapy 1.0.0 started (bot: scrapybot)
2015-06-27 14:34:15 [scrapy] INFO: Optional features available: ssl, http11
2015-06-27 14:34:15 [scrapy] INFO: Optional features available: ssl, http11
2015-06-27 14:34:15 [scrapy] INFO: Overridden settings: {}
2015-06-27 14:34:15 [scrapy] INFO: Overridden settings: {}
2015-06-27 14:34:15 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2015-06-27 14:34:15 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2015-06-27 14:34:15 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-06-27 14:34:15 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-06-27 14:34:15 [scrapy] INFO: Enabled item pipelines: 
2015-06-27 14:34:15 [scrapy] INFO: Spider opened
2015-06-27 14:34:15 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-06-27 14:34:15 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-06-27 14:34:15 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-06-27 14:34:15 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-06-27 14:34:15 [scrapy] INFO: Enabled item pipelines: 
2015-06-27 14:34:15 [scrapy] INFO: Spider opened
2015-06-27 14:34:15 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-06-27 14:34:15 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024
2015-06-27 14:34:15 [twisted] ERROR: Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 88, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 73, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 619, in _doReadOrWrite
    why = selectable.doWrite()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1117, in doWrite
    "doWrite called on a %s" % reflect.qual(self.__class__))
exceptions.RuntimeError: doWrite called on a twisted.internet.tcp.Port

Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 88, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 73, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 619, in _doReadOrWrite
    why = selectable.doWrite()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1117, in doWrite
    "doWrite called on a %s" % reflect.qual(self.__class__))
exceptions.RuntimeError: doWrite called on a twisted.internet.tcp.Port
2015-06-27 14:34:16 [twisted] ERROR: Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 88, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 73, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 619, in _doReadOrWrite
    why = selectable.doWrite()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1117, in doWrite
    "doWrite called on a %s" % reflect.qual(self.__class__))
exceptions.RuntimeError: doWrite called on a twisted.internet.tcp.Port

Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 88, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 73, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 619, in _doReadOrWrite
    why = selectable.doWrite()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1117, in doWrite
    "doWrite called on a %s" % reflect.qual(self.__class__))
exceptions.RuntimeError: doWrite called on a twisted.internet.tcp.Port
2015-06-27 14:34:17 [scrapy] DEBUG: Crawled (200) <GET http://bigbasket.com/ps/?q=apple> (referer: None)
hello, world
Current second: 17
Current microsecond: 546862
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 170', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/10000007_18-fresho-apple-washington.jpg', 'product_link': 'http://bigbasket.com/pd/10000007/fresho-apple-washington-1-kg/', 'productname': 'Apple - Washington', 'current_price': 'Rs. 170'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 199', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/10000003_7-fresho-apple-fuji.jpg', 'product_link': 'http://bigbasket.com/pd/10000003/fresho-apple-fuji-1-kg/', 'productname': 'Apple - Fuji', 'current_price': 'Rs. 199'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 229', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/10000005_16-fresho-apple-royal-gala.jpg', 'product_link': 'http://bigbasket.com/pd/10000005/fresho-apple-royal-gala-1-kg/', 'productname': 'Apple - Royal Gala', 'current_price': 'Rs. 229'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 156.75', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/205988_2-american-garden-vinegar-apple-cider.jpg', 'product_link': 'http://bigbasket.com/pd/205988/american-garden-vinegar-apple-cider-473-ml-bottle/', 'productname': 'Vinegar - Apple Cider', 'current_price': 'Rs. 156.75'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 151', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/10000397_7-fresho-apple-green.jpg', 'product_link': 'http://bigbasket.com/pd/10000397/fresho-apple-green-500-gm/', 'productname': 'Apple - Green', 'current_price': 'Rs. 151'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 114', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/229785_5-tropicana-100-juice-apple.jpg', 'product_link': 'http://bigbasket.com/pd/229785/tropicana-100-juice-apple-1-ltr-tetra/', 'productname': '100% Juice - Apple', 'current_price': 'Rs. 114'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 266', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/40015763_1-mylife-vinegar-apple-cider.jpg', 'product_link': 'http://bigbasket.com/pd/40015763/mylife-vinegar-apple-cider-300-ml/', 'productname': 'Vinegar - Apple Cider', 'current_price': 'Rs. 266'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 175', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/40015525_1-fresho-apple-chilli.jpg', 'product_link': 'http://bigbasket.com/pd/40015525/fresho-apple-chilli-1-kg/', 'productname': 'Apple - Chilli', 'current_price': 'Rs. 175'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 94.05', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/229791_3-tropicana-juice-apple.jpg', 'product_link': 'http://bigbasket.com/pd/229791/tropicana-juice-apple-1-ltr-tetra/', 'productname': 'Juice - Apple', 'current_price': 'Rs. 94.05'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 93', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/40015526_1-fresho-apple-chilli.jpg', 'product_link': 'http://bigbasket.com/pd/40015526/fresho-apple-chilli-500-gm/', 'productname': 'Apple - Chilli', 'current_price': 'Rs. 93'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 94.05', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/265854_2-real-fruit-power-juice-apple.jpg', 'product_link': 'http://bigbasket.com/pd/265854/real-fruit-power-juice-apple-1-ltr-carton/', 'productname': 'Fruit Power Juice - Apple', 'current_price': 'Rs. 94.05'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 143.10', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/252445_3-biotique-shampoo-and-conditioner-bio-green-apple.jpg', 'product_link': 'http://bigbasket.com/pd/252445/biotique-shampoo-and-conditioner-bio-green-apple-190-ml/', 'productname': 'Shampoo and Conditioner - Bio Green...', 'current_price': 'Rs. 143.10'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 250', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/30006470_1-fresho-apple-fuji-premium.jpg', 'product_link': 'http://bigbasket.com/pd/30006470/fresho-apple-fuji-premium-1-kg/', 'productname': 'Apple Fuji Premium', 'current_price': 'Rs. 250'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 19', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/282654_2-real-fruit-power-juice-apple.jpg', 'product_link': 'http://bigbasket.com/pd/282654/real-fruit-power-juice-apple-200-ml-carton/', 'productname': 'Fruit Power Juice - Apple', 'current_price': 'Rs. 19'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 14.25', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/100535016_4-quaker-oats-strawberry-flavor-with-apple.jpg', 'product_link': 'http://bigbasket.com/pd/100535016/quaker-oats-strawberry-flavor-with-apple-40-gm-pouch/', 'productname': 'Oats - Strawberry Flavor with Apple', 'current_price': 'Rs. 14.25'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 12.60', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/265705_1-appy-apple-juice-drink-classic.jpg', 'product_link': 'http://bigbasket.com/pd/265705/appy-apple-juice-drink-classic-200-ml-carton/', 'productname': 'Apple Juice Drink - Classic', 'current_price': 'Rs. 12.60'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 19', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/40012961_2-candy-clouds-cotton-candy-orange-green-apple.jpg', 'product_link': 'http://bigbasket.com/pd/40012961/candy-clouds-cotton-candy-orange-green-apple-30-gm-cup/', 'productname': 'Cotton Candy - Orange & Green Apple', 'current_price': 'Rs. 19'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 96', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/229945_1-real-activ-juice-apple.jpg', 'product_link': 'http://bigbasket.com/pd/229945/real-activ-juice-apple-1-ltr-carton/', 'productname': 'Activ Juice - Apple', 'current_price': 'Rs. 96'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 23.75', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/286759_1-minute-maid-juice-apple.jpg', 'product_link': 'http://bigbasket.com/pd/286759/minute-maid-juice-apple-400-ml-bottle/', 'productname': 'Juice - Apple', 'current_price': 'Rs. 23.75'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://bigbasket.com/ps/?q=apple>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 85.50', 'imageurl': 'http://bigbasket.com/media/uploads/p/s/40020508_1-fresho-freshly-baked-apple-pie.jpg', 'product_link': 'http://bigbasket.com/pd/40020508/fresho-freshly-baked-apple-pie-100-gm-pouch/', 'productname': 'Freshly Baked - Apple Pie', 'current_price': 'Rs. 85.50'}
2015-06-27 14:34:17 [scrapy] INFO: Closing spider (finished)
2015-06-27 14:34:17 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 222,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 54881,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 6, 27, 9, 4, 17, 621449),
 'item_scraped_count': 20,
 'log_count/DEBUG': 22,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2015, 6, 27, 9, 4, 15, 879467)}
2015-06-27 14:34:17 [scrapy] INFO: Spider closed (finished)
2015-06-27 14:34:17 [scrapy] DEBUG: Crawled (200) <GET http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0> (referer: None)
hello, world
Current second: 17
Current microsecond: 734324
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 31,800', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/178153.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-4-16-GB-Unlocked-Mobile-Phone-(Black)-pc-19326-97.aspx', 'productname': 'Apple iPhone 4 16 GB Unlocked Mobile Phone (Black)', 'current_price': 'Rs. 26,999'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/180356.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-5c-32-GB-GSM-Mobile-Phone-(White)-pc-20258-97.aspx', 'productname': 'Apple iPhone 5c 32 GB GSM Mobile Phone (White)', 'current_price': 'Rs. 53,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 53,500', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/180360.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-5s-16-GB-GSM-Mobile-Phone-(Grey)-pc-20262-97.aspx', 'productname': 'Apple iPhone 5s 16 GB GSM Mobile Phone (Grey)', 'current_price': 'Rs. 44,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 53,500', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/180362.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-5s-16-GB-GSM-Mobile-Phone-(Gold)-pc-20263-97.aspx', 'productname': 'Apple iPhone 5s 16 GB GSM Mobile Phone (Gold)', 'current_price': 'Rs. 44,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 53,500', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/180376.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-5s-16-GB-GSM-Mobile-Phone-(Silver)-pc-20277-97.aspx', 'productname': 'Apple iPhone 5s 16 GB GSM Mobile Phone (Silver)', 'current_price': 'Rs. 44,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 31,500', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/180443.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-4S-8-GB-GSM-Mobile-Phone-(Black)-pc-20318-97.aspx', 'productname': 'Apple iPhone 4S 8 GB GSM Mobile Phone (Black)', 'current_price': 'Rs. 16,990'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 31,500', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/180444.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-4S-8-GB-GSM-Mobile-Phone-(White)-pc-20319-97.aspx', 'productname': 'Apple iPhone 4S 8 GB GSM Mobile Phone (White)', 'current_price': 'Rs. 16,990'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 53,500', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/185039.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-5S-16-GB-GSM-Mobile-Phone-(Gold)-pc-23555-97.aspx', 'productname': 'Apple iPhone 5S 16 GB GSM Mobile Phone (Gold)', 'current_price': 'Rs. 49,999'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 53,500', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/185802.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-16-GB-GSM-Mobile-Phone-(Silver)-pc-23996-97.aspx', 'productname': 'Apple iPhone 6 16 GB GSM Mobile Phone (Silver)', 'current_price': 'Rs. 52,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/185805.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-64-GB-GSM-Mobile-Phone-(Silver)-pc-23999-97.aspx', 'productname': 'Apple iPhone 6 64 GB GSM Mobile Phone (Silver)', 'current_price': 'Rs. 62,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/185808.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-128-GB-GSM-(Silver)-pc-24002-97.aspx', 'productname': 'Apple iPhone 6 128 GB GSM (Silver)', 'current_price': 'Rs. 71,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/185880.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-Plus-16-GB-GSM-(Space-Grey)-pc-24004-97.aspx', 'productname': 'Apple iPhone 6 Plus 16 GB GSM (Space Grey)', 'current_price': 'Rs. 62,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/185881.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-Plus-16-GB-GSM-(Silver)-pc-24005-97.aspx', 'productname': 'Apple iPhone 6 Plus 16 GB GSM (Silver)', 'current_price': 'Rs. 62,500'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': 'Rs. 56,000', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/189360.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-16-GB-(Gold)-pc-26002-97.aspx', 'productname': 'Apple iPhone 6 16 GB (Gold)', 'current_price': 'Rs. 53,499'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/189363.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-64-GB-(Gold)-pc-26005-97.aspx', 'productname': 'Apple iPhone 6 64 GB (Gold)', 'current_price': 'Rs. 65,000'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/189364.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-64-GB-(Grey)-pc-26006-97.aspx', 'productname': 'Apple iPhone 6 64 GB (Grey)', 'current_price': 'Rs. 65,000'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/189365.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-64-GB-(Silver)-pc-26007-97.aspx', 'productname': 'Apple iPhone 6 64 GB (Silver)', 'current_price': 'Rs. 65,000'}
2015-06-27 14:34:17 [scrapy] DEBUG: Scraped from <200 http://www.cromaretail.com/productsearch.aspx?txtSearch=apple&x=0&y=0>
{'outofstock_status': 'In Stock', 'offer': 'No additional offer available', 'mrp': '', 'imageurl': 'http://www.cromaretail.com/Images/Catalogue/Product/medium/189366.jpg', 'product_link': 'http://www.cromaretail.comApple-iPhone-6-128-GB-(Gold)-pc-26008-97.aspx', 'productname': 'Apple iPhone 6 128 GB (Gold)', 'current_price': 'Rs. 74,000'}
2015-06-27 14:34:17 [scrapy] INFO: Closing spider (finished)
2015-06-27 14:34:17 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 259,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 16851,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 6, 27, 9, 4, 17, 764861),
 'item_scraped_count': 18,
 'log_count/DEBUG': 20,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2015, 6, 27, 9, 4, 15, 930386)}
2015-06-27 14:34:17 [scrapy] INFO: Spider closed (finished)
<Deferred at 0x7f02a29b7c68 current result: None>

      

        
        
        
      

    

If you see my output correctly, some error occurs initially and the program stops a little earlier

2015-06-27 14:34:17 [scrapy] DEBUG: Crawled (200) <GET http://bigbasket.com/ps/?q=apple> (referer: None) 

      

        
        
        
      

    

- this line is on my output and then runs but also outputs the result.

I cannot figure out the following two things:

• Why is this error occurring? Why is the program paused there (is it because this check is scanning the pages, and that's ok, or is it some kind of problem?)? What changes should I make to my code to fix this problem?
• If you see that I am printing the current second and the current microsecond, check when my function is entered and start processing. I did this mainly to check if both of my functions are actually getting multiprocessing or not. I know that even after using Multiprocessing there is a time lag of a few microseconds and this fits very well from my end. But I ran this script several times, and sometimes I noticed a time lag of just a few microseconds (which I could well accept, but this case usually occurred when sites only returned a few search results, so I don't know if they actually got multiprocessing or no, because usually my functions run independently in 1.3 seconds and produce about 20 results.),and sometimes I notice a time delay of about 1 or 2 seconds (which definitely suggests that my functions I got multiprocessing, at least after reading about the time - about 1.3 seconds, in which my functions were executed independently - were they are in different python scripts). So why does this time variation occur? How can I check if my functions are actually getting multiprocessing?

Please make corrections to my code and clarify my 2 questions.

Please, help! Any answers should be well appreciated!

Thank!