Scishing startproject gives strange error
I'm pretty sure this is due to issues with python2.7 and python3.5, but I haven't found a way to fix it.
I launched
$ scrapy startproject test1
and get this error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/bin/scrapy", line 9, in <module>
load_entry_point('Scrapy==1.0.1', 'console_scripts', 'scrapy')()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 122, in execute
cmds = _get_commands_dict(settings, inproject)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
for cmd in _iter_command_classes(module):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site- packages/scrapy/cmdline.py", line 21, in _iter_command_classes
for obj in vars(module).itervalues():
AttributeError: 'dict' object has no attribute 'itervalues'
source to share
It seems that the Python 3.5 dict type no longer has a 'itervalues' method. If they don't fix the problem, you may need to use a previous version of Python.
EDIT:
If you want, you can change the code and replace the itervalues () function with items (). I don't think it would be a good idea, but it might be a workaround if you have to use 3.5
source to share
python3, use dict.items () instead of dict.iteritems ()
iteritems () was removed in python3, so you can no longer use this method.
A simple fix would be doing a fresh install of Scrapy in python2.7 (Link python default 2.7) Then make a fresh insstalaltion from python-pip which will use python2.7 instead of Python3 + like: sudo apt-get install python-pip After installing the shim, use : sudo pip install Scrapy
For validation: SCRAPY sampling http://www.example.com/some/page.html
source to share
As others have said, you need to use Python 2.7 with Scrapy. Scrapy relies heavily on Twisted (albeit only a small part of it), which is incompatible with Python 3.
Since you plan on running multiple versions of Python on the same system, I would recommend that you take a look at virtualenv
(I would recommend this no matter what you do on the system, it's just THAT useful). Virtualenv is a tool that allows you to install and install a "virtual" python installation in your own directory with its own packages.
source to share