How do I install xvfb on Scrapinghub to use Selenium?
I am using Python-Selenium in my spider (Scrapy), to use Selenium I have to install xvfb on Scrapinghub .
when i use apt-get
xvfb to install i have this error message:
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?
Is there any other way to install xvfb on Scrapinghub ?
UPDATE 1
I read this , I tried using docker , I am stuck at this step
shub-image init --requirements path/to/requirements.txt
I read this
If you get an ImportError like when running shub-image init: You have to make sure you have the latest version of shub installed working:
$ pip install shub --upgrade
but i always have this error
Traceback (most recent call last):
File "/usr/local/bin/shub-image", line 7, in <module>
from shub_image.tool import cli
File "/usr/local/lib/python2.7/dist-packages/shub_image/tool.py", line 42, in <module>
command_module = importlib.import_module(module_path)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/dist-packages/shub_image/push.py", line 4, in <module>
from shub.deploy import list_targets
ImportError: cannot import name list_targets
source to share
I solved my problems ( use selenium in scrapinghub )
1 - for xvfb in docker, which I am using
RUN apt-get install -qy xvfb
2 - to create docker image I used this and to install geckodriver I use this code
#
# Geckodriver Dockerfile
#
FROM blueimp/basedriver
# Add the Firefox release channel of the Debian Mozilla team:
RUN echo 'deb http://mozilla.debian.net/ jessie-backports firefox-release' >> \
/etc/apt/sources.list \
&& curl -sL https://mozilla.debian.net/archive.asc | apt-key add -
# Install Firefox:
RUN export DEBIAN_FRONTEND=noninteractive \
&& apt-get update \
&& apt-get install --no-install-recommends --no-install-suggests -y \
firefox \
# Remove obsolete files:
&& apt-get clean \
&& rm -rf \
/tmp/* \
/usr/share/doc/* \
/var/cache/* \
/var/lib/apt/lists/* \
/var/tmp/*
# Install geckodriver:
RUN export BASE_URL=https://github.com/mozilla/geckodriver/releases/download \
&& export VERSION=$(curl -sL \
https://api.github.com/repos/mozilla/geckodriver/releases/latest | \
grep tag_name | cut -d '"' -f 4) \
&& curl -sL \
$BASE_URL/$VERSION/geckodriver-$VERSION-linux64.tar.gz | tar -xz \
&& mv geckodriver /usr/local/bin/geckodriver
USER webdriver
CMD ["geckodriver", "--host", "0.0.0.0"]
from here
source to share
Have you tried:
sudo apt-get install xvfb
Another way is to manually compile packages like:
apt-get source xvfb
./configure --prefix=$HOME/myapps
make
make install
And the third way, download the .deb from the original web page https://pkgs.org/download/xvfb
after downloading, you can mv it to the path of the downloaded sources:
mv xvfb_1.16.4-1_amd64.deb /var/cache/apt/archives/
then you change your directory and do:
sudo dpkg -i xvfb_1.16.4-1_amd64.deb
and that's all!
source to share