Python – Spynner Installation in Ubuntu

Spynner is a stateful programmatic web browser module for Python with Javascript/AJAX support based upon the QtWebKit framework.

Quick Install

# build LibQt4 dependencies
sudo apt-get build-dep -y libqt4-dev
# install python Qt4 library
sudo apt-get install -y python-qt4
# x11
sudo apt-get install -y libxtst-dev xvfb x11-xkb-utils
# fonts
sudo apt-get install -y xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic
sudo apt-get install -y x-ttcidfont-conf cabextract ttf-mscorefonts-installer
sudo dpkg-reconfigure --default-priority x-ttcidfont-conf
# install python-dev and pip
sudo apt-get install -y python-dev python-setuptools python-pip
# spynner
sudo pip install spynner

Step by Step Installation

Install QtWebKit Framework and dependencies

sudo apt-get build-dep -y libqt4-dev

Installation using PIP

sudo pip install spynner

Alternative Installation from the Source

mkdir -p ~/build
cd  ~/build
git clone https://github.com/makinacorpus/spynner
cd spynner
sudo python setup.py install

init.d script

cat << "_EOF_" | sudo tee /etc/init.d/xvfb
#!/bin/bash

if [ -z "$1" ]; then
echo "`basename $0` {start|stop}"
exit
fi

case "$1" in
start)
/usr/bin/Xvfb :99 -ac -screen 0 1024x768x8 &
;;

stop)
killall Xvfb
;;
esac
_EOF_

sudo chmod 755 /etc/init.d/xvfb
/etc/init.d/xvfb start

Troubleshooting

1. X11 Errors

src/keypress.c:10:35: fatal error: X11/extensions/XTest.h: No such file or directory
compilation terminated.

Fix

sudo apt-get install -y libxtst-dev xvfb

2. QT Errors

Traceback (most recent call last):
  File "spynner-test.py", line 2, in 
    import spynner
  File "/usr/local/lib/python2.7/dist-packages/spynner/__init__.py", line 2, in 
    from browser import *
  File "/usr/local/lib/python2.7/dist-packages/spynner/browser.py", line 51, in 
    from PyQt4 import QtCore
ImportError: No module named PyQt4

Fix

sudo apt-get install -y python-qt4

3. Font Errors

Save it to /etc/init.d/xvfb and chmod 755, then confirm that it works:
[dix] Could not init font path element /usr/share/fonts/X11/cyrillic, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/100dpi/:unscaled, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/75dpi/:unscaled, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/100dpi, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/75dpi, removing from list!
[dix] Could not init font path element /var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType, removing from list!

Fix

sudo apt-get install -y x11-xkb-utils

# add fonts
sudo apt-get install -y xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic
sudo apt-get install -y x-ttcidfont-conf cabextract ttf-mscorefonts-installer
#(you'll have to enable the multiverse repo to get ttf-mscorefonts-installer)
#Accept the EULA terms for ttf-mscorefonts-installer.
#Then:
sudo dpkg-reconfigure --default-priority x-ttcidfont-conf

References

1. Home: https://github.com/makinacorpus/spynner
2. xvfb Init.d script: http://www.labelmedia.co.uk/blog/setting-up-selenium-server-on-a-headless-jenkins-ci-build-machine.html
3. Installation Error Fixes: http://www.yodi.sg/install-spynner-on-virtualenv/
4. Font Fixes: http://www.cyberciti.biz/tips/linux-howto-install-truetype-freetype-and-msttcorefonts-fonts.html
5. xvfb Installation: http://blog.martin-lyness.com/tag/xvfb
6. Installation of Qt4 dependencies: http://qt-project.org/wiki/Compile_Qt_4.7_on_Ubuntu_10.10
7. Installation of Qt4: http://stackoverflow.com/questions/7119844/importerror-no-module-named-pyqt4-qtcore
8. spynner Test Scripts /2760/python-spynner-test-scripts