This is a scraper that runs on Morph. To get started see the documentation

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... -----> Python app detected -----> Stack changed, re-installing runtime -----> Installing runtime (python-2.7.9) -----> Installing dependencies with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to morph_defaults) to ./.heroku/src/scraperwiki  Collecting lxml==3.4.4 (from -r requirements.txt (line 8))  Downloading lxml-3.4.4.tar.gz (3.5MB)  Building lxml version 3.4.4.  Building without Cython.  Using build configuration of libxslt 1.1.28  /app/.heroku/python/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'  warnings.warn(msg)  Collecting cssselect==0.9.1 (from -r requirements.txt (line 9))  Downloading cssselect-0.9.1.tar.gz  Collecting splinter>=0.7.3 (from -r requirements.txt (line 10))  Downloading splinter-0.7.3.tar.gz  Collecting scrapy==0.24 (from -r requirements.txt (line 11))  Downloading Scrapy-0.24.0-py2-none-any.whl (501kB)  Collecting service-identity==14.0.0 (from -r requirements.txt (line 12))  Downloading service_identity-14.0.0-py2.py3-none-any.whl  Collecting dumptruck>=0.1.2 (from scraperwiki->-r requirements.txt (line 6))  Downloading dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r requirements.txt (line 6))  Downloading requests-2.7.0-py2.py3-none-any.whl (470kB)  Collecting selenium>=2.47.1 (from splinter>=0.7.3->-r requirements.txt (line 10))  Downloading selenium-2.47.1-py2-none-any.whl (3.0MB)  Collecting queuelib (from scrapy==0.24->-r requirements.txt (line 11))  Downloading queuelib-1.3.0-py2.py3-none-any.whl  Collecting pyOpenSSL (from scrapy==0.24->-r requirements.txt (line 11))  Downloading pyOpenSSL-0.15.1-py2.py3-none-any.whl (102kB)  Collecting Twisted>=10.0.0 (from scrapy==0.24->-r requirements.txt (line 11))  Downloading Twisted-15.4.0.tar.bz2 (3.1MB)  Collecting six>=1.5.2 (from scrapy==0.24->-r requirements.txt (line 11))  Downloading six-1.9.0-py2.py3-none-any.whl  Collecting w3lib>=1.2 (from scrapy==0.24->-r requirements.txt (line 11))  Downloading w3lib-1.12.0-py2.py3-none-any.whl  Collecting characteristic>=14.0.0 (from service-identity==14.0.0->-r requirements.txt (line 12))  Downloading characteristic-14.3.0-py2.py3-none-any.whl  Collecting pyasn1-modules (from service-identity==14.0.0->-r requirements.txt (line 12))  Downloading pyasn1-modules-0.0.7.tar.gz  Collecting pyasn1 (from service-identity==14.0.0->-r requirements.txt (line 12))  Downloading pyasn1-0.1.8.tar.gz (75kB)  Collecting cryptography>=0.7 (from pyOpenSSL->scrapy==0.24->-r requirements.txt (line 11))  Downloading cryptography-1.0.tar.gz (331kB)  Collecting zope.interface>=3.6.0 (from Twisted>=10.0.0->scrapy==0.24->-r requirements.txt (line 11))  Downloading zope.interface-4.1.2.tar.gz (919kB)  Collecting idna>=2.0 (from cryptography>=0.7->pyOpenSSL->scrapy==0.24->-r requirements.txt (line 11))  Downloading idna-2.0-py2.py3-none-any.whl (61kB)  Collecting enum34 (from cryptography>=0.7->pyOpenSSL->scrapy==0.24->-r requirements.txt (line 11))  Downloading enum34-1.0.4.tar.gz  Collecting ipaddress (from cryptography>=0.7->pyOpenSSL->scrapy==0.24->-r requirements.txt (line 11))  Downloading ipaddress-1.0.14-py27-none-any.whl  Collecting cffi>=1.1.0 (from cryptography>=0.7->pyOpenSSL->scrapy==0.24->-r requirements.txt (line 11))  Downloading cffi-1.2.1.tar.gz (335kB)  Collecting pycparser (from cffi>=1.1.0->cryptography>=0.7->pyOpenSSL->scrapy==0.24->-r requirements.txt (line 11))  Downloading pycparser-2.14.tar.gz (223kB)  Installing collected packages: pycparser, cffi, ipaddress, enum34, idna, zope.interface, cryptography, pyasn1, pyasn1-modules, characteristic, w3lib, six, Twisted, pyOpenSSL, queuelib, selenium, requests, dumptruck, service-identity, scrapy, splinter, cssselect, lxml, scraperwiki  Running setup.py install for pycparser  Build the lexing/parsing tables  Running setup.py install for cffi  building '_cffi_backend' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/app/.heroku/python/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/c/_cffi_backend.o -lffi -o build/lib.linux-x86_64-2.7/_cffi_backend.so   Running setup.py install for enum34   Running setup.py install for zope.interface  building 'zope.interface._zope_interface_coptimizations' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c src/zope/interface/_zope_interface_coptimizations.c -o build/temp.linux-x86_64-2.7/src/zope/interface/_zope_interface_coptimizations.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/src/zope/interface/_zope_interface_coptimizations.o -o build/lib.linux-x86_64-2.7/zope/interface/_zope_interface_coptimizations.so  Skipping installation of /app/.heroku/python/lib/python2.7/site-packages/zope/__init__.py (namespace package)  Installing /app/.heroku/python/lib/python2.7/site-packages/zope.interface-4.1.2-py2.7-nspkg.pth  Running setup.py install for cryptography  generating cffi module 'build/temp.linux-x86_64-2.7/_padding.c'  generating cffi module 'build/temp.linux-x86_64-2.7/_constant_time.c'  generating cffi module 'build/temp.linux-x86_64-2.7/_openssl.c'  building '_openssl' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c build/temp.linux-x86_64-2.7/_openssl.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o -lssl -lcrypto -o build/lib.linux-x86_64-2.7/cryptography/hazmat/bindings/_openssl.so  building '_constant_time' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c build/temp.linux-x86_64-2.7/_constant_time.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_constant_time.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_constant_time.o -o build/lib.linux-x86_64-2.7/cryptography/hazmat/bindings/_constant_time.so  building '_padding' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c build/temp.linux-x86_64-2.7/_padding.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_padding.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_padding.o -o build/lib.linux-x86_64-2.7/cryptography/hazmat/bindings/_padding.so  Running setup.py install for pyasn1  Running setup.py install for pyasn1-modules     Running setup.py install for Twisted  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c conftest.c -o conftest.o  building 'twisted.test.raiser' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c twisted/test/raiser.c -o build/temp.linux-x86_64-2.7/twisted/test/raiser.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/twisted/test/raiser.o -o build/lib.linux-x86_64-2.7/twisted/test/raiser.so  building 'twisted.python._sendmsg' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c twisted/python/_sendmsg.c -o build/temp.linux-x86_64-2.7/twisted/python/_sendmsg.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/twisted/python/_sendmsg.o -o build/lib.linux-x86_64-2.7/twisted/python/_sendmsg.so  building 'twisted.runner.portmap' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python2.7 -c twisted/runner/portmap.c -o build/temp.linux-x86_64-2.7/twisted/runner/portmap.o  gcc -pthread -shared build/temp.linux-x86_64-2.7/twisted/runner/portmap.o -o build/lib.linux-x86_64-2.7/twisted/runner/portmap.so  changing mode of build/scripts-2.7/tkconch from 644 to 755  changing mode of build/scripts-2.7/conch from 644 to 755  changing mode of build/scripts-2.7/pyhtmlizer from 644 to 755  changing mode of build/scripts-2.7/tap2rpm from 644 to 755  changing mode of build/scripts-2.7/manhole from 644 to 755  changing mode of build/scripts-2.7/ckeygen from 644 to 755  changing mode of build/scripts-2.7/tap2deb from 644 to 755  changing mode of build/scripts-2.7/mailmail from 644 to 755  changing mode of build/scripts-2.7/trial from 644 to 755  changing mode of build/scripts-2.7/twistd from 644 to 755  changing mode of build/scripts-2.7/cftp from 644 to 755  changing mode of /app/.heroku/python/bin/tkconch to 755  changing mode of /app/.heroku/python/bin/conch to 755  changing mode of /app/.heroku/python/bin/pyhtmlizer to 755  changing mode of /app/.heroku/python/bin/tap2rpm to 755  changing mode of /app/.heroku/python/bin/manhole to 755  changing mode of /app/.heroku/python/bin/ckeygen to 755  changing mode of /app/.heroku/python/bin/tap2deb to 755  changing mode of /app/.heroku/python/bin/mailmail to 755  changing mode of /app/.heroku/python/bin/trial to 755  changing mode of /app/.heroku/python/bin/twistd to 755  changing mode of /app/.heroku/python/bin/cftp to 755    Compiling /tmp/pip-build-cf1_q8/selenium/selenium/test/selenium/webdriver/browser_specific_template.py     Running setup.py install for dumptruck    Running setup.py install for splinter  Running setup.py install for cssselect  Running setup.py install for lxml  Building lxml version 3.4.4.  Building without Cython.  Using build configuration of libxslt 1.1.28  /app/.heroku/python/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'  warnings.warn(msg)  building 'lxml.etree' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-cf1_q8/lxml/src/lxml/includes -I/app/.heroku/python/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w  gcc -pthread -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7/lxml/etree.so  building 'lxml.objectify' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-cf1_q8/lxml/src/lxml/includes -I/app/.heroku/python/include/python2.7 -c src/lxml/lxml.objectify.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -w  gcc -pthread -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7/lxml/objectify.so  Running setup.py develop for scraperwiki  Creating /app/.heroku/python/lib/python2.7/site-packages/scraperwiki.egg-link (link to .)  Adding scraperwiki 0.3.7 to easy-install.pth file  Installed /app/.heroku/src/scraperwiki  Successfully installed Twisted-15.4.0 cffi-1.2.1 characteristic-14.3.0 cryptography-1.0 cssselect-0.9.1 dumptruck-0.1.6 enum34-1.0.4 idna-2.0 ipaddress-1.0.14 lxml-3.4.4 pyOpenSSL-0.15.1 pyasn1-0.1.8 pyasn1-modules-0.0.7 pycparser-2.14 queuelib-1.3.0 requests-2.7.0 scraperwiki scrapy-0.24.0 selenium-2.47.1 service-identity-14.0.0 six-1.9.0 splinter-0.7.3 w3lib-1.12.0 zope.interface-4.1.2  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scrapy 0.24.0 - no active project Unknown command: crawl Use "scrapy" to see available commands

Statistics

Average successful run time: 1 minute

Total run time: about 2 hours

Total cpu time used: 5 minutes

Total disk space used: 47.1 KB

History

  • Manually ran revision e3ae7399 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1be7ae1d and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Manually ran revision 3a86dd76 and failed .
    nothing changed in the database
  • Manually ran revision d161339d and failed .
    nothing changed in the database
    22 pages scraped
  • Manually ran revision d93dcbd0 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

cet_lebanon / scraper.py