duncanparkes / papuanewguinea

Morph scraper for Papua New Guinea politicians


Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Collecting lxml==3.4.4 (from -r /tmp/build/requirements.txt (line 8))  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  SNIMissingWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning  Downloading https://files.pythonhosted.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz (3.5MB)  Collecting cssselect==0.9.1 (from -r /tmp/build/requirements.txt (line 9))  Downloading https://files.pythonhosted.org/packages/aa/e5/9ee1460d485b94a6d55732eb7ad5b6c084caf73dd6f9cb0bb7d2a78fafe8/cssselect-0.9.1.tar.gz  Collecting awesome-slugify==1.6.4 (from -r /tmp/build/requirements.txt (line 13))  Downloading https://files.pythonhosted.org/packages/2f/2a/adc6ce1178b938286c9ecc81b99c60dcac3e66c1b3f0fa72255af03a5cf6/awesome-slugify-1.6.4.tar.gz  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)  Collecting regex (from awesome-slugify==1.6.4->-r /tmp/build/requirements.txt (line 13))  Downloading https://files.pythonhosted.org/packages/9f/79/adbe79373ec3c56ae61670bbed30609fe4a9b6af835478fa12bf8795cdec/regex-2018.07.11.tar.gz (628kB)  Collecting Unidecode<0.05,>=0.04.14 (from awesome-slugify==1.6.4->-r /tmp/build/requirements.txt (line 13))  Downloading https://files.pythonhosted.org/packages/01/a1/9d7f3138ee3d79a1ab865a2cb38200ca778d85121db19fe264c76c981184/Unidecode-0.04.21-py2.py3-none-any.whl (228kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting urllib3<1.24,>=1.21.1 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)  Collecting idna<2.8,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)  Installing collected packages: dumptruck, chardet, urllib3, certifi, idna, requests, scraperwiki, lxml, cssselect, regex, Unidecode, awesome-slugify  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for lxml: started  Running setup.py install for lxml: still running...  Running setup.py install for lxml: finished with status 'done'  Running setup.py install for cssselect: started  Running setup.py install for cssselect: finished with status 'done'  Running setup.py install for regex: started  Running setup.py install for regex: finished with status 'done'  Running setup.py install for awesome-slugify: started  Running setup.py install for awesome-slugify: finished with status 'done'  Successfully installed Unidecode-0.4.21 awesome-slugify-1.6.4 certifi-2018.4.16 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.7 lxml-3.4.4 regex-2018.7.11 requests-2.19.1 scraperwiki urllib3-1.23   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 13, in <module> region_lis = root.xpath("//li[contains(., 'Members')]")[0].find('ul').findall('li') IndexError: list index out of range

Data

Downloaded 327 times by everypolitician duncanparkes tmtmtmtm

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 10 / 112

name area province fax email party region party_id phone id image term_id executive district
Hon. Daniel Mona, MP (Deceased)
Goilala District
Central Province
3277480
People's National Congress
Southern Region
PNC
3277626
Daniel-Mona
2012
Goilala District
Hon. Johnson Tuke, MP
Kainantu District
Eastern Highlands Province
3277743
johnsontuke@gmail.com
People's Progress
Highlands Region
PPP
3277671
Johnson-Tuke
2012
Kainantu District
Hon. Benny Allan, MP
Unggai Bena District
Eastern Highlands Province
3277480
unggaibena2012@gmail.com
People's National Congress
Highlands Region
PNC
3277553
Benny-Allan
2012
Unggai Bena District
Hon. Don Polye, MP
Kandep District
Enga Province
Triumph Heritage Empowerment
Highlands Region
THE
3277567
Don-Polye
2012
Kandep District
Hon. John Pundari, MP
Kompiam Ambum District
Enga Province
3277480
People's National Congress
Highlands Region
PNC
3277386
John-Pundari
2012
Kompiam Ambum District
Hon. Nixon Mangape, MP
Lagaip-Porgera District
Enga Province
3277480
Independent
Highlands Region
I
3277702
Nixon-Mangape
2012
Lagaip-Porgera District
Hon. Robert Ganim, MP
Wabag District
Enga Province
3277697
robert.ganiml@gmail.com
People's National Congress
Highlands Region
PNC
3277697/561
Robert-Ganim
2012
Wabag District
Hon. Rimbink Pato, MP
Wapenamanda District
Enga Province
3277480
United
Highlands Region
U
3277502
Rimbink-Pato
2012
Wapenamanda District
Hon. Noah Kool, MP
Chimbu Province
Chimbu Province
3277480
pgamai.npps@gmail.com
People's National Congress
Highlands Region
PNC
3277640
Noah-Kool
2012
Governor
Hon. Wera Mori, MP
Chuave District
Chimbu Province
3277480
meramorill@gmail.com
Triumph Heritage Empowerment
Highlands Region
THE
3277597
Wera-Mori
2012
Chuave District

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 1 / 1

id name start_date end_date
2012
2012-2017
2012
2017

Statistics

Average successful run time: 9 minutes

Total run time: 3 months

Total cpu time used: 11 minutes

Total disk space used: 70.3 KB

History

  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
    1 page scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

papuanewguinea / scraper.py