duncanparkes / papuanewguinea

Morph scraper for Papua New Guinea politicians


Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:380: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  SNIMissingWarning,  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:139: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Collecting lxml==3.4.4  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:139: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Downloading lxml-3.4.4.tar.gz (3.5 MB)  Collecting cssselect==0.9.1  Downloading cssselect-0.9.1.tar.gz (32 kB)  Collecting awesome-slugify==1.6.4  Downloading awesome-slugify-1.6.4.tar.gz (8.4 kB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting requests  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)  Collecting regex  Downloading regex-2020.1.8.tar.gz (681 kB)  Collecting Unidecode<0.05,>=0.04.14  Downloading Unidecode-0.04.21-py2.py3-none-any.whl (228 kB)  Collecting chardet<3.1.0,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting idna<2.9,>=2.5  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2019.11.28-py2.py3-none-any.whl (156 kB)  Building wheels for collected packages: lxml, cssselect, awesome-slugify, dumptruck, regex  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27m-linux_x86_64.whl size=2989821 sha256=f3956263f68b2f9998e56f01f65eeaab0e4e0fb315a7e7e3679a6194147e0d92  Stored in directory: /tmp/pip-ephem-wheel-cache-xZ4SHy/wheels/d6/de/81/11ae6edd05c75aac677e67dd154c85da758ba6f3e8e80e962e  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-py2-none-any.whl size=26994 sha256=519e3e01d4f5f635cae814067eb10f0cfa84997c57b397a121074b34a70baaec  Stored in directory: /tmp/pip-ephem-wheel-cache-xZ4SHy/wheels/85/fe/00/b94036d8583cec9791d8cda24c184f2d2ac1397822f7f0e8d4  Building wheel for awesome-slugify (setup.py): started  Building wheel for awesome-slugify (setup.py): finished with status 'done'  Created wheel for awesome-slugify: filename=awesome_slugify-1.6.4-py2-none-any.whl size=8335 sha256=c459d9ab4a006068f441fd66456cb02a5a75b1091b3b9ed0514c7b5d4e5473ec  Stored in directory: /tmp/pip-ephem-wheel-cache-xZ4SHy/wheels/cd/07/d3/77bf73ea75d022f81c5322df341df4cba0d92a09e268353254  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11842 sha256=8ef338795b7e75dd1100334f4d10e3aad33b577b149d59f63bf39aed6b8a65fc  Stored in directory: /tmp/pip-ephem-wheel-cache-xZ4SHy/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Building wheel for regex (setup.py): started  Building wheel for regex (setup.py): finished with status 'done'  Created wheel for regex: filename=regex-2020.1.8-cp27-cp27m-linux_x86_64.whl size=627690 sha256=96d7fd5923a19ea0062ca03f1183656667d82f12f4218fadea5ad957279bb018  Stored in directory: /tmp/pip-ephem-wheel-cache-xZ4SHy/wheels/70/3f/c6/b4ee9f661fc9d09ac578b6f6c4e02341c3a4bd4347d0e53afe  Successfully built lxml cssselect awesome-slugify dumptruck regex  Installing collected packages: dumptruck, chardet, idna, urllib3, certifi, requests, scraperwiki, lxml, cssselect, regex, Unidecode, awesome-slugify  Running setup.py develop for scraperwiki  Successfully installed Unidecode-0.4.21 awesome-slugify-1.6.4 certifi-2019.11.28 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.8 lxml-3.4.4 regex-2020.1.8 requests-2.22.0 scraperwiki urllib3-1.25.8   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 13, in <module> region_lis = root.xpath("//li[contains(., 'Members')]")[0].find('ul').findall('li') IndexError: list index out of range

Data

Downloaded 327 times by everypolitician duncanparkes tmtmtmtm

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 10 / 112

name area province fax email party region party_id phone id image term_id executive district
Hon. Daniel Mona, MP (Deceased)
Goilala District
Central Province
3277480
People's National Congress
Southern Region
PNC
3277626
Daniel-Mona
2012
Goilala District
Hon. Johnson Tuke, MP
Kainantu District
Eastern Highlands Province
3277743
johnsontuke@gmail.com
People's Progress
Highlands Region
PPP
3277671
Johnson-Tuke
2012
Kainantu District
Hon. Benny Allan, MP
Unggai Bena District
Eastern Highlands Province
3277480
unggaibena2012@gmail.com
People's National Congress
Highlands Region
PNC
3277553
Benny-Allan
2012
Unggai Bena District
Hon. Don Polye, MP
Kandep District
Enga Province
Triumph Heritage Empowerment
Highlands Region
THE
3277567
Don-Polye
2012
Kandep District
Hon. John Pundari, MP
Kompiam Ambum District
Enga Province
3277480
People's National Congress
Highlands Region
PNC
3277386
John-Pundari
2012
Kompiam Ambum District
Hon. Nixon Mangape, MP
Lagaip-Porgera District
Enga Province
3277480
Independent
Highlands Region
I
3277702
Nixon-Mangape
2012
Lagaip-Porgera District
Hon. Robert Ganim, MP
Wabag District
Enga Province
3277697
robert.ganiml@gmail.com
People's National Congress
Highlands Region
PNC
3277697/561
Robert-Ganim
2012
Wabag District
Hon. Rimbink Pato, MP
Wapenamanda District
Enga Province
3277480
United
Highlands Region
U
3277502
Rimbink-Pato
2012
Wapenamanda District
Hon. Noah Kool, MP
Chimbu Province
Chimbu Province
3277480
pgamai.npps@gmail.com
People's National Congress
Highlands Region
PNC
3277640
Noah-Kool
2012
Governor
Hon. Wera Mori, MP
Chuave District
Chimbu Province
3277480
meramorill@gmail.com
Triumph Heritage Empowerment
Highlands Region
THE
3277597
Wera-Mori
2012
Chuave District

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 1 / 1

id name start_date end_date
2012
2012-2017
2012
2017

Statistics

Average successful run time: 9 minutes

Total run time: 3 months

Total cpu time used: 15 minutes

Total disk space used: 70.3 KB

History

  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

papuanewguinea / scraper.py