duncanparkes / papuanewguinea

Morph scraper for Papua New Guinea politicians


Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:387: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  SNIMissingWarning,  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:142: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Collecting lxml==3.4.4  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:142: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Downloading lxml-3.4.4.tar.gz (3.5 MB)  Collecting cssselect==0.9.1  Downloading cssselect-0.9.1.tar.gz (32 kB)  Collecting awesome-slugify==1.6.4  Downloading awesome-slugify-1.6.4.tar.gz (8.4 kB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting requests  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)  Collecting regex  Downloading regex-2020.7.14.tar.gz (690 kB)  Collecting Unidecode<0.05,>=0.04.14  Downloading Unidecode-0.04.21-py2.py3-none-any.whl (228 kB)  Collecting idna<3,>=2.5  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)  Collecting chardet<4,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)  Building wheels for collected packages: lxml, cssselect, awesome-slugify, dumptruck, regex  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27m-linux_x86_64.whl size=2989842 sha256=15b16753137dbbf3d486294eb84f27eb5149c55d33ee286e5f8c6ffdcce9d0ec  Stored in directory: /tmp/pip-ephem-wheel-cache-NERtEx/wheels/d6/de/81/11ae6edd05c75aac677e67dd154c85da758ba6f3e8e80e962e  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-py2-none-any.whl size=26993 sha256=1c03fa40ac2f9d3a09537b7410cde2afff1d3ba255320de84cfa5e6a9fbc1d1f  Stored in directory: /tmp/pip-ephem-wheel-cache-NERtEx/wheels/85/fe/00/b94036d8583cec9791d8cda24c184f2d2ac1397822f7f0e8d4  Building wheel for awesome-slugify (setup.py): started  Building wheel for awesome-slugify (setup.py): finished with status 'done'  Created wheel for awesome-slugify: filename=awesome_slugify-1.6.4-py2-none-any.whl size=8335 sha256=0370eeaf9d82875ce829e6f86577d890fdb3a760449db152f7be92eeea1fb421  Stored in directory: /tmp/pip-ephem-wheel-cache-NERtEx/wheels/cd/07/d3/77bf73ea75d022f81c5322df341df4cba0d92a09e268353254  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11842 sha256=bec1736c3579c4bd3d4d0970969ec71f20629823e73615128a8ee0a388f9bed0  Stored in directory: /tmp/pip-ephem-wheel-cache-NERtEx/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Building wheel for regex (setup.py): started  Building wheel for regex (setup.py): finished with status 'done'  Created wheel for regex: filename=regex-2020.7.14-cp27-cp27m-linux_x86_64.whl size=603287 sha256=d4a9e24d0026d498f7f1cf3f880f08ddd707a056d0a0355ab223ceee413c440a  Stored in directory: /tmp/pip-ephem-wheel-cache-NERtEx/wheels/c4/ce/1e/c2303bc9574c7641ea1d26060f8634e7964b5b5f6513f68d40  Successfully built lxml cssselect awesome-slugify dumptruck regex  Installing collected packages: dumptruck, idna, chardet, urllib3, certifi, requests, scraperwiki, lxml, cssselect, regex, Unidecode, awesome-slugify  Running setup.py develop for scraperwiki  Successfully installed Unidecode-0.4.21 awesome-slugify-1.6.4 certifi-2020.6.20 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.10 lxml-3.4.4 regex-2020.7.14 requests-2.24.0 scraperwiki urllib3-1.25.10   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 13, in <module> region_lis = root.xpath("//li[contains(., 'Members')]")[0].find('ul').findall('li') IndexError: list index out of range

Data

Downloaded 327 times by everypolitician duncanparkes tmtmtmtm

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 10 / 112

name area province fax email party region party_id phone id image term_id executive district
Hon. Daniel Mona, MP (Deceased)
Goilala District
Central Province
3277480
People's National Congress
Southern Region
PNC
3277626
Daniel-Mona
2012
Goilala District
Hon. Johnson Tuke, MP
Kainantu District
Eastern Highlands Province
3277743
johnsontuke@gmail.com
People's Progress
Highlands Region
PPP
3277671
Johnson-Tuke
2012
Kainantu District
Hon. Benny Allan, MP
Unggai Bena District
Eastern Highlands Province
3277480
unggaibena2012@gmail.com
People's National Congress
Highlands Region
PNC
3277553
Benny-Allan
2012
Unggai Bena District
Hon. Don Polye, MP
Kandep District
Enga Province
Triumph Heritage Empowerment
Highlands Region
THE
3277567
Don-Polye
2012
Kandep District
Hon. John Pundari, MP
Kompiam Ambum District
Enga Province
3277480
People's National Congress
Highlands Region
PNC
3277386
John-Pundari
2012
Kompiam Ambum District
Hon. Nixon Mangape, MP
Lagaip-Porgera District
Enga Province
3277480
Independent
Highlands Region
I
3277702
Nixon-Mangape
2012
Lagaip-Porgera District
Hon. Robert Ganim, MP
Wabag District
Enga Province
3277697
robert.ganiml@gmail.com
People's National Congress
Highlands Region
PNC
3277697/561
Robert-Ganim
2012
Wabag District
Hon. Rimbink Pato, MP
Wapenamanda District
Enga Province
3277480
United
Highlands Region
U
3277502
Rimbink-Pato
2012
Wapenamanda District
Hon. Noah Kool, MP
Chimbu Province
Chimbu Province
3277480
pgamai.npps@gmail.com
People's National Congress
Highlands Region
PNC
3277640
Noah-Kool
2012
Governor
Hon. Wera Mori, MP
Chuave District
Chimbu Province
3277480
meramorill@gmail.com
Triumph Heritage Empowerment
Highlands Region
THE
3277597
Wera-Mori
2012
Chuave District

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 1 / 1

id name start_date end_date
2012
2012-2017
2012
2017

Statistics

Average successful run time: 9 minutes

Total run time: 3 months

Total cpu time used: 16 minutes

Total disk space used: 70.3 KB

History

  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

papuanewguinea / scraper.py