duncanparkes / papuanewguinea

Morph scraper for Papua New Guinea politicians


Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:380: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  SNIMissingWarning,  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:139: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Collecting lxml==3.4.4  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:139: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Downloading https://files.pythonhosted.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz (3.5MB)  Collecting cssselect==0.9.1  Downloading https://files.pythonhosted.org/packages/aa/e5/9ee1460d485b94a6d55732eb7ad5b6c084caf73dd6f9cb0bb7d2a78fafe8/cssselect-0.9.1.tar.gz  Collecting awesome-slugify==1.6.4  Downloading https://files.pythonhosted.org/packages/2f/2a/adc6ce1178b938286c9ecc81b99c60dcac3e66c1b3f0fa72255af03a5cf6/awesome-slugify-1.6.4.tar.gz  Collecting dumptruck>=0.1.2  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting requests  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)  Collecting regex  Downloading https://files.pythonhosted.org/packages/fc/1d/13cc7d174cd2d05808abac3f5fb37433e30c4cd93b152d2a9c09c926d7e8/regex-2019.11.1.tar.gz (669kB)  Collecting Unidecode<0.05,>=0.04.14  Downloading https://files.pythonhosted.org/packages/01/a1/9d7f3138ee3d79a1ab865a2cb38200ca778d85121db19fe264c76c981184/Unidecode-0.04.21-py2.py3-none-any.whl (228kB)  Collecting chardet<3.1.0,>=3.0.2  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting idna<2.9,>=2.5  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl (125kB)  Collecting certifi>=2017.4.17  Downloading https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl (154kB)  Building wheels for collected packages: lxml, cssselect, awesome-slugify, dumptruck, regex  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27m-linux_x86_64.whl size=2989823 sha256=ecb0a8f32dd983bc5f2f99d50b4ec4ccdbfd9361a42b31458eaf95802b5b380d  Stored in directory: /tmp/pip-ephem-wheel-cache-LlC3pv/wheels/f6/df/7b/af9cace9baf95a6e4a2b5790e30da55fc780ddee598314d1ed  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-cp27-none-any.whl size=26994 sha256=b19b30f3caf6383d8976094b23f5dda9d6097577e9f0e9ade65ec62ac2204815  Stored in directory: /tmp/pip-ephem-wheel-cache-LlC3pv/wheels/45/25/d7/5a3b06d22b1ffb616f868a74729a5a002bcc04d45109b4f223  Building wheel for awesome-slugify (setup.py): started  Building wheel for awesome-slugify (setup.py): finished with status 'done'  Created wheel for awesome-slugify: filename=awesome_slugify-1.6.4-cp27-none-any.whl size=8334 sha256=7f8c350fe867ecb5153066ea5a4b6ebe41ca6c697422c31433e302f4b0fd2911  Stored in directory: /tmp/pip-ephem-wheel-cache-LlC3pv/wheels/71/bf/7f/2a1535926ab6419a15cd28eee5f3f6107c88fd2fe2c99dfa08  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-cp27-none-any.whl size=11845 sha256=a25b51e2aad6a3a0d6edcd193b97ce8765594910cd75df6adada45523d14c825  Stored in directory: /tmp/pip-ephem-wheel-cache-LlC3pv/wheels/57/df/83/32654ae89119876c7a7db66829bbdb646caa151589dbaf226e  Building wheel for regex (setup.py): started  Building wheel for regex (setup.py): finished with status 'done'  Created wheel for regex: filename=regex-2019.11.1-cp27-cp27m-linux_x86_64.whl size=627264 sha256=0443d605fef8e3d06dfec51b1000d4120047889b839212045520b8a68d57ca01  Stored in directory: /tmp/pip-ephem-wheel-cache-LlC3pv/wheels/5c/c6/c1/0bc8d16ea38c44536a82dd1bec665996e5af37489fa88826b6  Successfully built lxml cssselect awesome-slugify dumptruck regex  Installing collected packages: dumptruck, chardet, idna, urllib3, certifi, requests, scraperwiki, lxml, cssselect, regex, Unidecode, awesome-slugify  Running setup.py develop for scraperwiki  Successfully installed Unidecode-0.4.21 awesome-slugify-1.6.4 certifi-2019.9.11 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.8 lxml-3.4.4 regex-2019.11.1 requests-2.22.0 scraperwiki urllib3-1.25.7   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku. DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting configuration and compiling... Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 13, in <module> region_lis = root.xpath("//li[contains(., 'Members')]")[0].find('ul').findall('li') IndexError: list index out of range

Data

Downloaded 327 times by everypolitician duncanparkes tmtmtmtm

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 10 / 112

name area province fax email party region party_id phone id image term_id executive district
Hon. Daniel Mona, MP (Deceased)
Goilala District
Central Province
3277480
People's National Congress
Southern Region
PNC
3277626
Daniel-Mona
2012
Goilala District
Hon. Johnson Tuke, MP
Kainantu District
Eastern Highlands Province
3277743
johnsontuke@gmail.com
People's Progress
Highlands Region
PPP
3277671
Johnson-Tuke
2012
Kainantu District
Hon. Benny Allan, MP
Unggai Bena District
Eastern Highlands Province
3277480
unggaibena2012@gmail.com
People's National Congress
Highlands Region
PNC
3277553
Benny-Allan
2012
Unggai Bena District
Hon. Don Polye, MP
Kandep District
Enga Province
Triumph Heritage Empowerment
Highlands Region
THE
3277567
Don-Polye
2012
Kandep District
Hon. John Pundari, MP
Kompiam Ambum District
Enga Province
3277480
People's National Congress
Highlands Region
PNC
3277386
John-Pundari
2012
Kompiam Ambum District
Hon. Nixon Mangape, MP
Lagaip-Porgera District
Enga Province
3277480
Independent
Highlands Region
I
3277702
Nixon-Mangape
2012
Lagaip-Porgera District
Hon. Robert Ganim, MP
Wabag District
Enga Province
3277697
robert.ganiml@gmail.com
People's National Congress
Highlands Region
PNC
3277697/561
Robert-Ganim
2012
Wabag District
Hon. Rimbink Pato, MP
Wapenamanda District
Enga Province
3277480
United
Highlands Region
U
3277502
Rimbink-Pato
2012
Wapenamanda District
Hon. Noah Kool, MP
Chimbu Province
Chimbu Province
3277480
pgamai.npps@gmail.com
People's National Congress
Highlands Region
PNC
3277640
Noah-Kool
2012
Governor
Hon. Wera Mori, MP
Chuave District
Chimbu Province
3277480
meramorill@gmail.com
Triumph Heritage Empowerment
Highlands Region
THE
3277597
Wera-Mori
2012
Chuave District

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (42 KB) Use the API

rows 1 / 1

id name start_date end_date
2012
2012-2017
2012
2017

Statistics

Average successful run time: 9 minutes

Total run time: 3 months

Total cpu time used: 14 minutes

Total disk space used: 70.3 KB

History

  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • Auto ran revision 6a2caf4e and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

papuanewguinea / scraper.py