duncanparkes / namibia

Scrape the members of the Namibian parliament.


This is a scraper that runs on Morph. To get started see the documentation

Contributors duncanparkes

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.9 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  Collecting lxml==3.4.4  Downloading https://files.pythonhosted.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz (3.5MB)  Collecting cssselect==0.9.1  Downloading https://files.pythonhosted.org/packages/aa/e5/9ee1460d485b94a6d55732eb7ad5b6c084caf73dd6f9cb0bb7d2a78fafe8/cssselect-0.9.1.tar.gz  Collecting PyExecJS==1.1.0  Downloading https://files.pythonhosted.org/packages/0d/6b/5183948f2a12b7df0d597273f941a26baa35520505ec8d670f4c9b568749/PyExecJS-1.1.0.tar.gz  Collecting awesome-slugify==1.6.4  Downloading https://files.pythonhosted.org/packages/2f/2a/adc6ce1178b938286c9ecc81b99c60dcac3e66c1b3f0fa72255af03a5cf6/awesome-slugify-1.6.4.tar.gz  Collecting dumptruck>=0.1.2  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting requests  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)  Collecting regex  Downloading https://files.pythonhosted.org/packages/73/d9/b58289d885180b5d538aa6df07974b5fe6088547ac846c0f76f77259c304/regex-2020.1.8.tar.gz (681kB)  Collecting Unidecode<0.05,>=0.04.14  Downloading https://files.pythonhosted.org/packages/01/a1/9d7f3138ee3d79a1ab865a2cb38200ca778d85121db19fe264c76c981184/Unidecode-0.04.21-py2.py3-none-any.whl (228kB)  Collecting certifi>=2017.4.17  Downloading https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl (156kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl (125kB)  Collecting idna<2.9,>=2.5  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)  Collecting chardet<3.1.0,>=3.0.2  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Building wheels for collected packages: lxml, cssselect, PyExecJS, awesome-slugify, dumptruck, regex  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27m-linux_x86_64.whl size=2989872 sha256=ec992ff653ff2656cb4de489eafd709e5a1ef8ace66250f393b738c1a9c67161  Stored in directory: /tmp/pip-ephem-wheel-cache-21HbGg/wheels/f6/df/7b/af9cace9baf95a6e4a2b5790e30da55fc780ddee598314d1ed  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-cp27-none-any.whl size=26994 sha256=79f2d8d4f2ce893e23483a18d0a98ceb25b773f37c680cedc22d2f767ac0ebd8  Stored in directory: /tmp/pip-ephem-wheel-cache-21HbGg/wheels/45/25/d7/5a3b06d22b1ffb616f868a74729a5a002bcc04d45109b4f223  Building wheel for PyExecJS (setup.py): started  Building wheel for PyExecJS (setup.py): finished with status 'done'  Created wheel for PyExecJS: filename=PyExecJS-1.1.0-cp27-none-any.whl size=9431 sha256=b09e2ce6b114c23196ecad773b2da35147cbbc8d5f2df7847b7bfc90b0d6315d  Stored in directory: /tmp/pip-ephem-wheel-cache-21HbGg/wheels/9e/e4/21/d78fd3ddb7cee84c8d37c9dc4c57837f08b012cd0045cf36f7  Building wheel for awesome-slugify (setup.py): started  Building wheel for awesome-slugify (setup.py): finished with status 'done'  Created wheel for awesome-slugify: filename=awesome_slugify-1.6.4-cp27-none-any.whl size=8334 sha256=deb72609938b99308de3087c78a7c39c20caa444b66b47a32fdce584dd069255  Stored in directory: /tmp/pip-ephem-wheel-cache-21HbGg/wheels/71/bf/7f/2a1535926ab6419a15cd28eee5f3f6107c88fd2fe2c99dfa08  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-cp27-none-any.whl size=11845 sha256=ba70cf5776a25ff4fc364e86a67eac5825a2743cb845e767d39099ffc047e25e  Stored in directory: /tmp/pip-ephem-wheel-cache-21HbGg/wheels/57/df/83/32654ae89119876c7a7db66829bbdb646caa151589dbaf226e  Building wheel for regex (setup.py): started  Building wheel for regex (setup.py): finished with status 'done'  Created wheel for regex: filename=regex-2020.1.8-cp27-cp27m-linux_x86_64.whl size=627674 sha256=1ebb77cd3d4785540b0c57236dccd2272b04e5d9368a14c8da491051cd051267  Stored in directory: /tmp/pip-ephem-wheel-cache-21HbGg/wheels/1c/78/87/21be0303007ee5d1483df56703c9c7e5a44873e8f0c51d65f8  Successfully built lxml cssselect PyExecJS awesome-slugify dumptruck regex  Installing collected packages: dumptruck, certifi, urllib3, idna, chardet, requests, scraperwiki, lxml, cssselect, PyExecJS, regex, Unidecode, awesome-slugify  Running setup.py develop for scraperwiki  Successfully installed PyExecJS-1.1.0 Unidecode-0.4.21 awesome-slugify-1.6.4 certifi-2019.11.28 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.8 lxml-3.4.4 regex-2020.1.8 requests-2.22.0 scraperwiki urllib3-1.25.7 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... http://www.parliament.na/index.php/5th-national-council-2015-2020 Traceback (most recent call last): File "scraper.py", line 135, in <module> handle_chamber(chamber, source_url, data, term_data) File "scraper.py", line 96, in handle_chamber member['email'] = unjs_email(script) File "scraper.py", line 23, in unjs_email jslines = [x.strip() for x in re.search(r'<!--(.*)//-->', script, re.M | re.S).group(1).strip().splitlines() if not x.strip().startswith('document')] AttributeError: 'NoneType' object has no attribute 'group'

Data

Downloaded 1451 times by everypolitician duncanparkes tmtmtmtm MikeRalphson Created4

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (441 KB) Use the API

rows 10 / 12

term_number name start_date id end_date
5
5th National Council: 2015 - 2020
2015
5th National Council: 2015 - 2020
2020
4
4th National Council: 2010 - 2015
2010
4th National Council: 2010 - 2015
2015
3
3rd National Council: 2004 - 2010
2004
3rd National Council: 2004 - 2010
2010
2
2nd National Council: 1998 - 2004
1998
2nd National Council: 1998 - 2004
2004
1
1st National Council: 1993 - 1998
1993
1st National Council: 1993 - 1998
1998
6
6th National Assembly: 2015 - 2020
2015
6th National Assembly: 2015 - 2020
2020
5
5th National Assembly: 2010 - 2015
2010
5th National Assembly: 2010 - 2015
2015
4
4th National Assembly: 2005 - 2010
2005
4th National Assembly: 2005 - 2010
2010
3
3rd National Assembly: 2000 - 2005
2000
3rd National Assembly: 2000 - 2005
2005
2
2nd National Assembly: 1995 - 2000
1995
2nd National Assembly: 1995 - 2000
2000

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (441 KB) Use the API

rows 10 / 749

image party term details_url chamber area name term_id id email note
DTA
6th National Assembly: 2015 - 2020
National Assembly
Van den Heever, Jenniffer Muriel
6th National Assembly: 2015 - 2020
Van-den-Heever-Jenniffer-Muriel
jennifer@dtaofnamibia.org
Appointed by President
5th National Council: 2015 - 2020
National Council
Shangula, Kalumbi
5th National Council: 2015 - 2020
Shangula-Kalumbi
RDP
5th National Assembly: 2010 - 2015
National Assembly
Nehova "Kandy" Shafodino Hinananye
5th National Assembly: 2010 - 2015
Nehova-Kandy-Shafodino-Hinananye
SWAPO
1st National Assembly: 1990 - 1995
National Assembly
Mweshihange Peter
1st National Assembly: 1990 - 1995
Mweshihange-Peter
SWAPO
Constituent Assembly: 1989 - 1990
National Assembly
Meroro David
Constituent Assembly: 1989 - 1990
Meroro-David
SWAPO
1st National Council: 1993 - 1998
National Council
Kaiyamo Hosea Nico
1st National Council: 1993 - 1998
Kaiyamo-Hosea-Nico
SWAPO
6th National Assembly: 2015 - 2020
National Assembly
Shapwa, Lidwina Ndeshimona
6th National Assembly: 2015 - 2020
Shapwa-Lidwina-Ndeshimona
lshapwa@moj.gov.na
DTA
1st National Assembly: 1990 - 1995
National Assembly
Gende Allois
1st National Assembly: 1990 - 1995
Gende-Allois
SWAPO
3rd National Assembly: 2000 - 2005
National Assembly
Amathila Appolus Libertine
3rd National Assembly: 2000 - 2005
Amathila-Appolus-Libertine
SWAPO
4th National Assembly: 2005 - 2010
National Assembly
Geingob Hage
4th National Assembly: 2005 - 2010
Geingob-Hage

Statistics

Average successful run time: about 1 hour

Total run time: 3 months

Total cpu time used: about 9 hours

Total disk space used: 478 KB

History

  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

namibia / scraper.py