duncanparkes / namibia

Scrape the members of the Namibian parliament.


This is a scraper that runs on Morph. To get started see the documentation

Contributors duncanparkes

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.9 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Collecting lxml==3.4.4  Downloading lxml-3.4.4.tar.gz (3.5 MB)  Collecting cssselect==0.9.1  Downloading cssselect-0.9.1.tar.gz (32 kB)  Collecting PyExecJS==1.1.0  Downloading PyExecJS-1.1.0.tar.gz (11 kB)  Collecting awesome-slugify==1.6.4  Downloading awesome-slugify-1.6.4.tar.gz (8.4 kB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting requests  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)  Collecting regex  Downloading regex-2020.7.14.tar.gz (690 kB)  Collecting Unidecode<0.05,>=0.04.14  Downloading Unidecode-0.04.21-py2.py3-none-any.whl (228 kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)  Collecting chardet<4,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting idna<3,>=2.5  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)  Building wheels for collected packages: lxml, cssselect, PyExecJS, awesome-slugify, dumptruck, regex  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27m-linux_x86_64.whl size=2989863 sha256=98a7ea75ffc3abf5d1ad0d929878fdf3ee13ff75388b6023486c215c4f8a3631  Stored in directory: /tmp/pip-ephem-wheel-cache-eCvcsq/wheels/d6/de/81/11ae6edd05c75aac677e67dd154c85da758ba6f3e8e80e962e  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-py2-none-any.whl size=26993 sha256=5a71bfc260df5171e24e7c4569f7181f81ebb7fb7e192ec08cbc4721993cf83d  Stored in directory: /tmp/pip-ephem-wheel-cache-eCvcsq/wheels/85/fe/00/b94036d8583cec9791d8cda24c184f2d2ac1397822f7f0e8d4  Building wheel for PyExecJS (setup.py): started  Building wheel for PyExecJS (setup.py): finished with status 'done'  Created wheel for PyExecJS: filename=PyExecJS-1.1.0-py2-none-any.whl size=9432 sha256=0ea476b2a0791fe6b3ba5f97a47c2d69ad5227c129f86ef532b4f5edcdf22657  Stored in directory: /tmp/pip-ephem-wheel-cache-eCvcsq/wheels/d3/5d/f1/9b4c3099891293ad4da409d1a49f60a89fce12605c54a8019a  Building wheel for awesome-slugify (setup.py): started  Building wheel for awesome-slugify (setup.py): finished with status 'done'  Created wheel for awesome-slugify: filename=awesome_slugify-1.6.4-py2-none-any.whl size=8335 sha256=953cb72eda446792e05bc95c66e20e2e9f789a457753ddac7f9bcce794ac9ece  Stored in directory: /tmp/pip-ephem-wheel-cache-eCvcsq/wheels/cd/07/d3/77bf73ea75d022f81c5322df341df4cba0d92a09e268353254  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11842 sha256=b362cbc8dc947461b20a747dfb1a4b01dac36ee153f2763dd32fce731e049620  Stored in directory: /tmp/pip-ephem-wheel-cache-eCvcsq/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Building wheel for regex (setup.py): started  Building wheel for regex (setup.py): finished with status 'done'  Created wheel for regex: filename=regex-2020.7.14-cp27-cp27m-linux_x86_64.whl size=603229 sha256=b346483e65f71f9df194cfd1fe281a31796ed242594777713549faad3801dfb4  Stored in directory: /tmp/pip-ephem-wheel-cache-eCvcsq/wheels/c4/ce/1e/c2303bc9574c7641ea1d26060f8634e7964b5b5f6513f68d40  Successfully built lxml cssselect PyExecJS awesome-slugify dumptruck regex  Installing collected packages: dumptruck, urllib3, certifi, chardet, idna, requests, scraperwiki, lxml, cssselect, PyExecJS, regex, Unidecode, awesome-slugify  Running setup.py develop for scraperwiki  Successfully installed PyExecJS-1.1.0 Unidecode-0.4.21 awesome-slugify-1.6.4 certifi-2020.6.20 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.10 lxml-3.4.4 regex-2020.7.14 requests-2.24.0 scraperwiki urllib3-1.25.10 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... http://www.parliament.na/index.php/5th-national-council-2015-2020 Traceback (most recent call last): File "scraper.py", line 135, in <module> handle_chamber(chamber, source_url, data, term_data) File "scraper.py", line 96, in handle_chamber member['email'] = unjs_email(script) File "scraper.py", line 23, in unjs_email jslines = [x.strip() for x in re.search(r'<!--(.*)//-->', script, re.M | re.S).group(1).strip().splitlines() if not x.strip().startswith('document')] AttributeError: 'NoneType' object has no attribute 'group'

Data

Downloaded 1452 times by everypolitician duncanparkes tmtmtmtm MikeRalphson Created4 stodart

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (441 KB) Use the API

rows 10 / 12

term_number name start_date id end_date
5
5th National Council: 2015 - 2020
2015
5th National Council: 2015 - 2020
2020
4
4th National Council: 2010 - 2015
2010
4th National Council: 2010 - 2015
2015
3
3rd National Council: 2004 - 2010
2004
3rd National Council: 2004 - 2010
2010
2
2nd National Council: 1998 - 2004
1998
2nd National Council: 1998 - 2004
2004
1
1st National Council: 1993 - 1998
1993
1st National Council: 1993 - 1998
1998
6
6th National Assembly: 2015 - 2020
2015
6th National Assembly: 2015 - 2020
2020
5
5th National Assembly: 2010 - 2015
2010
5th National Assembly: 2010 - 2015
2015
4
4th National Assembly: 2005 - 2010
2005
4th National Assembly: 2005 - 2010
2010
3
3rd National Assembly: 2000 - 2005
2000
3rd National Assembly: 2000 - 2005
2005
2
2nd National Assembly: 1995 - 2000
1995
2nd National Assembly: 1995 - 2000
2000

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (441 KB) Use the API

rows 10 / 749

image party term details_url chamber area name term_id id email note
DTA
6th National Assembly: 2015 - 2020
National Assembly
Van den Heever, Jenniffer Muriel
6th National Assembly: 2015 - 2020
Van-den-Heever-Jenniffer-Muriel
jennifer@dtaofnamibia.org
Appointed by President
5th National Council: 2015 - 2020
National Council
Shangula, Kalumbi
5th National Council: 2015 - 2020
Shangula-Kalumbi
RDP
5th National Assembly: 2010 - 2015
National Assembly
Nehova "Kandy" Shafodino Hinananye
5th National Assembly: 2010 - 2015
Nehova-Kandy-Shafodino-Hinananye
SWAPO
1st National Assembly: 1990 - 1995
National Assembly
Mweshihange Peter
1st National Assembly: 1990 - 1995
Mweshihange-Peter
SWAPO
Constituent Assembly: 1989 - 1990
National Assembly
Meroro David
Constituent Assembly: 1989 - 1990
Meroro-David
SWAPO
1st National Council: 1993 - 1998
National Council
Kaiyamo Hosea Nico
1st National Council: 1993 - 1998
Kaiyamo-Hosea-Nico
SWAPO
6th National Assembly: 2015 - 2020
National Assembly
Shapwa, Lidwina Ndeshimona
6th National Assembly: 2015 - 2020
Shapwa-Lidwina-Ndeshimona
lshapwa@moj.gov.na
DTA
1st National Assembly: 1990 - 1995
National Assembly
Gende Allois
1st National Assembly: 1990 - 1995
Gende-Allois
SWAPO
3rd National Assembly: 2000 - 2005
National Assembly
Amathila Appolus Libertine
3rd National Assembly: 2000 - 2005
Amathila-Appolus-Libertine
SWAPO
4th National Assembly: 2005 - 2010
National Assembly
Geingob Hage
4th National Assembly: 2005 - 2010
Geingob-Hage

Statistics

Average successful run time: about 1 hour

Total run time: 3 months

Total cpu time used: about 9 hours

Total disk space used: 478 KB

History

  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • Auto ran revision 92bb4aa6 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

namibia / scraper.py