ErinClark / td_afdb

Scrapes www.afdb.org

The African Development Bank is a regional multilateral development bank, engaged in promoting the economic development and social progress of its Regional...


td_afdb

Contributors ErinClark

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 84, in <module> links = get_links(portal) File "scraper.py", line 22, in get_links html = urllib.urlopen(next_url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 87, in urlopen return opener.open(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 359, in open_http return self.http_error(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 372, in http_error result = method(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 665, in http_error_301 return self.http_error_302(url, fp, errcode, errmsg, headers, data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 635, in http_error_302 data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 661, in redirect_internal return self.open(newurl) File "/app/.heroku/python/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 451, in open_https return self.http_error(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 372, in http_error result = method(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 665, in http_error_301 return self.http_error_302(url, fp, errcode, errmsg, headers, data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 635, in http_error_302 data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 661, in redirect_internal return self.open(newurl) File "/app/.heroku/python/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 444, in open_https 'got a bad status line', None) IOError: ('http protocol error', 0, 'got a bad status line', None)

Data

Downloaded 4 times by MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (961 KB) Use the API

rows 10 / 959

title details_download categories publishing_date date tender_url
Annual Procurement Plan 2016
08/03/2016
2016-05-01 20:28:12.296060
AOI - Cameroun - Fourniture d’équipements pour enlèvement des déchets solides - projet d’alimentation en eau potable et d’assainissement en milieu semi-urbain -PAEPA-MSU – 12 2014
Cameroon, Agriculture & Agro-industries, Water Supply & Sanitation
29/12/2014
2016-05-01 20:28:12.296060
IFB - Supply - Installation - Training and Maintenance of Simultaneous Interpretation for the AfDB conference Rooms in the Immeuble Zahrabed Tunis - AfDB/ICB/CGSP/2009/0135
15/01/2014
2016-05-01 20:28:12.296060
IFB - Supply Install and Configure a Storage Area Network (SAN) to Support a Multi-Vendor Dynamic Virtualized Data Centre
15/01/2014
2016-05-01 20:28:12.296060
IFB - Réparation de documents de scanning et numérisation de documents - ADB/RFP/CGSP/2010/0006
15/01/2014
2016-05-01 20:28:12.296060
IFB - The Supply, Installation, Configuration, Commissioning and Maintenance of a Storage Area Network (SAN) - ADB/ICBP/CGSP/2010/0021
15/01/2014
2016-05-01 20:28:12.296060
IFB - Supply, Install and Configure a Storage Area Network (SAN) to Support a Multi-Vendor Dynamic Virtualized Data Center - ADB/CGSP/ICBP/2010/0021
15/01/2014
2016-05-01 20:28:12.296060
IFB - Advertising Services - ADB/CGSP/ICBP/2010/0001
15/01/2014
2016-05-01 20:28:12.296060
IFB - Amendment N°1 - Advertising Services - ADB/CGSP/ICBP/2010/0001
Tunisia
15/01/2014
2016-05-01 20:28:12.296060
IFB - Prequalification Document - Provision of Advertising Services for the AfDB - ADB/CGSP/ICBP/2010/0001
15/01/2014
2016-05-01 20:28:12.296060

Statistics

Average successful run time: about 1 hour

Total run time: 2 months

Total cpu time used: 1 day

Total disk space used: 985 KB

History

  • Auto ran revision 333fbffe and failed .
    nothing changed in the database
    127 pages scraped
  • Auto ran revision 333fbffe and completed successfully .
    896 records added, 896 records removed in the database
    1931 pages scraped
  • Auto ran revision 333fbffe and completed successfully .
    895 records added, 895 records removed in the database
    1931 pages scraped
  • Auto ran revision 333fbffe and completed successfully .
    896 records added, 895 records removed in the database
  • Auto ran revision 333fbffe and completed successfully .
    895 records added, 894 records removed in the database
    1931 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

td_afdb / scraper.py