ErinClark / td_afdb

Scrapes www.afdb.org

The African Development Bank is a regional multilateral development bank, engaged in promoting the economic development and social progress of its Regional...

README.md

td_afdb

Contributors ErinClark

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 84, in <module> links = get_links(portal) File "scraper.py", line 22, in get_links html = urllib.urlopen(next_url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 87, in urlopen return opener.open(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 359, in open_http return self.http_error(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 372, in http_error result = method(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 665, in http_error_301 return self.http_error_302(url, fp, errcode, errmsg, headers, data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 635, in http_error_302 data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 661, in redirect_internal return self.open(newurl) File "/app/.heroku/python/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 451, in open_https return self.http_error(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 372, in http_error result = method(url, fp, errcode, errmsg, headers) File "/app/.heroku/python/lib/python2.7/urllib.py", line 665, in http_error_301 return self.http_error_302(url, fp, errcode, errmsg, headers, data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 635, in http_error_302 data) File "/app/.heroku/python/lib/python2.7/urllib.py", line 661, in redirect_internal return self.open(newurl) File "/app/.heroku/python/lib/python2.7/urllib.py", line 208, in open return getattr(self, name)(url) File "/app/.heroku/python/lib/python2.7/urllib.py", line 444, in open_https 'got a bad status line', None) IOError: ('http protocol error', 0, 'got a bad status line', None)

Data

Downloaded 4 times by MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (961 KB) Use the API

rows 10 / 959

title details_download categories publishing_date date tender_url
Annual Procurement Plan 2016
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/Annual_Procurement_Plan_2016.xls
08/03/2016
2016-05-01 20:28:12.296060
http://www.afdb.org/en/annual-procurement-plan-2016-86966/
AOI - Cameroun - Fourniture d’équipements pour enlèvement des déchets solides - projet d’alimentation en eau potable et d’assainissement en milieu semi-urbain -PAEPA-MSU – 12 2014
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Project-related-Procurement/AOI_-_Cameroun_-_Fourniture_d%E2%80%99%C3%A9quipements_pour_enl%C3%A8vement_des_d%C3%A9chets_solides_-PAEPA-MSU_%E2%80%93_12_2014.pdf
Cameroon, Agriculture & Agro-industries, Water Supply & Sanitation
29/12/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/aoi-cameroun-fourniture-dequipements-pour-enlevement-des-dechets-solides-projet-dalimentation-en-eau-potable-et-dassainissement-en-milieu-semi-urbain-paepa-msu-12-2014-51020/
IFB - Supply - Installation - Training and Maintenance of Simultaneous Interpretation for the AfDB conference Rooms in the Immeuble Zahrabed Tunis - AfDB/ICB/CGSP/2009/0135
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/RFPCGSP_IFB.pdf
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-supply-installation-training-and-maintenance-of-simultaneous-interpretation-for-the-afdb-conference-rooms-in-the-immeuble-zahrabed-tunis-afdb-icb-cgsp-2009-0135-19146/
IFB - Supply Install and Configure a Storage Area Network (SAN) to Support a Multi-Vendor Dynamic Virtualized Data Centre
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/PrequalCGSP.pdf
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-supply-install-and-configure-a-storage-area-network-san-to-support-a-multi-vendor-dynamic-virtualized-data-centre-19145/
IFB - Réparation de documents de scanning et numérisation de documents - ADB/RFP/CGSP/2010/0006
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/DAO%20sanning%20ROSA.pdf
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-reparation-de-documents-de-scanning-et-numerisation-de-documents-adb-rfp-cgsp-2010-0006-19167/
IFB - The Supply, Installation, Configuration, Commissioning and Maintenance of a Storage Area Network (SAN) - ADB/ICBP/CGSP/2010/0021
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/PREQUALIFICATION%20ADB2.pdf
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-the-supply-installation-configuration-commissioning-and-maintenance-of-a-storage-area-network-san-adb-icbp-cgsp-2010-0021-19210/
IFB - Supply, Install and Configure a Storage Area Network (SAN) to Support a Multi-Vendor Dynamic Virtualized Data Center - ADB/CGSP/ICBP/2010/0021
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/Procurement%20Notice%20ADB%20SAN2.pdf
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-supply-install-and-configure-a-storage-area-network-san-to-support-a-multi-vendor-dynamic-virtualized-data-center-adb-cgsp-icbp-2010-0021-19211/
IFB - Advertising Services - ADB/CGSP/ICBP/2010/0001
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/PRE%20QUALIFICATION%20NOTICE%20_%20ADVERTISING%20SERVICES_ADB%20ICBP%20CGSP%202010%20001_01.pdf
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-advertising-services-adb-cgsp-icbp-2010-0001-19899/
IFB - Amendment N°1 - Advertising Services - ADB/CGSP/ICBP/2010/0001
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/Amendment%20No%201_PROVISION%20OF%20ADVERTISING%20FOR%20THE%20AfDB_ADB%20ICPB%20CGSP%202010%200001.pdf
Tunisia
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-amendment-n-1-advertising-services-adb-cgsp-icbp-2010-0001-19898/
IFB - Prequalification Document - Provision of Advertising Services for the AfDB - ADB/CGSP/ICBP/2010/0001
http://www.afdb.org/fileadmin/uploads/afdb/Documents/Procurement/Corporate-Procurement/ADVERTISING%20SERVICES_%20ADB_CGSP_ICBP_2010_0001_01.pdf
15/01/2014
2016-05-01 20:28:12.296060
http://www.afdb.org/en/ifb-prequalification-document-provision-of-advertising-services-for-the-afdb-adb-cgsp-icbp-2010-0001-19897/

Statistics

Average successful run time: about 1 hour

Total run time: 2 months

Total cpu time used: 1 day

Total disk space used: 985 KB

History

  • Auto ran revision 333fbffe and failed .
    nothing changed in the database
    127 pages scraped
  • Auto ran revision 333fbffe and completed successfully .
    896 records added, 896 records removed in the database
    1931 pages scraped
  • Auto ran revision 333fbffe and completed successfully .
    895 records added, 895 records removed in the database
    1931 pages scraped
  • Auto ran revision 333fbffe and completed successfully .
    896 records added, 895 records removed in the database
  • Auto ran revision 333fbffe and completed successfully .
    895 records added, 894 records removed in the database
    1931 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

td_afdb / scraper.py