td_iadb

Contributors ErinClark

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 1))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to morph_defaults) to /app/.heroku/src/scraperwiki  Collecting lxml==3.4.4 (from -r /tmp/build/requirements.txt (line 3))  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.  SNIMissingWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.  InsecurePlatformWarning  Downloading lxml-3.4.4.tar.gz (3.5MB)  Collecting cssselect==0.9.1 (from -r /tmp/build/requirements.txt (line 4))  Downloading cssselect-0.9.1.tar.gz  Collecting beautifulsoup4 (from -r /tmp/build/requirements.txt (line 5))  Downloading beautifulsoup4-4.6.0-py2-none-any.whl (86kB)  Collecting python-dateutil (from -r /tmp/build/requirements.txt (line 6))  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)  Collecting selenium (from -r /tmp/build/requirements.txt (line 7))  Downloading selenium-3.9.0-py2.py3-none-any.whl (942kB)  Collecting splinter>=0.7.3 (from -r /tmp/build/requirements.txt (line 8))  Downloading splinter-0.7.7-py2-none-any.whl  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)  Collecting six>=1.5 (from python-dateutil->-r /tmp/build/requirements.txt (line 6))  Downloading six-1.11.0-py2.py3-none-any.whl  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)  Collecting urllib3<1.23,>=1.21.1 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)  Collecting idna<2.7,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading idna-2.6-py2.py3-none-any.whl (56kB)  Installing collected packages: dumptruck, chardet, certifi, urllib3, idna, requests, scraperwiki, lxml, cssselect, beautifulsoup4, six, python-dateutil, selenium, splinter  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for lxml: started  Running setup.py install for lxml: still running...  Running setup.py install for lxml: finished with status 'done'  Running setup.py install for cssselect: started  Running setup.py install for cssselect: finished with status 'done'  Successfully installed beautifulsoup4-4.6.0 certifi-2018.1.18 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.6 lxml-3.4.4 python-dateutil-2.6.1 requests-2.18.4 scraperwiki selenium-3.9.0 six-1.11.0 splinter-0.7.7 urllib3-1.22   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 98, in <module> tenders = get_tenders(portal) File "scraper.py", line 25, in get_tenders tenders_table = soup.find('div', {"class":"container_projects"})\ AttributeError: 'NoneType' object has no attribute 'find'

Data

Downloaded 5 times by MikeRalphson ErinClark

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (561 KB) Use the API

rows 10 / 511

type approval_date project_name due_date reporting_date idb_financing operation_numbers tender_url title e_and_s_impact disbursed_to_date project_status project_subtype project_number reporting_currency publication_date income_collected date undisbursed_amount notice_id financing_type total_cost contract_sig_date amount_outstanding project_type subsector country fund tender_doc sector
SPECIFIC
DEC 17, 2012
Technical Cooperation for the CTF-IADB ECOCASA Program ME-L1121
26/Feb/2016
JAN 31, 2016
ATN/TC-13620-ME
Consulting Services
EXD
Implementation
ME-T1201
USD - United States Dollar
Feb 23, 2016
2016-02-25 13:35:51.347160
40133237
Non-Reimbursable Technical Cooperation
DEC 17, 2012
Technical Cooperation
URBAN DEVELOPMENT AND HOUSING-HOUSING
MEXICO
CTF
URBAN DEVELOPMENT AND HOUSING
SPECIFIC
DEC 11, 2008
Second Phase Health Sector Reform Program - PARSalud II
29/Feb/2016
JAN 31, 2016
2092/OC-PE
Esquema Cajamarquilla Nievería y Cerro Camote - Ampliación de los sistemas de
SP
Implementation
PE-L1005
USD - United States Dollar
Dec 30, 2015
2016-02-28 10:13:20.402028
40045804
Single Currency Facility
NOV 16, 2009
Loan Operation
HEALTH-HEALTH SERVICES
Peru
Ordinary Capital
HEALTH
SPECIFIC
Drinking Water and Sanitation Sector Reform
29/Feb/2016
JAN 31, 2016
['1010/SF-HA', '2381/GR-HA']
Fourniture et Livraison de pièces de branchements en trois (3) lots
Y
Completed
HA0014
USD - United States Dollar
Jan 22, 2016
2016-02-28 10:13:20.402028
40084365
Loan
OCT 5, 1998
Loan Operation
WATER AND SANITATION
Haiti
Fund for Special Operations
WATER AND SANITATION
SPECIFIC
JAN 23, 2014
Knowledge Management on Solar in Chile
04/Mar/2016
JAN 31, 2016
ATN/TC-14296-CH
Consulting Services
CLS
Implementation
CH-T1141
USD - United States Dollar
Feb 24, 2016
2016-03-03 07:15:56.442864
40136322
Non-Reimbursable Technical Cooperation
JUL 22, 2014
Technical Cooperation
ENERGY-ENERGY INSTITUTIONAL STRENGTHENING AND CAPACITY BUILDING
CHILE
CTF
ENERGY
SPECIFIC
Sustainable Development for Chiriqui Region. Phase I
04/Mar/2016
JAN 31, 2016
['1768/OC-PN', '2025/OC-PN-1']
DEADLINE UPDATE - EQUIPAMIENTO DE TRES (3) CAMIONES COMPACTADORES PARA LA RECO
T
Completed
PN-L1005
USD - United States Dollar
Feb 23, 2016
2016-03-03 07:15:56.442864
40133243
Single Currency Facility
SEP 9, 2006
Loan Operation
SOCIAL INVESTMENT-HUMAN RESOURCES & WORKFORCE DEVELOPMENT
PANAMA
Ordinary Capital
SOCIAL INVESTMENT
SPECIFIC
OCT 15, 2015
Encouraging the Deployment of Sustainable Energy in the Caribbean
09/Mar/2016
JAN 31, 2016
ATN/OC-15218-RG
REQUEST FOR EXPRESSIONS OF INTEREST CONSULTING SERVICES
A
Implementation
RG-T2260
USD - United States Dollar
Feb 29, 2016
2016-03-07 23:44:42.145371
40142587
Non-Reimbursable Technical Cooperation
OCT 15, 2015
Technical Cooperation
ENERGY-ENERGY EFFICIENCY AND RENEWABLE ENERGY IN END USE
REGIONAL
Ordinary Capital
ENERGY
SPECIFIC
NOV 19, 2014
Strengthen Social Intersectoral Coordination Program
14/Mar/2016
JAN 31, 2016
3341/OC-EC
Adquisición e Instalación de Mobiliario para la Plataforma Gubernamental de Ge
PAG
Implementation
EC-L1129
USD - United States Dollar
Feb 01, 2016
2016-03-13 22:23:22.023930
40098335
FFF
FEB 5, 2015
Loan Operation
SOCIAL INVESTMENT-POVERTY ALLEVIATION
ECUADOR
Ordinary Capital
SOCIAL INVESTMENT
SPECIFIC
APR 1, 2014
Greenhouse Gas Inventory for Exporting Companies in Central America
15/Mar/2016
JAN 31, 2016
ATN/OC-14350-RG
Consluting Services - Adecuación de Factores de Emisión/Remoción de Gases de E
CNI
Implementation
RG-T2420
USD - United States Dollar
Mar 02, 2016
2016-03-14 08:38:04.102269
40148873
RPG
JUL 15, 2014
Technical Cooperation
TRADE-TRADE FACILITATION, TRADE LOGISTICS AND CUSTOMS
REGIONAL
Ordinary Capital
TRADE
SPECIFIC
JUL 3, 2014
Programs of Opportunities and Rights in the state of Rio Grande do Sul
17/Mar/2016
FEB 29, 2016
3241/OC-BR
Implantação de 05 Centro(s) da Juventude (CJ), em bairros dos Municipios de Po
B
Implementation
BR-L1343
USD - United States Dollar
Mar 15, 2016
2016-03-16 10:51:39.354181
40182333
FFF
SEP 1, 2014
Loan Operation
SOCIAL INVESTMENT-CITIZEN SAFETY
BRAZIL
Ordinary Capital
SOCIAL INVESTMENT
SPECIFIC
Support for the Implementation of the Extreme poverty reduction strategy
18/Mar/2016
FEB 29, 2016
['ATN/OC-14965-CR', 'ATN/OC-14993-CR']
Determinar la pertinencia, eficiencia y eficacia de los procesos de elegibilid
Implementation
CR-T1126
USD - United States Dollar
Mar 08, 2016
2016-03-17 21:16:08.161568
40168340
Non-Reimbursable Technical Cooperation
MAY 21, 2015
Technical Cooperation
SOCIAL INVESTMENT-POVERTY ALLEVIATION
COSTA RICA
Ordinary Capital
SOCIAL INVESTMENT

Statistics

Average successful run time: 9 minutes

Total run time: 4 days

Total cpu time used: about 4 hours

Total disk space used: 588 KB

History

  • Auto ran revision 6f27e02c and failed .
    nothing changed in the database
  • Auto ran revision 6f27e02c and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 6f27e02c and failed .
    nothing changed in the database
    33 pages scraped
  • Auto ran revision 6f27e02c and failed .
    nothing changed in the database
  • Auto ran revision 6f27e02c and failed .
    nothing changed in the database
    1 page scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

td_iadb / scraper.py