tdgetsnz

Contributors ErinClark

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 1))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  Collecting lxml==3.4.4 (from -r /tmp/build/requirements.txt (line 3))  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:354: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  SNIMissingWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning  Downloading https://files.pythonhosted.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz (3.5MB)  Collecting cssselect==0.9.1 (from -r /tmp/build/requirements.txt (line 4))  Downloading https://files.pythonhosted.org/packages/aa/e5/9ee1460d485b94a6d55732eb7ad5b6c084caf73dd6f9cb0bb7d2a78fafe8/cssselect-0.9.1.tar.gz  Collecting beautifulsoup4 (from -r /tmp/build/requirements.txt (line 5))  Downloading https://files.pythonhosted.org/packages/8b/0e/048a2f88bc4be5e3697df9dc1f7b9d5c9c75be62676feeeb91d2e896c5ea/beautifulsoup4-4.7.1-py2-none-any.whl (94kB)  Collecting python-dateutil (from -r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)  Collecting selenium (from -r /tmp/build/requirements.txt (line 7))  Downloading https://files.pythonhosted.org/packages/80/d6/4294f0b4bce4de0abf13e17190289f9d0613b0a44e5dd6a7f5ca98459853/selenium-3.141.0-py2.py3-none-any.whl (904kB)  Collecting splinter>=0.7.3 (from -r /tmp/build/requirements.txt (line 8))  Downloading https://files.pythonhosted.org/packages/a8/f3/7e8ee6296ea7a6db2deb2355166a372bdc9f2e2de4527b1151c025ca8b62/splinter-0.10.0-py2-none-any.whl  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)  Collecting soupsieve>=1.2 (from beautifulsoup4->-r /tmp/build/requirements.txt (line 5))  Downloading https://files.pythonhosted.org/packages/b9/a5/7ea40d0f8676bde6e464a6435a48bc5db09b1a8f4f06d41dd997b8f3c616/soupsieve-1.9.1-py2.py3-none-any.whl  Collecting six>=1.5 (from python-dateutil->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl  Collecting urllib3 (from selenium->-r /tmp/build/requirements.txt (line 7))  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting idna<2.9,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)  Collecting backports.functools-lru-cache; python_version < "3" (from soupsieve>=1.2->beautifulsoup4->-r /tmp/build/requirements.txt (line 5))  Downloading https://files.pythonhosted.org/packages/03/8e/2424c0e65c4a066e28f539364deee49b6451f8fcd4f718fefa50cc3dcf48/backports.functools_lru_cache-1.5-py2.py3-none-any.whl  Building wheels for collected packages: lxml, cssselect, dumptruck  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Stored in directory: /tmp/pip-ephem-wheel-cache-h5oczZ/wheels/f6/df/7b/af9cace9baf95a6e4a2b5790e30da55fc780ddee598314d1ed  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Stored in directory: /tmp/pip-ephem-wheel-cache-h5oczZ/wheels/45/25/d7/5a3b06d22b1ffb616f868a74729a5a002bcc04d45109b4f223  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Stored in directory: /tmp/pip-ephem-wheel-cache-h5oczZ/wheels/57/df/83/32654ae89119876c7a7db66829bbdb646caa151589dbaf226e  Successfully built lxml cssselect dumptruck  Installing collected packages: dumptruck, chardet, idna, urllib3, certifi, requests, scraperwiki, lxml, cssselect, backports.functools-lru-cache, soupsieve, beautifulsoup4, six, python-dateutil, selenium, splinter  Running setup.py develop for scraperwiki  Successfully installed backports.functools-lru-cache-1.5 beautifulsoup4-4.7.1 certifi-2019.3.9 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.8 lxml-3.4.4 python-dateutil-2.8.0 requests-2.22.0 scraperwiki selenium-3.141.0 six-1.12.0 soupsieve-1.9.1 splinter-0.10.0 urllib3-1.25.3   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku. DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 75, in <module> links = get_links(portal, base_url) File "scraper.py", line 24, in get_links table_contents = soup.find('table', id="treetbl").findAllNext('tr') AttributeError: 'NoneType' object has no attribute 'findAllNext'

Data

Downloaded 5 times by MikeRalphson ErinClark

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (21.3 MB) Use the API

rows 10 / 13254

tender_url title rfx_id alternate_contact organisation tender_coverage open_date overview department tender_status fax contact tender_type regions reference categories close_date date required_qualifications
Pesticides in Fresh Frozen Produce - Sampling
14964100
Ministry for Primary Industries
Sole Agency  [?]
Tuesday, 11 August 2015 12:00 PM (Pacific/Auckland UTC+12:00)
The Ministry for Primary Industries (MPI) Food Residue Surveillance Program (FRSP) was established in 2003 to investigate residues and contaminants in food for which there was no existing or suitable verification programme. The Ministry intends to contract for a sample collection service to facilitate the sampling of produce and transportation to testing laboratories.
awarded tenders
Stephen Haran, contract-admin@mpi.govt.nz
Request for Proposals (RFP)
New Zealand
17685
['85150000 - Food and nutrition services']
Friday, 4 September 2015 12:00 PM (Pacific/Auckland UTC+12:00)
2016-02-24 09:53:31.046940
None
Pesticides in Fresh & Frozen Produce - Laboratory Sample Testing
15873623
Ministry for Primary Industries
Sole Agency  [?]
Monday, 31 August 2015 4:00 PM (Pacific/Auckland UTC+12:00)
The Ministry for Primary Industries (MPI) Food Residue Surveillance Programme (FRSP) was established in 2003 to investigate residues and contaminants in food not covered by other major monitoring programmes. The Ministry intends to contract for the provision of laboratory testing services for the 2015/2016 FRSP to evaluate pesticide residues in fresh and frozen produce.
awarded tenders
Sue Escott-Brown, mailbox_contracts@mpi.govt.nz
Request for Proposals (RFP)
New Zealand
17709
['85150000 - Food and nutrition services']
Wednesday, 23 September 2015 4:00 PM (Pacific/Auckland UTC+12:00)
2016-02-24 09:53:31.046940
None
Validating Methods Characterising Manuka Honey
4910975
Ministry for Primary Industries
Sole Agency  [?]
Monday, 29 September 2014 3:00 PM (Pacific/Auckland UTC+13:00)
The objective of this RFP is to identify an appropriately skilled service provider (or providers) to develop and validate techniques that will provide test methods for identifying characteristics associated with honey derived from Leptospermum scoparium (mānuka). These test methods will then be used to determine levels of specific characteristics and allow the establishment of definitions for monofloral and multifloral L. scoparium derived honey.Please Note: Q&A's have now closed and no further questions will be responded to
awarded tenders
Sue Escott-Brown, sue.escott-brown@mpi.govt.nz
Request for Proposals (RFP)
International
17275
['81102600 - Sampling services', '81170000 - Biological science services']
Tuesday, 28 October 2014 2:00 PM (Pacific/Auckland UTC+13:00)
2016-02-24 09:53:31.046940
None
Registration of Interest for Air Traffic Management System
17230433
Airways Corporation New Zealand
Sole Agency  [?]
Monday, 22 February 2016 8:00 AM (Pacific/Auckland UTC+13:00)
Airways Corporation New Zealand would like to advise that a Registration of Interest (ROI) for its Air Traffic Management System will be released Friday 26th February 2016 (New Zealand time) Full details will be provided in the ROI document, and available for download via https://www.gets.govt.nz. Alternatively potential respondents may contact the following to receive a copy of the ROI documentation. Email: Robert.Hughes@airways.co.nz; Tel Direct: +64 3 358 1628 Cell: +64 27 702 8070
current tenders
Rob Hughes, Robert.Hughes@airways.co.nz, 64 3 358 1628
Notice of Information (NOI)
International
['25000000 - Commercial and Military and Private Vehicles and their Accessories and Components']
Friday, 26 February 2016 10:00 PM (Pacific/Auckland UTC+13:00)
2016-02-25 20:39:22.595835
None
Orakei School Classrooms
16894945
Ministry of Education - Regional
Sole Agency  [?]
Friday, 16 October 2015 8:00 AM (Pacific/Auckland UTC+13:00)
No Overview
awarded tenders
Gilbert Wallace, 09 361 5901
Registration of Interest (ROI)
Auckland
['72000000 - Building and Facility Construction and Maintenance Services']
Wednesday, 11 November 2015 10:00 PM (Pacific/Auckland UTC+13:00)
2016-02-27 09:10:59.112791
None
Waterview New School Project_Main Contractor Services
14082385
Level 2 25 Teed Street Newmarket Auckland
Ministry of Education - Regional
Sole Agency  [?]
Friday, 17 July 2015 9:00 AM (Pacific/Auckland UTC+12:00)
Waterview School is a Contributing Primary School, (Yr 1-6) situated in the Auckland suburb of Waterview. An inner city suburb the school is located on 1,925m2 of relatively flat land between Oakley Ave and Herdman St with a roll of approx. 260 pupils. The Ministry of Education, (MOE) have entered into an agreement with New Zealand Transport Association, to rebuild a new school to mitigate the effect of the new Waterview Tunnel Northern Exit, which is under construction on the adjacent eastern boundary. The school is currently operating out of temporary accommodation placed on site in Jan 2013. The new School Project includes a Multi Purpose Hall, Administration building, Library, Staff Room, 12 Classrooms, Existing Hall demolition, Removal of Temporary School, remediation of sport field and all site specific work.
awarded tenders
Bradley M Hill, bradleyh@greenstonegroup.co.nz, 0275247957
Request for Tenders (RFT)
Auckland
14A-448
['72000000 - Building and Facility Construction and Maintenance Services']
Wednesday, 19 August 2015 5:00 PM (Pacific/Auckland UTC+12:00)
2016-02-27 09:10:59.112791
None
Building Contractors - Classrooms Upgrade, Haumoana School, Hawke's Bay
13407396
Ministry of Education - Regional
Sole Agency  [?]
Wednesday, 1 July 2015 9:00 AM (Pacific/Auckland UTC+12:00)
Building Contractors are invited to submit Registrations of Interest to carry out the interior upgrade of 3 classrooms of Block A at Haumoana School, Hawke’s Bay. The project involves replacement floor and wall coverings, minor carpentry work, some joinery upgrade work, interior classroom re-decoration, related services adjustments – and also re-lining of Block A verandah decking. The work is proposed for the end of 2015 school year holidays.
awarded tenders
Tim Judd - Judd Dougan Team Architects Ltd
Registration of Interest (ROI)
Hawke's Bay
1420
['72000000 - Building and Facility Construction and Maintenance Services']
Friday, 17 July 2015 5:00 PM (Pacific/Auckland UTC+12:00)
2016-02-27 09:10:59.112791
None
Point View School - Minor MLE Upgrading in Relocatable Classrooms
14304423
Ministry of Education - Regional
Sole Agency  [?]
Monday, 27 July 2015 9:00 AM (Pacific/Auckland UTC+12:00)
Thirteen relocatable classrooms at Point View School are to be relined with new acoustic wall coverings under the 20111/2012 FY 5YA programme. The existing entry lobbies will be transformed into break-out spaces with new floor coverings, vision panels and relocated coat hooks. Work is to be undertaken in the Third Term break.
awarded tenders
Paul Styles, Styles Project Management Limited
Request for Proposals (RFP)
Auckland
SPM 10383 MLE 2015
['72000000 - Building and Facility Construction and Maintenance Services']
Wednesday, 19 August 2015 5:00 PM (Pacific/Auckland UTC+12:00)
2016-02-27 09:10:59.112791
None
Annual science stakeholder survey
13661535
Ministry of Business, Innovation & Employment
Sole Agency  [?]
Monday, 6 July 2015 8:00 AM (Pacific/Auckland UTC+12:00)
We seek a research contractor to proactively advise on, design and run an annual science stakeholder survey that will contribute to the performance monitoring of Crown research institutes (CRIs) and National Science Challenges (NSCs). The contract will run for three years with a right to renewal. The first survey needs to be in field in March 2016.The Ministry is looking for proactive insights and practical advice on survey design. The provider needs to have a good track record delivering services of this nature and must meet the pre-conditions listed in section 3.2. We are seeking advice on the best methodology for this type of survey, minimising response burden, and ensuring high response rates. This is an opportunity to shape a survey that will contribute towards understanding CRIs’ and NSCs’ contribution to New Zealand.
awarded tenders
Sharon Pells, sharon.pells@mbie.govt.nz
Request for Proposals (RFP)
Auckland , Bay of Plenty , Canterbury , Chatham Islands , Gisborne , Hawke's Bay , Manawatu-Wanganui , Marlborough , Nelson , Northland , Otago , Southland , Taranaki , Tasman , Waikato , Wellington , West Coast
562
['80000000 - Management and Business Professionals and Administrative Services']
Friday, 7 August 2015 12:00 PM (Pacific/Auckland UTC+12:00)
2016-02-27 09:10:59.112791
None
Four Brothers track Upgrade
15899361
There are no hard copies being received for this Tender.
Department of Conservation
Sole Agency  [?]
Tuesday, 1 September 2015 12:00 PM (Pacific/Auckland UTC+12:00)
This contract is for the upgrade of the Four Brothers Track. A 628m length of existing walking track commencing adjacent to the Karamu carpark, which is located on the eastern side of SH23, approximately 22km from both Hamilton and Raglan.• Re-formation of 575m length of 750mm wide aggregate walking track, including cut and fill earthworks.• Formation of 17m length of new 750mm wide aggregate walking track.• Installation of 41m length of aggregate surfaced timber boxed steps.• Installation of timber track edging and retaining wall.• Installation of side drains, grade dips and culverts.• Associated establishment, site clearance, removals and other work necessary to complete walking tracks.
awarded tenders
The Procurement Team, procurement@doc.govt.nz, Please refer all questions through GETS.
Request for Tenders (RFT)
Auckland , Bay of Plenty , Manawatu-Wanganui , Northland , Taranaki , Waikato
['72000000 - Building and Facility Construction and Maintenance Services', '95000000 - Land and Buildings and Structures and Thoroughfares']
Tuesday, 6 October 2015 12:00 PM (Pacific/Auckland UTC+13:00)
2016-02-27 09:10:59.112791
None

Statistics

Average successful run time: about 3 hours

Total run time: 5 months

Total cpu time used: 5 days

Total disk space used: 21.3 MB

History

  • Auto ran revision 31685957 and failed .
    nothing changed in the database
  • Auto ran revision 31685957 and failed .
    nothing changed in the database
  • Auto ran revision 31685957 and failed .
    nothing changed in the database
  • Auto ran revision 31685957 and failed .
    nothing changed in the database
  • Auto ran revision 31685957 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

td_gets_nz / scraper.py