equivalentideas / nsw_contracts_json_archiver

Archive the json api of nsw government contracts

Contributors

Last run failed 2021-04-26 with status code 1.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.2.4 [1G-----> Installing dependencies using bundler 1.15.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Fetching gem metadata from https://rubygems.org/...... [1G Fetching version metadata from https://rubygems.org/. [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Using bundler 1.15.2 [1G Fetching unf_ext 0.0.7.1 [1G Fetching httpclient 2.6.0.1 [1G Fetching mime-types 2.5 [1G Installing httpclient 2.6.0.1 [1G Installing mime-types 2.5 [1G Fetching net-http-digest_auth 1.4 [1G Installing unf_ext 0.0.7.1 with native extensions [1G Installing net-http-digest_auth 1.4 [1G Fetching net-http-persistent 2.9.4 [1G Fetching mini_portile 0.6.2 [1G Installing mini_portile 0.6.2 [1G Installing net-http-persistent 2.9.4 [1G Fetching ntlm-http 0.1.1 [1G Fetching webrobots 0.1.1 [1G Installing ntlm-http 0.1.1 [1G Fetching netrc 0.11.0 [1G Installing netrc 0.11.0 [1G Installing webrobots 0.1.1 [1G Fetching sqlite3 1.3.10 [1G Fetching nokogiri 1.6.6.2 [1G Installing sqlite3 1.3.10 with native extensions [1G Installing nokogiri 1.6.6.2 with native extensions [1G Fetching unf 0.1.4 [1G Installing unf 0.1.4 [1G Fetching domain_name 0.5.24 [1G Installing domain_name 0.5.24 [1G Fetching http-cookie 1.0.2 [1G Installing http-cookie 1.0.2 [1G Fetching rest-client 2.0.2 [1G Installing rest-client 2.0.2 [1G Fetching sqlite_magic 0.0.3 [1G Installing sqlite_magic 0.0.3 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching mechanize 2.7.3 [1G Installing mechanize 2.7.3 [1G Bundle complete! 3 Gemfile dependencies, 19 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into ./vendor/bundle. [1G Bundle completed (20.13s) [1G Cleaning up the bundler cache. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:220:in `rescue in exception_with_response': HTTP status code 520 (RestClient::RequestFailed) from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:217:in `exception_with_response' from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!' from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result' from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit' from /app/vendor/ruby-2.2.4/lib/ruby/2.2.0/net/http.rb:853:in `start' from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit' from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute' from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute' from /app/vendor/bundle/ruby/2.2.0/gems/rest-client-2.0.2/lib/restclient.rb:67:in `get' from scraper.rb:5:in `web_archive' from scraper.rb:12:in `<main>'

Data

Downloaded 6 times by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (151 KB) Use the API

rows 10 / 1109

scraped_date	web_archive_url
2017-09-14	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-16	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-17	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-18	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-19	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-20	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-21	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-22	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-23	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search
2017-09-24	https://web.archive.org/web/https://tenders.nsw.gov.au/?event=public.api.contract.search

Statistics

Average successful run time: 1 minute

Total run time: 1 day

Total cpu time used: 19 minutes

Total disk space used: 174 KB

History

Auto ran revision 75f9d2ac and failed 2021-04-26.

run time 56 s

nothing changed in the database
Auto ran revision 75f9d2ac and failed 2021-04-25.

run time 50 s

nothing changed in the database
Auto ran revision 75f9d2ac and failed 2021-04-24.

run time 52 s

nothing changed in the database
Auto ran revision 75f9d2ac and failed 2021-04-23.

run time 50 s

nothing changed in the database
Auto ran revision 75f9d2ac and failed 2021-04-22.

run time 2 minutes

nothing changed in the database
...
Created on morph.io 2017-09-13

Show complete history

Scraper code

nsw_contracts_json_archiver

git clone URL