This is a scraper that runs on Morph. To get started see the documentation

Contributors tmtmtmtm octopusinvitro

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.3 -----> Installing dependencies using bundler 1.13.7  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/everypolitician/scraped.git  Fetching https://github.com/everypolitician/scraped_page_archive.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/everypolitician/table_unspanner.git  Installing ast 2.3.0  Installing rake 12.0.0  Installing public_suffix 2.0.4  Installing coderay 1.1.0  Installing safe_yaml 1.0.4  Installing field_serializer 0.3.0  Installing git 1.3.0  Installing hashdiff 0.3.1  Installing method_source 0.8.2  Installing mini_portile2 2.1.0  Installing httpclient 2.6.0.1  Installing minitest 5.10.1  Installing vcr 3.0.3  Installing open-uri-cached 0.0.5  Installing powerpack 0.1.1  Installing slop 3.6.0  Installing rainbow 2.1.0  Installing require_all 1.3.3  Installing ruby-progressbar 1.8.1  Installing unicode-display_width 1.1.2  Using bundler 1.13.7  Installing sqlite3 1.3.10 with native extensions  Installing addressable 2.5.0  Installing parser 2.3.3.1  Installing crack 0.4.3  Installing minispec-metadata 2.0.0  Installing nokogiri 1.7.0.1 with native extensions  Installing minitest-around 0.4.0  Installing pry 0.10.1  Installing rubocop 0.46.0  Installing webmock 2.0.3  Installing minitest-vcr 1.4.0  Installing sqlite_magic 0.0.3  Installing vcr-archive 0.3.0  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Using scraped_page_archive 0.5.0 from https://github.com/everypolitician/scraped_page_archive.git (at master@28f93d7)  Using scraped 0.2.0 from https://github.com/everypolitician/scraped.git (at master@c3fc06e)  Using table_unspanner 0.1.0 from https://github.com/everypolitician/table_unspanner.git (at master@a70a98a)  Bundle complete! 14 Gemfile dependencies, 38 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from webmock:  WebMock 2.0 has some breaking changes. Please check the CHANGELOG: https://goo.gl/piDGLu  Bundle completed (29.69s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/net/http.rb:882:in `rescue in block in connect': Failed to open TCP connection to wolesi.website:80 (getaddrinfo: Name or service not known) (SocketError) from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/net/http.rb:879:in `block in connect' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/timeout.rb:101:in `timeout' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/net/http.rb:878:in `connect' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/net/http.rb:863:in `do_start' from /app/vendor/bundle/ruby/2.3.0/gems/webmock-2.0.3/lib/webmock/http_lib_adapters/net_http.rb:136:in `start_with_connect_without_finish' from /app/vendor/bundle/ruby/2.3.0/gems/webmock-2.0.3/lib/webmock/http_lib_adapters/net_http.rb:104:in `request' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:325:in `block in open_http' from /app/vendor/bundle/ruby/2.3.0/gems/webmock-2.0.3/lib/webmock/http_lib_adapters/net_http.rb:123:in `start_without_connect' from /app/vendor/bundle/ruby/2.3.0/gems/webmock-2.0.3/lib/webmock/http_lib_adapters/net_http.rb:150:in `start' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:319:in `open_http' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:737:in `buffer_open' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:212:in `block in open_loop' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:210:in `catch' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:210:in `open_loop' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:151:in `open_uri' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped_page_archive-28f93d74b1c1/lib/scraped_page_archive/open-uri.rb:9:in `block in open_uri' from /app/vendor/bundle/ruby/2.3.0/gems/vcr-3.0.3/lib/vcr/util/variable_args_block_caller.rb:9:in `call_block' from /app/vendor/bundle/ruby/2.3.0/gems/vcr-3.0.3/lib/vcr.rb:189:in `use_cassette' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped_page_archive-28f93d74b1c1/lib/scraped_page_archive.rb:36:in `record' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped_page_archive-28f93d74b1c1/lib/scraped_page_archive/open-uri.rb:9:in `open_uri' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:717:in `open' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:35:in `open' from scraper.rb:14:in `noko_for' from scraper.rb:18:in `scrape_list' from scraper.rb:41:in `<main>'

Statistics

Average successful run time: about 3 hours

Total run time: 19 days

Total cpu time used: 8 days

Total disk space used: 184 MB

History

  • Auto ran revision 01f0d739 and failed .
    nothing changed in the database
  • Auto ran revision 01f0d739 and failed .
    258 records removed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    33 records added in the database
  • Auto ran revision 01f0d739 and failed .
    225 records added in the database
  • Auto ran revision 01f0d739 and failed .
    258 records removed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Manually ran revision 01f0d739 and completed successfully .
    51 records added in the database
  • Auto ran revision 01f0d739 and failed .
    51 records removed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    258 records added in the database
  • Auto ran revision 01f0d739 and failed .
    258 records removed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    1 record updated in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    134 records added in the database
  • Auto ran revision 01f0d739 and failed .
    134 records removed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    55 records added in the database
  • Auto ran revision 01f0d739 and failed .
    55 records removed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 01f0d739 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    28 records added in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    28 records removed in the database
    1106 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    nothing changed in the database
    79 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    100 records added in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    100 records removed in the database
    762 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    105 records added in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    105 records removed in the database
    739 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    nothing changed in the database
    39 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    27 records added in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    27 records removed in the database
    2357 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    34 records added in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    34 records removed in the database
    1079 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    2492 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    14900 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    175 records added in the database
    1860 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    175 records removed in the database
    403 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1253 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and failed .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and failed .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and failed .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and failed .
    nothing changed in the database
  • Auto ran revision 2b238ba9 and completed successfully .
    46 records added in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    131 records added in the database
    1023 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    177 records removed in the database
    395 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    nothing changed in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and completed successfully .
    248 records added in the database
    1242 pages scraped
  • Auto ran revision 2b238ba9 and failed .
    248 records removed in the database
    50 pages scraped
  • Manually ran revision 2b238ba9 and completed successfully .
    30 records added, 228 records updated in the database
    1242 pages scraped
  • Manually ran revision 0bccb197 and completed successfully .
    228 records added in the database
    46 pages scraped
  • Created on morph.io

Scraper code

Ruby

afghanistan-assembly / scraper.rb