everypolitician-scrapers / singapore-current

Scrapes github.com and www.parliament.gov.sg

Build software better, together.


Contributors tmtmtmtm

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.0 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/..........  Fetching version metadata from https://rubygems.org/...  Fetching dependency metadata from https://rubygems.org/..  Fetching https://github.com/everypolitician/scraped_page_archive  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Installing colorize 0.7.7  Installing coderay 1.1.0  Installing public_suffix 2.0.4  Installing safe_yaml 1.0.4  Installing excon 0.45.4  Installing multipart-post 2.0.0  Installing hashie 3.4.2  Installing git 1.3.0  Installing hashdiff 0.3.1  Installing method_source 0.8.2  Installing mini_portile 0.6.2  Installing httpclient 2.6.0.1  Installing slop 3.6.0  Installing open-uri-cached 0.0.5  Installing sqlite3 1.3.10 with native extensions  Using bundler 1.11.2  Installing vcr 3.0.3  Installing addressable 2.5.0  Installing faraday 0.9.1  Installing crack 0.4.3  Installing pry 0.10.1  Installing faraday_middleware 0.10.0  Installing nokogiri 1.6.6.2 with native extensions  Installing webmock 2.0.3  Installing wikidata-client 0.0.7  Installing vcr-archive 0.3.0  Using scraped_page_archive 0.5.0 from https://github.com/everypolitician/scraped_page_archive (at master@28f93d7)  Installing sqlite_magic 0.0.3  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Bundle complete! 7 Gemfile dependencies, 29 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from webmock:  WebMock 2.0 has some breaking changes. Please check the CHANGELOG: https://goo.gl/piDGLu  Bundle completed (18.88s)  Cleaning up the bundler cache.  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Statistics

Average successful run time: about 1 hour

Total run time: 4 days

Total cpu time used: 1 day

Total disk space used: 21.8 MB

History

  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    836 pages scraped
  • Auto ran revision b65023c5 and failed .
    nothing changed in the database
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision b65023c5 and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    1248 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran and failed .
  • Auto ran revision d60dbf50 and failed .
    nothing changed in the database
  • Auto ran revision d60dbf50 and failed .
    nothing changed in the database
    310 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    626 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    623 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and failed .
    nothing changed in the database
    182 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and failed .
    nothing changed in the database
    324 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and failed .
    nothing changed in the database
    431 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    623 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and failed .
    nothing changed in the database
    469 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision d60dbf50 and completed successfully .
    nothing changed in the database
    624 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    419 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    419 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    422 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    418 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
    432 pages scraped
  • Auto ran revision 3c584d2c and completed successfully .
    nothing changed in the database
  • Manually ran revision 3c584d2c and completed successfully .
    nothing changed in the database
  • Manually ran revision ff2dd15f and completed successfully .
    nothing changed in the database
  • Created on morph.io

Scraper code

Ruby

singapore-current / scraper.rb