morph-test-scrapers / test-ruby

Simple test scraper for Ruby - PLEASE DO NOT DELETE


This is a scraper that runs on Morph. To get started see the documentation

Contributors mlandauer henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby/Rack -----> Using Ruby version: ruby-1.9.3 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.......  Fetching version metadata from https://rubygems.org/...  Fetching dependency metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 1.8.23.2 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Installing Ascii85 1.0.1  Installing aasm 3.0.8  Installing i18n 0.6.0  Installing multi_json 1.1.0  Installing builder 3.0.0  Installing arel 3.0.2  Installing tzinfo 0.3.33  Installing addressable 2.2.7  Installing mime-types 1.17.2  Installing awesome_print 1.0.2  Installing multi_xml 0.5.1  Installing json 1.6.5 with native extensions  Installing nokogiri 1.5.0 with native extensions  Installing uuidtools 2.1.3  Installing choice 0.1.4  Installing chronic 0.6.7  Installing fastercsv 1.5.4  Installing oauth 0.4.5  Installing multipart-post 1.1.5  Installing ruby-ole 1.2.11.2  Installing log4r 1.1.9  Installing posix-spawn 0.3.6 with native extensions  Installing to_regexp 0.1.1  Installing upsert 0.3.4  Installing unf_ext 0.0.4 with native extensions  Installing fast-stemmer 1.0.1 with native extensions  Installing hashie 1.2.0  Installing gdata 1.1.2  Installing hpricot 0.8.4 with native extensions  Installing htmlentities 4.3.1  Installing httpauth 0.1  Installing httpclient 2.2.4  Installing libxml-ruby 2.8.0 with native extensions  Installing net-http-digest_auth 1.1.1  Installing net-http-persistent 2.5.2  Installing ntlm-http 0.1.1  Installing ruby-rc4 0.1.5  Installing shoulda-context 1.0.0  Installing shoulda-matchers 1.0.0  Installing polylines 0.1.0  Installing rack 1.4.1  Installing rubyzip 0.9.4  Installing sqlite3 1.3.9 with native extensions  Installing simple_oauth 0.2.0  Installing stringex 1.4.0  Installing trollop 1.16.2  Installing stackdeck 0.2.0  Installing tmail 1.2.7.1 with native extensions  Using bundler 1.11.2  Installing activesupport 3.2.6  Installing typhoeus 0.3.3 with native extensions  Installing httparty 0.8.3  Installing money 5.0.0  Installing jwt 0.1.4  Installing rfgraph 0.3  Installing webrobots 0.0.12  Installing sanitize 2.0.3  Installing faraday 0.8.1  Installing spreadsheet 0.6.5.7  Installing todonotes 0.1.0  Installing unix_utils 0.0.14  Installing unf 0.0.4  Installing pdf-reader 1.0.0  Installing shoulda 3.0.1  Installing sqlite_magic 0.0.3  Installing activemodel 3.2.6  Installing fixed_width-multibyte 0.2.3  Installing hash_digest 1.0.0  Installing sinew 1.0.3  Installing monster_mash 0.3.0  Installing aws-sdk 1.6.1  Installing fletcher 0.4.1  Installing oauth2 0.5.2  Installing twitter 4.5.0  Installing domain_name 0.5.2  Installing pismo 0.7.2  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing activerecord 3.2.6  Installing activeresource 3.2.6  Installing alchemy_api 0.1.2  Installing google-spreadsheet-ruby 0.1.8  Installing google_drive 0.3.1  Installing mechanize 2.5.1  Installing active_record_inline_schema 0.5.7  Installing highrise 3.0.3  Installing roo 1.10.0  Installing remote_table 2.0.2  Installing errata 1.1.1  Installing data_miner 2.3.4  Bundle complete! 86 Gemfile dependencies, 89 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from httparty:  When you HTTParty, you must party hard!  Post-install message from pdf-reader:  ********************************************  v1.0.0 of PDF::Reader introduced a new page-based API. There are extensive  examples showing how to use it in the README and examples directory.  For detailed documentation, check the rdocs for the PDF::Reader,  PDF::Reader::Page and PDF::Reader::ObjectHash classes.  The old API is marked as deprecated but will continue to work with no  visible warnings for now.  ********************************************  Bundle completed (65.63s)  Cleaning up the bundler cache.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Running scraper Sleeping for 5 seconds 1... 2... 3... 4... 5... Finished

Data

Downloaded 1 time by MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3 KB) Use the API

rows 1 / 1

name occupation time
susan
software developer
2016-11-25T00:58:31

Statistics

Average successful run time: 2 minutes

Total run time: about 6 hours

Total cpu time used: 3 minutes

Total disk space used: 30.4 KB

History

  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran and failed .
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran and failed .
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    4 pages scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    3 pages scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    131 pages scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    64 pages scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added in the database
    1 page scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    1 record added, 1 record removed in the database
    5 pages scraped
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 1cc60ac3 and completed successfully .
    nothing changed in the database
  • Manually ran revision 07c612d1 and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran and failed .
  • Manually ran and failed .
  • Manually ran and failed .
  • Manually ran revision 07c612d1 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 07c612d1 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and failed .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and failed .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
    1 page scraped
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and failed .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and failed .
    1 record added, 1 record removed in the database
  • Manually ran revision 8634c7ab and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision e1d7e751 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision e1d7e751 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision e1d7e751 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision e1d7e751 and failed .
    nothing changed in the database
  • Manually ran revision e1d7e751 and completed successfully .
    1 record added, 1 record removed in the database
  • Manually ran revision e1d7e751 and completed successfully .
    1 record added in the database
  • Created on morph.io

Scraper code

Ruby

test-ruby / scraper.rb