tmtmtmtm / niue-assembly

Scrapes github.com and www.gov.nu

Build software better, together.


This is a scraper that runs on Morph. To get started see the documentation

Contributors tmtmtmtm ondenman

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/..........  Fetching version metadata from https://rubygems.org/..  Fetching https://github.com/everypolitician/scraped_page_archive.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Installing public_suffix 2.0.4  Installing coderay 1.1.0  Installing colorize 0.7.7  Installing safe_yaml 1.0.4  Installing execjs 2.5.2  Installing fuzzy_match 2.1.0  Installing git 1.3.0  Installing hashdiff 0.3.0  Installing httpclient 2.8.2.4  Installing method_source 0.8.2  Installing mini_portile 0.6.2  Installing open-uri-cached 0.0.5  Installing slop 3.6.0  Installing vcr 3.0.3  Installing sqlite3 1.3.12 with native extensions  Using bundler 1.11.2  Installing addressable 2.5.0  Installing crack 0.4.3  Installing nokogiri 1.6.6.2 with native extensions  Installing pry 0.10.1  Installing sqlite_magic 0.0.6  Installing webmock 2.0.3  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing vcr-archive 0.3.0  Using scraped_page_archive 0.5.0 from https://github.com/everypolitician/scraped_page_archive.git (at master@28f93d7)  Bundle complete! 8 Gemfile dependencies, 25 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from webmock:  WebMock 2.0 has some breaking changes. Please check the CHANGELOG: https://goo.gl/piDGLu  Bundle completed (20.75s)  Cleaning up the bundler cache.  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... {:name=>"Dalton Tagelagi", :constituency=>"Alofi South", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Va’aiga Tukuitoga", :constituency=>"Alofi North", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Tofua Puletama", :constituency=>"Makefu", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Fisa Igilisi Pihigia", :constituency=>"Tuapa", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Jack Willie Lipitoa, QSM", :constituency=>"Namukulu", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Opili Talafasi", :constituency=>"Hikutavake", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Dion Paki Taufitu", :constituency=>"Toi", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Bill Vakaafi Motufoou", :constituency=>"Mutalau", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Kupa Magatogia", :constituency=>"Lakepa", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Pokotoa Sipeli", :constituency=>"Liku", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Mititaiagimene Young Vivian", :constituency=>"Hakupu", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Talaititama Talaiti", :constituency=>"Vaiea", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Billy Graham Talagi", :constituency=>"Avatele", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Peter Andrew Funaki", :constituency=>"Tamakautoga", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Stanley Atuvaha Kalauni", :constituency=>"Common Roll", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Toke Tufukia Talagi", :constituency=>"Common Roll", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Crossley Tatui", :constituency=>"Common Roll", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Togia Likalika Sioneholo", :constituency=>"Common Roll", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Terry Donald Coe", :constituency=>"Common Roll", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"} {:name=>"Joan Sisiata Tahafa Viliamu", :constituency=>"Common Roll", :party=>"Independent", :term=>15, :source=>"http://www.gov.nu/wb/pages/parliament/assembly.php"}

Data

Downloaded 258 times by everypolitician MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (6 KB) Use the API

rows 10 / 20

name constituency party term source
Dalton Tagelagi
Alofi South
Independent
15
Va’aiga Tukuitoga
Alofi North
Independent
15
Tofua Puletama
Makefu
Independent
15
Fisa Igilisi Pihigia
Tuapa
Independent
15
Jack Willie Lipitoa, QSM
Namukulu
Independent
15
Opili Talafasi
Hikutavake
Independent
15
Dion Paki Taufitu
Toi
Independent
15
Bill Vakaafi Motufoou
Mutalau
Independent
15
Kupa Magatogia
Lakepa
Independent
15
Pokotoa Sipeli
Liku
Independent
15

Statistics

Average successful run time: less than a minute

Total run time: 10 minutes

Total cpu time used: half a minute

Total disk space used: 159 KB

History

  • Auto ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
    14 pages scraped
  • Auto ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
    14 pages scraped
  • Auto ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
  • Auto ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
    14 pages scraped
  • Auto ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
  • Auto ran revision e5825355 and completed successfully .
    nothing changed in the database
    14 pages scraped
  • Auto ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
    14 pages scraped
  • Auto ran revision e5825355 and completed successfully .
    nothing changed in the database
  • Auto ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
    14 pages scraped
  • Manually ran revision e5825355 and completed successfully .
    20 records added, 20 records removed in the database
    14 pages scraped
  • Manually ran and failed .
  • Manually ran and failed .
  • Auto ran revision 86c5fdb4 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Manually ran revision 86c5fdb4 and completed successfully .
    20 records added, 20 records removed in the database
    1 page scraped
  • Manually ran revision 86c5fdb4 and completed successfully .
    20 records added in the database
    1 page scraped
  • Created on morph.io

Scraper code

Ruby

niue-assembly / scraper.rb