alisonkeen / SA_MHRS_for_OA_input

SA MHRs in format for OpenAustralia parser to read

Scrapes www2.parliament.sa.gov.au

Resources Default Page


This scraper generates just a list of people who have sat in either house of the South Australian Parliament (scraper #1 in the list below).

scraper #1: List of MPS - https://morph.io/alisonkeen/SA_MHRS_for_OA_input fields {id, saparl ID, full name} destination: /var/www/openaustralia/openaustralia-parser/data/people.csv

scraper #2: List of MHRs - https://morph.io/alisonkeen/SA_members_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/representatives.csv

scraper #3: List of Senators - https://morph.io/alisonkeen/SA_senators_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/senators.csv

Ruby script to read these three .csv files into OpenAustralia: see https://github.com/alisonkeen/openaustralia-parser-sandbox/import-data-from-morph.rb

This is a scraper that runs on Morph. To get started see the documentation

Contributors alisonkeen

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Installing unf_ext 0.0.7.1 with native extensions  Installing httpclient 2.6.0.1  Installing mime-types 2.5  Installing net-http-digest_auth 1.4  Installing net-http-persistent 2.9.4  Installing mini_portile 0.6.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.1  Installing sqlite3 1.3.10 with native extensions  Using bundler 1.11.2  Installing unf 0.1.4  Installing nokogiri 1.6.6.2 with native extensions  Installing sqlite_magic 0.0.3  Installing domain_name 0.5.24  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing http-cookie 1.0.2  Installing mechanize 2.7.3  Bundle complete! 2 Gemfile dependencies, 17 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (31.51s)  Cleaning up the bundler cache.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... 531, 531, Hon Michael Atkinson 543, 543, Ms Frances Bedford 4844, 4844, Mr Troy Bell 4623, 4623, Hon Zoe Bettison 3117, 3117, Hon Leon Bignell 3533, 3533, Hon Geoff Brock 3489, 3489, Hon Robert Brokenshire 1802, 1802, Hon Paul Caica 1804, 1804, Ms Vickie Chapman 4622, 4622, Hon Susan Close 4978, 4978, Ms Nat Cook 3404, 3404, Hon John Darley 599, 599, Hon John Dawkins 4838, 4838, Mrs Annabel Digance 4991, 4991, Mr Sam Duluk 4363, 4363, Hon Tammy Franks 1821, 1821, Hon Gail Gago 4343, 4343, Mr John Gardner 1822, 1822, Hon John Gazzola 4845, 4845, Mr Jon Gee 1805, 1805, Mr Mark Goldsworthy 3118, 3118, Mr Steven Griffiths 631, 631, Hon Martin Hamilton-Smith 4846, 4846, Ms Katrine Hildyard 3126, 3126, Hon Dennis Hood 4839, 4839, Mr Eddie Hughes 3122, 3122, Hon Ian Hunter 4564, 4564, Hon Gerry Kandelaars 3119, 3119, Hon Tom Kenyon 539, 539, Hon Steph Key 4847, 4847, Mr Stephan Knoll 633, 633, Hon Tom Koutsantonis 4362, 4362, Hon Jing Lee 2742, 2742, Hon Michelle Lensink 605, 605, Hon Rob Lucas 4697, 4697, Hon Kyam Maher 5084, 5084, Hon Peter Malinauskas 4338, 4338, Mr Steven Marshall 1807, 1807, Dr Duncan McFetridge 4866, 4866, Hon Andrew McLachlan CSC 4842, 4842, Hon Stephen Mullighan 4867, 4867, Hon Tung Ngo 4335, 4335, Mr Lee Odenwalder 3130, 3130, Hon Mark Parnell 3120, 3120, Mr Adrian Pederick 3121, 3121, Mr Michael Pengilly 3123, 3123, Hon Tony Piccolo 4841, 4841, Mr Chris Picton 3124, 3124, Mr David Pisoni 614, 614, Hon Jennifer Rankine 1810, 1810, Hon John Rau 1813, 1813, Ms Isobel Redmond 1820, 1820, Hon David Ridgway 4340, 4340, Ms Rachel Sanderson 627, 627, Hon Jack Snelling 4837, 4837, Mr David Speirs 1819, 1819, Hon Terry Stephens 4840, 4840, Mr Vincent Tarzia 4341, 4341, Mr Peter Treloar 4342, 4342, Mr Dan van Holst Pellekaan 4364, 4364, Hon Kelly Vincent 4336, 4336, Hon Leesa Vlahos 3164, 3164, Hon Stephen Wade 1812, 1812, Hon Jay Weatherill 4339, 4339, Mr Tim Whetstone 546, 546, Mr Mitch Williams 4843, 4843, Mr Corey Wingard 4848, 4848, Ms Dana Wortley 3125, 3125, Hon Russell Wortley

Data

Downloaded 4 times by alisonkeen

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (7 KB) Use the API

rows 10 / 69

id id__saparl full_name birthday
531
531
Hon Michael Atkinson
543
543
Ms Frances Bedford
4844
4844
Mr Troy Bell
4623
4623
Hon Zoe Bettison
3117
3117
Hon Leon Bignell
3533
3533
Hon Geoff Brock
3489
3489
Hon Robert Brokenshire
1802
1802
Hon Paul Caica
1804
1804
Ms Vickie Chapman
4622
4622
Hon Susan Close

Statistics

Average successful run time: half a minute

Total run time: 4 minutes

Total cpu time used: less than 5 seconds

Total disk space used: 51.2 KB

History

  • Manually ran revision d26b2999 and completed successfully .
    69 records added, 69 records removed in the database
    1 page scraped
  • Manually ran revision d26b2999 and completed successfully .
    69 records added, 69 records removed in the database
    1 page scraped
  • Manually ran revision dcb15cc7 and completed successfully .
    69 records added, 69 records removed in the database
    1 page scraped
  • Manually ran revision dcb15cc7 and completed successfully .
    69 records added, 69 records removed in the database
    1 page scraped
  • Manually ran revision dcb15cc7 and completed successfully .
    69 records added in the database
    1 page scraped
  • Manually ran revision dcb15cc7 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Manually ran revision 1315a259 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Manually ran revision 4dc31caf and completed successfully .
    nothing changed in the database
    1 page scraped
  • Manually ran revision 037fe4a9 and failed .
    nothing changed in the database
    27 pages scraped
  • Created on morph.io

Scraper code

Ruby

SA_MHRS_for_OA_input / scraper.rb