alisonkeen / SA_members_for_OA_parser

Scraper part #2, SA Lower House Member Data

Scrapes www2.parliament.sa.gov.au

Resources Default Page


The data in this scraper is incomplete - actual election type and past MPs aren't included in the database yet. (sorry). Maybe one day... or you could write it if you want!

This parser repeats the scrape of the MP list, and then reads out information for lower house MPs only, into the fields expected by OpenAustralia's parser.

Those two csv files need to be generated separately to get their own URLs, so that forked openaustralia parsers can pull them into a database.

scraper #1: List of MPS - https://morph.io/alisonkeen/SA_MHRS_for_OA_input fields {id, saparl ID, full name} destination: /var/www/openaustralia/openaustralia-parser/data/people.csv

scraper #2: List of MHRs - https://morph.io/alisonkeen/SA_members_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/representatives.csv

scraper #3: List of Senators - https://morph.io/alisonkeen/SA_senators_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/senators.csv

Ruby script to read these three .csv files into OpenAustralia: see https://github.com/alisonkeen/openaustralia-parser-sandbox/import-data-from-morph.rb

This is a scraper that runs on Morph. To get started see the documentation

Contributors alisonkeen

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Installing unf_ext 0.0.7.1 with native extensions  Installing httpclient 2.6.0.1  Installing mime-types 2.5  Installing net-http-digest_auth 1.4  Installing net-http-persistent 2.9.4  Installing mini_portile 0.6.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.1  Installing sqlite3 1.3.10 with native extensions  Using bundler 1.11.2  Installing unf 0.1.4  Installing nokogiri 1.6.6.2 with native extensions  Installing sqlite_magic 0.0.3  Installing domain_name 0.5.24  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing http-cookie 1.0.2  Installing mechanize 2.7.3  Bundle complete! 2 Gemfile dependencies, 17 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (24.40s)  Cleaning up the bundler cache.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... (+ upper house Senator ignored) Name: Hon Michael Atkinson {:member_count=>"531", :id__saparl=>"531", :full_name=>"Hon Michael Atkinson", :division=>"Croydon", :state=>"SA", :start_date=>"25.11.1989", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Frances Bedford {:member_count=>"543", :id__saparl=>"543", :full_name=>"Ms Frances Bedford", :division=>"Florey", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Troy Bell {:member_count=>"4844", :id__saparl=>"4844", :full_name=>"Mr Troy Bell", :division=>"Mount Gambier", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Zoe Bettison {:member_count=>"4623", :id__saparl=>"4623", :full_name=>"Hon Zoe Bettison", :division=>"Ramsay", :state=>"SA", :start_date=>"11.02.2012", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Leon Bignell {:member_count=>"3117", :id__saparl=>"3117", :full_name=>"Hon Leon Bignell", :division=>"Mawson", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Geoff Brock {:member_count=>"3533", :id__saparl=>"3533", :full_name=>"Hon Geoff Brock", :division=>"Frome", :state=>"SA", :start_date=>"17.01.2009", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"IND"} (+ upper house Senator ignored) Name: Hon Paul Caica {:member_count=>"1802", :id__saparl=>"1802", :full_name=>"Hon Paul Caica", :division=>"Colton", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Vickie Chapman {:member_count=>"1804", :id__saparl=>"1804", :full_name=>"Ms Vickie Chapman", :division=>"Bragg", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Susan Close {:member_count=>"4622", :id__saparl=>"4622", :full_name=>"Hon Susan Close", :division=>"Port Adelaide", :state=>"SA", :start_date=>"11.02.2012", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Nat Cook {:member_count=>"4978", :id__saparl=>"4978", :full_name=>"Ms Nat Cook", :division=>"Fisher", :state=>"SA", :start_date=>"16.12.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Mrs Annabel Digance {:member_count=>"4838", :id__saparl=>"4838", :full_name=>"Mrs Annabel Digance", :division=>"Elder", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Sam Duluk {:member_count=>"4991", :id__saparl=>"4991", :full_name=>"Mr Sam Duluk", :division=>"Davenport", :state=>"SA", :start_date=>"31.01.2015", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Mr John Gardner {:member_count=>"4343", :id__saparl=>"4343", :full_name=>"Mr John Gardner", :division=>"Morialta", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Mr Jon Gee {:member_count=>"4845", :id__saparl=>"4845", :full_name=>"Mr Jon Gee", :division=>"Napier", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Mark Goldsworthy {:member_count=>"1805", :id__saparl=>"1805", :full_name=>"Mr Mark Goldsworthy", :division=>"Kavel", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Steven Griffiths {:member_count=>"3118", :id__saparl=>"3118", :full_name=>"Mr Steven Griffiths", :division=>"Goyder", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Martin Hamilton-Smith {:member_count=>"631", :id__saparl=>"631", :full_name=>"Hon Martin Hamilton-Smith", :division=>"Waite", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"IND"} Name: Ms Katrine Hildyard {:member_count=>"4846", :id__saparl=>"4846", :full_name=>"Ms Katrine Hildyard", :division=>"Reynell", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Mr Eddie Hughes {:member_count=>"4839", :id__saparl=>"4839", :full_name=>"Mr Eddie Hughes", :division=>"Giles", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Hon Tom Kenyon {:member_count=>"3119", :id__saparl=>"3119", :full_name=>"Hon Tom Kenyon", :division=>"Newland", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Steph Key {:member_count=>"539", :id__saparl=>"539", :full_name=>"Hon Steph Key", :division=>"Ashford", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Stephan Knoll {:member_count=>"4847", :id__saparl=>"4847", :full_name=>"Mr Stephan Knoll", :division=>"Schubert", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Tom Koutsantonis {:member_count=>"633", :id__saparl=>"633", :full_name=>"Hon Tom Koutsantonis", :division=>"West Torrens", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) (+ upper house Senator ignored) (+ upper house Senator ignored) (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Mr Steven Marshall {:member_count=>"4338", :id__saparl=>"4338", :full_name=>"Mr Steven Marshall", :division=>"Dunstan", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Dr Duncan McFetridge {:member_count=>"1807", :id__saparl=>"1807", :full_name=>"Dr Duncan McFetridge", :division=>"Morphett", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Hon Stephen Mullighan {:member_count=>"4842", :id__saparl=>"4842", :full_name=>"Hon Stephen Mullighan", :division=>"Lee", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Mr Lee Odenwalder {:member_count=>"4335", :id__saparl=>"4335", :full_name=>"Mr Lee Odenwalder", :division=>"Little Para", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Mr Adrian Pederick {:member_count=>"3120", :id__saparl=>"3120", :full_name=>"Mr Adrian Pederick ", :division=>"Hammond", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Michael Pengilly {:member_count=>"3121", :id__saparl=>"3121", :full_name=>"Mr Michael Pengilly ", :division=>"Finniss", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Tony Piccolo {:member_count=>"3123", :id__saparl=>"3123", :full_name=>"Hon Tony Piccolo", :division=>"Light", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Chris Picton {:member_count=>"4841", :id__saparl=>"4841", :full_name=>"Mr Chris Picton", :division=>"Kaurna", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr David Pisoni {:member_count=>"3124", :id__saparl=>"3124", :full_name=>"Mr David Pisoni", :division=>"Unley", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Jennifer Rankine {:member_count=>"614", :id__saparl=>"614", :full_name=>"Hon Jennifer Rankine", :division=>"Wright", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon John Rau {:member_count=>"1810", :id__saparl=>"1810", :full_name=>"Hon John Rau", :division=>"Enfield", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Isobel Redmond {:member_count=>"1813", :id__saparl=>"1813", :full_name=>"Ms Isobel Redmond", :division=>"Heysen", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Ms Rachel Sanderson {:member_count=>"4340", :id__saparl=>"4340", :full_name=>"Ms Rachel Sanderson", :division=>"Adelaide", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Jack Snelling {:member_count=>"627", :id__saparl=>"627", :full_name=>"Hon Jack Snelling", :division=>"Playford", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr David Speirs {:member_count=>"4837", :id__saparl=>"4837", :full_name=>"Mr David Speirs", :division=>"Bright", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Mr Vincent Tarzia {:member_count=>"4840", :id__saparl=>"4840", :full_name=>"Mr Vincent Tarzia", :division=>"Hartley", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Peter Treloar {:member_count=>"4341", :id__saparl=>"4341", :full_name=>"Mr Peter Treloar", :division=>"Flinders", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Dan van Holst Pellekaan {:member_count=>"4342", :id__saparl=>"4342", :full_name=>"Mr Dan van Holst Pellekaan", :division=>"Stuart", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Hon Leesa Vlahos {:member_count=>"4336", :id__saparl=>"4336", :full_name=>"Hon Leesa Vlahos ", :division=>"Taylor", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Hon Jay Weatherill {:member_count=>"1812", :id__saparl=>"1812", :full_name=>"Hon Jay Weatherill", :division=>"Cheltenham", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Tim Whetstone {:member_count=>"4339", :id__saparl=>"4339", :full_name=>"Mr Tim Whetstone", :division=>"Chaffey", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Mitch Williams {:member_count=>"546", :id__saparl=>"546", :full_name=>"Mr Mitch Williams", :division=>"MacKillop", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Corey Wingard {:member_count=>"4843", :id__saparl=>"4843", :full_name=>"Mr Corey Wingard", :division=>"Mitchell", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Ms Dana Wortley {:member_count=>"4848", :id__saparl=>"4848", :full_name=>"Ms Dana Wortley", :division=>"Torrens", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored)

Data

Downloaded 10 times by alisonkeen

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (9 KB) Use the API

rows 10 / 47

member_count id__saparl full_name division state start_date election_type end_date reason_left party
531
531
Hon Michael Atkinson
Croydon
SA
25.11.1989
still_in_office
ALP
543
543
Ms Frances Bedford
Florey
SA
11.10.1997
still_in_office
ALP
4844
4844
Mr Troy Bell
Mount Gambier
SA
15.03.2014
still_in_office
LIB
4623
4623
Hon Zoe Bettison
Ramsay
SA
11.02.2012
still_in_office
ALP
3117
3117
Hon Leon Bignell
Mawson
SA
18.03.2006
still_in_office
ALP
3533
3533
Hon Geoff Brock
Frome
SA
17.01.2009
still_in_office
IND
1802
1802
Hon Paul Caica
Colton
SA
09.02.2002
still_in_office
ALP
1804
1804
Ms Vickie Chapman
Bragg
SA
09.02.2002
still_in_office
LIB
4622
4622
Hon Susan Close
Port Adelaide
SA
11.02.2012
still_in_office
ALP
4978
4978
Ms Nat Cook
Fisher
SA
16.12.2014
still_in_office
ALP

Statistics

Average successful run time: 2 minutes

Total run time: 15 minutes

Total cpu time used: half a minute

Total disk space used: 50.7 KB

History

  • Manually ran revision 90aa1921 and completed successfully .
    47 records added, 47 records removed in the database
    48 pages scraped
  • Manually ran revision 90aa1921 and failed .
    5 records added, 5 records removed in the database
    6 pages scraped
  • Manually ran revision 90aa1921 and completed successfully .
    47 records added, 47 records removed in the database
    48 pages scraped
  • Manually ran revision 9c00567a and failed .
    nothing changed in the database
    2 pages scraped
  • Manually ran revision de32df18 and completed successfully .
    47 records added, 47 records removed in the database
    48 pages scraped
  • Manually ran revision 6c12bff1 and completed successfully .
    47 records added, 47 records removed in the database
    48 pages scraped
  • Manually ran revision 76700439 and completed successfully .
    47 records added in the database
    48 pages scraped
  • Manually ran revision 85d3e949 and completed successfully .
    nothing changed in the database
    70 pages scraped
  • Created on morph.io