alisonkeen / SA_members_for_OA_parser

Scraper part #2, SA Lower House Member Data

Resources Default Page

README.md

The data in this scraper is incomplete - actual election type and past MPs aren't included in the database yet. (sorry). Maybe one day... or you could write it if you want!

This parser repeats the scrape of the MP list, and then reads out information for lower house MPs only, into the fields expected by OpenAustralia's parser.

Those two csv files need to be generated separately to get their own URLs, so that forked openaustralia parsers can pull them into a database.

scraper #1: List of MPS - https://morph.io/alisonkeen/SA_MHRS_for_OA_input fields {id, saparl ID, full name} destination: /var/www/openaustralia/openaustralia-parser/data/people.csv

scraper #2: List of MHRs - https://morph.io/alisonkeen/SA_members_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/representatives.csv

scraper #3: List of Senators - https://morph.io/alisonkeen/SA_senators_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/senators.csv

Ruby script to read these three .csv files into OpenAustralia: see https://github.com/alisonkeen/openaustralia-parser-sandbox/import-data-from-morph.rb

This is a scraper that runs on Morph. To get started see the documentation

Contributors

Last run completed successfully 2016-10-29.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.0.0 [1G-----> Installing dependencies using bundler 1.11.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Fetching gem metadata from https://rubygems.org/......... [1G Fetching version metadata from https://rubygems.org/.. [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation. [1G Installing unf_ext 0.0.7.1 with native extensions [1G Installing httpclient 2.6.0.1 [1G Installing mime-types 2.5 [1G Installing net-http-digest_auth 1.4 [1G Installing net-http-persistent 2.9.4 [1G Installing mini_portile 0.6.2 [1G Installing ntlm-http 0.1.1 [1G Installing webrobots 0.1.1 [1G Installing sqlite3 1.3.10 with native extensions [1G Using bundler 1.11.2 [1G Installing unf 0.1.4 [1G Installing nokogiri 1.6.6.2 with native extensions [1G Installing sqlite_magic 0.0.3 [1G Installing domain_name 0.5.24 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Installing http-cookie 1.0.2 [1G Installing mechanize 2.7.3 [1G Bundle complete! 2 Gemfile dependencies, 17 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into ./vendor/bundle. [1G Bundle completed (24.40s) [1G Cleaning up the bundler cache. [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... (+ upper house Senator ignored) Name: Hon Michael Atkinson {:member_count=>"531", :id__saparl=>"531", :full_name=>"Hon Michael Atkinson", :division=>"Croydon", :state=>"SA", :start_date=>"25.11.1989", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Frances Bedford {:member_count=>"543", :id__saparl=>"543", :full_name=>"Ms Frances Bedford", :division=>"Florey", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Troy Bell {:member_count=>"4844", :id__saparl=>"4844", :full_name=>"Mr Troy Bell", :division=>"Mount Gambier", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Zoe Bettison {:member_count=>"4623", :id__saparl=>"4623", :full_name=>"Hon Zoe Bettison", :division=>"Ramsay", :state=>"SA", :start_date=>"11.02.2012", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Leon Bignell {:member_count=>"3117", :id__saparl=>"3117", :full_name=>"Hon Leon Bignell", :division=>"Mawson", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Geoff Brock {:member_count=>"3533", :id__saparl=>"3533", :full_name=>"Hon Geoff Brock", :division=>"Frome", :state=>"SA", :start_date=>"17.01.2009", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"IND"} (+ upper house Senator ignored) Name: Hon Paul Caica {:member_count=>"1802", :id__saparl=>"1802", :full_name=>"Hon Paul Caica", :division=>"Colton", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Vickie Chapman {:member_count=>"1804", :id__saparl=>"1804", :full_name=>"Ms Vickie Chapman", :division=>"Bragg", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Susan Close {:member_count=>"4622", :id__saparl=>"4622", :full_name=>"Hon Susan Close", :division=>"Port Adelaide", :state=>"SA", :start_date=>"11.02.2012", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Nat Cook {:member_count=>"4978", :id__saparl=>"4978", :full_name=>"Ms Nat Cook", :division=>"Fisher", :state=>"SA", :start_date=>"16.12.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Mrs Annabel Digance {:member_count=>"4838", :id__saparl=>"4838", :full_name=>"Mrs Annabel Digance", :division=>"Elder", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Sam Duluk {:member_count=>"4991", :id__saparl=>"4991", :full_name=>"Mr Sam Duluk", :division=>"Davenport", :state=>"SA", :start_date=>"31.01.2015", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Mr John Gardner {:member_count=>"4343", :id__saparl=>"4343", :full_name=>"Mr John Gardner", :division=>"Morialta", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Mr Jon Gee {:member_count=>"4845", :id__saparl=>"4845", :full_name=>"Mr Jon Gee", :division=>"Napier", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Mark Goldsworthy {:member_count=>"1805", :id__saparl=>"1805", :full_name=>"Mr Mark Goldsworthy", :division=>"Kavel", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Steven Griffiths {:member_count=>"3118", :id__saparl=>"3118", :full_name=>"Mr Steven Griffiths", :division=>"Goyder", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Martin Hamilton-Smith {:member_count=>"631", :id__saparl=>"631", :full_name=>"Hon Martin Hamilton-Smith", :division=>"Waite", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"IND"} Name: Ms Katrine Hildyard {:member_count=>"4846", :id__saparl=>"4846", :full_name=>"Ms Katrine Hildyard", :division=>"Reynell", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Mr Eddie Hughes {:member_count=>"4839", :id__saparl=>"4839", :full_name=>"Mr Eddie Hughes", :division=>"Giles", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Hon Tom Kenyon {:member_count=>"3119", :id__saparl=>"3119", :full_name=>"Hon Tom Kenyon", :division=>"Newland", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Steph Key {:member_count=>"539", :id__saparl=>"539", :full_name=>"Hon Steph Key", :division=>"Ashford", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Stephan Knoll {:member_count=>"4847", :id__saparl=>"4847", :full_name=>"Mr Stephan Knoll", :division=>"Schubert", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Tom Koutsantonis {:member_count=>"633", :id__saparl=>"633", :full_name=>"Hon Tom Koutsantonis", :division=>"West Torrens", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) (+ upper house Senator ignored) (+ upper house Senator ignored) (+ upper house Senator ignored) (+ upper house Senator ignored) Name: Mr Steven Marshall {:member_count=>"4338", :id__saparl=>"4338", :full_name=>"Mr Steven Marshall", :division=>"Dunstan", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Dr Duncan McFetridge {:member_count=>"1807", :id__saparl=>"1807", :full_name=>"Dr Duncan McFetridge", :division=>"Morphett", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Hon Stephen Mullighan {:member_count=>"4842", :id__saparl=>"4842", :full_name=>"Hon Stephen Mullighan", :division=>"Lee", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Mr Lee Odenwalder {:member_count=>"4335", :id__saparl=>"4335", :full_name=>"Mr Lee Odenwalder", :division=>"Little Para", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Mr Adrian Pederick {:member_count=>"3120", :id__saparl=>"3120", :full_name=>"Mr Adrian Pederick ", :division=>"Hammond", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Michael Pengilly {:member_count=>"3121", :id__saparl=>"3121", :full_name=>"Mr Michael Pengilly ", :division=>"Finniss", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Tony Piccolo {:member_count=>"3123", :id__saparl=>"3123", :full_name=>"Hon Tony Piccolo", :division=>"Light", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Chris Picton {:member_count=>"4841", :id__saparl=>"4841", :full_name=>"Mr Chris Picton", :division=>"Kaurna", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr David Pisoni {:member_count=>"3124", :id__saparl=>"3124", :full_name=>"Mr David Pisoni", :division=>"Unley", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Jennifer Rankine {:member_count=>"614", :id__saparl=>"614", :full_name=>"Hon Jennifer Rankine", :division=>"Wright", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon John Rau {:member_count=>"1810", :id__saparl=>"1810", :full_name=>"Hon John Rau", :division=>"Enfield", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Ms Isobel Redmond {:member_count=>"1813", :id__saparl=>"1813", :full_name=>"Ms Isobel Redmond", :division=>"Heysen", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Ms Rachel Sanderson {:member_count=>"4340", :id__saparl=>"4340", :full_name=>"Ms Rachel Sanderson", :division=>"Adelaide", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Jack Snelling {:member_count=>"627", :id__saparl=>"627", :full_name=>"Hon Jack Snelling", :division=>"Playford", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr David Speirs {:member_count=>"4837", :id__saparl=>"4837", :full_name=>"Mr David Speirs", :division=>"Bright", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Mr Vincent Tarzia {:member_count=>"4840", :id__saparl=>"4840", :full_name=>"Mr Vincent Tarzia", :division=>"Hartley", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Peter Treloar {:member_count=>"4341", :id__saparl=>"4341", :full_name=>"Mr Peter Treloar", :division=>"Flinders", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Dan van Holst Pellekaan {:member_count=>"4342", :id__saparl=>"4342", :full_name=>"Mr Dan van Holst Pellekaan", :division=>"Stuart", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ upper house Senator ignored) Name: Hon Leesa Vlahos {:member_count=>"4336", :id__saparl=>"4336", :full_name=>"Hon Leesa Vlahos ", :division=>"Taylor", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored) Name: Hon Jay Weatherill {:member_count=>"1812", :id__saparl=>"1812", :full_name=>"Hon Jay Weatherill", :division=>"Cheltenham", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Mr Tim Whetstone {:member_count=>"4339", :id__saparl=>"4339", :full_name=>"Mr Tim Whetstone", :division=>"Chaffey", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Mitch Williams {:member_count=>"546", :id__saparl=>"546", :full_name=>"Mr Mitch Williams", :division=>"MacKillop", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Mr Corey Wingard {:member_count=>"4843", :id__saparl=>"4843", :full_name=>"Mr Corey Wingard", :division=>"Mitchell", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Ms Dana Wortley {:member_count=>"4848", :id__saparl=>"4848", :full_name=>"Ms Dana Wortley", :division=>"Torrens", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ upper house Senator ignored)

Data

Downloaded 10 times by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (9 KB) Use the API

rows 10 / 47

member_count	id__saparl	full_name	division	state	start_date	reason_left	party
531	531	Hon Michael Atkinson	Croydon	SA	25.11.1989	still_in_office	ALP
543	543	Ms Frances Bedford	Florey	SA	11.10.1997	still_in_office	ALP
4844	4844	Mr Troy Bell	Mount Gambier	SA	15.03.2014	still_in_office	LIB
4623	4623	Hon Zoe Bettison	Ramsay	SA	11.02.2012	still_in_office	ALP
3117	3117	Hon Leon Bignell	Mawson	SA	18.03.2006	still_in_office	ALP
3533	3533	Hon Geoff Brock	Frome	SA	17.01.2009	still_in_office	IND
1802	1802	Hon Paul Caica	Colton	SA	09.02.2002	still_in_office	ALP
1804	1804	Ms Vickie Chapman	Bragg	SA	09.02.2002	still_in_office	LIB
4622	4622	Hon Susan Close	Port Adelaide	SA	11.02.2012	still_in_office	ALP
4978	4978	Ms Nat Cook	Fisher	SA	16.12.2014	still_in_office	ALP

Statistics

Average successful run time: 2 minutes

Total run time: 15 minutes

Total cpu time used: half a minute

Total disk space used: 50.7 KB

History

Manually ran revision 90aa1921 and completed successfully 2016-10-29.

run time 3 minutes

47 records added, 47 records removed in the database

48 pages scraped
Manually ran revision 90aa1921 and failed 2016-09-22.

run time 18 s

5 records added, 5 records removed in the database

6 pages scraped
Manually ran revision 90aa1921 and completed successfully 2016-09-22.

run time 2 minutes

47 records added, 47 records removed in the database

48 pages scraped
Manually ran revision 9c00567a and failed 2016-09-22.

run time 7 s

nothing changed in the database

2 pages scraped
Manually ran revision de32df18 and completed successfully 2016-09-15.

run time 2 minutes

47 records added, 47 records removed in the database

48 pages scraped
Manually ran revision 6c12bff1 and completed successfully 2016-09-15.

run time 2 minutes

47 records added, 47 records removed in the database

48 pages scraped
Manually ran revision 76700439 and completed successfully 2016-09-15.

run time 2 minutes

47 records added in the database

48 pages scraped
Manually ran revision 85d3e949 and completed successfully 2016-09-15.

run time 3 minutes

nothing changed in the database

70 pages scraped
Created on morph.io 2016-09-15

Scraper code

Ruby

SA_members_for_OA_parser / scraper.rb

git clone URL