alisonkeen / SA_senators_for_OA_parser

List of Senators with details to read in to OpenAustralia Parser existing scripts

Scrapes www2.parliament.sa.gov.au

Resources Default Page


The data in this scraper is incomplete - actual election type and past MPs aren't included in the database yet. (sorry). Maybe one day... or you could write it if you want!

This parser repeats the scrape of the MP list, and then reads out information for upper house MPs only, into the fields expected by OpenAustralia's parser.

Those two csv files need to be generated separately to get their own URLs, so that forked openaustralia parsers can pull them into a database.

scraper #1: List of MPS - https://morph.io/alisonkeen/SA_MHRS_for_OA_input fields {id, saparl ID, full name} destination: /var/www/openaustralia/openaustralia-parser/data/people.csv

scraper #2: List of MHRs - https://morph.io/alisonkeen/SA_members_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/representatives.csv

scraper #3: List of Senators - https://morph.io/alisonkeen/SA_senators_for_OA_parser fields {id, id2, Name, Electorate, State/Territory(?), Start date, Election type, End date, reason, Most recent party} destination: /var/www/openaustralia/openaustralia-parser/data/senators.csv

Ruby script to read these three .csv files into OpenAustralia: see https://github.com/alisonkeen/openaustralia-parser-sandbox/import-data-from-morph.rb

This is a scraper that runs on Morph. To get started see the documentation

Contributors alisonkeen

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon Robert Brokenshire {:member_count=>"3489", :id__saparl=>"3489", :full_name=>"Hon Robert Brokenshire", :division=>" ", :state=>"SA", :start_date=>"24.07.2008", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"FFP"} (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon John Darley {:member_count=>"3404", :id__saparl=>"3404", :full_name=>"Hon John Darley", :division=>" ", :state=>"SA", :start_date=>"21.11.2007", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"IND"} Name: Hon John Dawkins {:member_count=>"599", :id__saparl=>"599", :full_name=>"Hon John Dawkins", :division=>" ", :state=>"SA", :start_date=>"11.10.1997", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ lower house member ignored) (+ lower house member ignored) Name: Hon Tammy Franks {:member_count=>"4363", :id__saparl=>"4363", :full_name=>"Hon Tammy Franks", :division=>" ", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"GSA"} Name: Hon Gail Gago {:member_count=>"1821", :id__saparl=>"1821", :full_name=>"Hon Gail Gago", :division=>" ", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ lower house member ignored) Name: Hon John Gazzola {:member_count=>"1822", :id__saparl=>"1822", :full_name=>"Hon John Gazzola", :division=>" ", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon Dennis Hood {:member_count=>"3126", :id__saparl=>"3126", :full_name=>"Hon Dennis Hood", :division=>" ", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"FFP"} (+ lower house member ignored) Name: Hon Ian Hunter {:member_count=>"3122", :id__saparl=>"3122", :full_name=>"Hon Ian Hunter", :division=>" ", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Gerry Kandelaars {:member_count=>"4564", :id__saparl=>"4564", :full_name=>"Hon Gerry Kandelaars", :division=>" ", :state=>"SA", :start_date=>"13.09.2011", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon Jing Lee {:member_count=>"4362", :id__saparl=>"4362", :full_name=>"Hon Jing Lee", :division=>" ", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Michelle Lensink {:member_count=>"2742", :id__saparl=>"2742", :full_name=>"Hon Michelle Lensink", :division=>" ", :state=>"SA", :start_date=>"26.06.2003", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Rob Lucas {:member_count=>"605", :id__saparl=>"605", :full_name=>"Hon Rob Lucas", :division=>" ", :state=>"SA", :start_date=>"06.11.1982", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} Name: Hon Kyam Maher {:member_count=>"4697", :id__saparl=>"4697", :full_name=>"Hon Kyam Maher", :division=>" ", :state=>"SA", :start_date=>"17.10.2012", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} Name: Hon Peter Malinauskas {:member_count=>"5084", :id__saparl=>"5084", :full_name=>"Hon Peter Malinauskas", :division=>" ", :state=>"SA", :start_date=>"01.12.2015", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ lower house member ignored) (+ lower house member ignored) Name: Hon Andrew McLachlan CSC {:member_count=>"4866", :id__saparl=>"4866", :full_name=>"Hon Andrew McLachlan CSC", :division=>" ", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ lower house member ignored) Name: Hon Tung Ngo {:member_count=>"4867", :id__saparl=>"4867", :full_name=>"Hon Tung Ngo", :division=>" ", :state=>"SA", :start_date=>"15.03.2014", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"} (+ lower house member ignored) Name: Hon Mark Parnell {:member_count=>"3130", :id__saparl=>"3130", :full_name=>"Hon Mark Parnell", :division=>" ", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"GSA"} (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon David Ridgway {:member_count=>"1820", :id__saparl=>"1820", :full_name=>"Hon David Ridgway", :division=>" ", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon Terry Stephens {:member_count=>"1819", :id__saparl=>"1819", :full_name=>"Hon Terry Stephens", :division=>" ", :state=>"SA", :start_date=>"09.02.2002", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon Kelly Vincent {:member_count=>"4364", :id__saparl=>"4364", :full_name=>"Hon Kelly Vincent", :division=>" ", :state=>"SA", :start_date=>"20.03.2010", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"D4D"} (+ lower house member ignored) Name: Hon Stephen Wade {:member_count=>"3164", :id__saparl=>"3164", :full_name=>"Hon Stephen Wade", :division=>" ", :state=>"SA", :start_date=>"02.05.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"LIB"} (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) (+ lower house member ignored) Name: Hon Russell Wortley {:member_count=>"3125", :id__saparl=>"3125", :full_name=>"Hon Russell Wortley", :division=>" ", :state=>"SA", :start_date=>"18.03.2006", :election_type=>"", :end_date=>"", :reason_left=>"still_in_office", :party=>"ALP"}

Data

Downloaded 2 times by alisonkeen

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (6 KB) Use the API

rows 10 / 22

member_count id__saparl full_name division state start_date election_type end_date reason_left party
3489
3489
Hon Robert Brokenshire
SA
24.07.2008
still_in_office
FFP
3404
3404
Hon John Darley
SA
21.11.2007
still_in_office
IND
599
599
Hon John Dawkins
SA
11.10.1997
still_in_office
LIB
4363
4363
Hon Tammy Franks
SA
20.03.2010
still_in_office
GSA
1821
1821
Hon Gail Gago
SA
09.02.2002
still_in_office
ALP
1822
1822
Hon John Gazzola
SA
09.02.2002
still_in_office
ALP
3126
3126
Hon Dennis Hood
SA
18.03.2006
still_in_office
FFP
3122
3122
Hon Ian Hunter
SA
18.03.2006
still_in_office
ALP
4564
4564
Hon Gerry Kandelaars
SA
13.09.2011
still_in_office
ALP
4362
4362
Hon Jing Lee
SA
20.03.2010
still_in_office
LIB

Statistics

Average successful run time: less than a minute

Total run time: 2 minutes

Total cpu time used: less than 10 seconds

Total disk space used: 36.6 KB

History

  • Manually ran revision b298e373 and completed successfully .
    22 records added, 22 records removed in the database
    23 pages scraped
  • Manually ran revision bdc464d6 and completed successfully .
    22 records added in the database
    23 pages scraped
  • Created on morph.io