kaydensigh / australia_lg_councillors_emails

Scrapes api.morph.io

Get structured data out of the web. Code collaboration through GitHub. Run your scrapers in the cloud.


This pulls lists of Australian local government councillors from other scrapers and builds a database of emails.

Put the list of state databases in an environment variable, e.g.: export MORPH_STATE_DATABASES=openaustralia/qld_lg_directory_councillors,openaustralia/nsw_lg_directory_councillors,kaydensigh/sa_lg_councillors export MORPH_API_KEY=your-api-key The API key lets this scraper download from other scrapers.

Internally it processes all sqlite rows as json objects. Rows are merged in from the state databases if they have the following fields: * councillor or name * council_name or council * council_website or council_url * optionally email

The councillor and council names are used to 'uniquely' identify a person. The council website is used to search for their email. Some state databases already contain emails.

It can handle basic error cases like empty fields, but ideally the input data should be as clean as possible.

It takes many runs to build up the database as it only searches for a limited number of councillors per run.

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Read database (4098 rows). Current memory usage: 25.20 MB no-council-website for Hansford, Nathan (null) no-councillor-name for null (Central Darling Shire Council) no-council-website for COATES, Stephanie (Bayswater) no-council-website for KENYON, Terry (Bayswater) no-council-website for McKENNA, Barry (Bayswater) no-council-website for RADFORD, Alan (Bayswater) no-council-website for SUTHERLAND, Michelle (Bayswater) no-council-website for RIFICI, John (Bayswater) no-council-website for TOLDO, Martin (Bayswater) no-council-website for SABATINO, Michael (Bayswater) no-council-website for CORNISH, Chris (Bayswater) no-council-website for ANDERTON, Mike (Bayswater) no-council-website for ALBERT, Sylvan (Bayswater) Fetching state databases: [ 'kaydensigh/sa_lg_councillors', 'kaydensigh/nsw_lg_directory_councillors', 'kaydensigh/vic_lg_directory_councillors', 'openaustralia/qld_lg_directory_councillors', 'kaydensigh/wa_lg_election_results' ] Merging 855 rows from: kaydensigh/sa_lg_councillors Added 0 new rows. Merging 1463 rows from: kaydensigh/nsw_lg_directory_councillors Added 0 new rows. Merging 624 rows from: kaydensigh/vic_lg_directory_councillors Added 0 new rows. Merging 569 rows from: openaustralia/qld_lg_directory_councillors Added 0 new rows. Merging 593 rows from: kaydensigh/wa_lg_election_results Added 0 new rows. Writing new rows to database. Writing database to disk. Finished. Memory usage: 29.12 MB

Data

Downloaded 6 times by dotnetnoobie kaydensigh otherchirps MikeRalphson TheStarChamberInquirer

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (653 KB) Use the API

rows 10 / 4098

councillor position council_name ward council_website email
Knight, Kevin
Mayor
City of Tea Tree Gully
kevin.knight@cttg.sa.gov.au
Harbinson, Matthew
Councillor
City of Tea Tree Gully
Drumminor Ward
matthew.harbinson@cttg.sa.gov.au
Barbaro, Paul
Councillor
City of Tea Tree Gully
Hillcott Ward
paul.barbaro@cttg.sa.gov.au
Keane, Bernie
Councillor
City of Tea Tree Gully
Pedare Ward
bernie.keane@cttg.sa.gov.au
Jones JP, Lucas
Councillor
City of Tea Tree Gully
Steventon Ward
lucas.jones@cttg.sa.gov.au
McLafferty, Jim
Councillor
City of Tea Tree Gully
Steventon Ward
jim.mclafferty@cttg.sa.gov.au
Massey, Brian
Councillor
City of Tea Tree Gully
Water Gully Ward
brian.massey@cttg.sa.gov.au
Moran, Anne
Adelaide City Council
Area Councillor
dcw.adelaidecitycouncil.com
city@adelaidecitycouncil.com
Haese, Martin
Adelaide City Council
Lord Mayor
dcw.adelaidecitycouncil.com
city@adelaidecitycouncil.com
Hender, Megan
Adelaide City Council
Central Ward
dcw.adelaidecitycouncil.com
city@adelaidecitycouncil.com

Statistics

Average successful run time: 9 minutes

Total run time: about 5 hours

Total cpu time used: 15 minutes

Total disk space used: 706 KB

History

  • Auto ran revision 57a64e1f and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    102 records added, 102 records removed in the database
    238 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    200 records added, 200 records removed in the database
    553 pages scraped
  • ...
  • Created on morph.io

Show complete history