kaydensigh / australia_lg_councillors_emails

Scrapes api.morph.io

Get structured data out of the web. Code collaboration through GitHub. Run your scrapers in the cloud.


This pulls lists of Australian local government councillors from other scrapers and builds a database of emails.

Put the list of state databases in an environment variable, e.g.: export MORPH_STATE_DATABASES=openaustralia/qld_lg_directory_councillors,openaustralia/nsw_lg_directory_councillors,kaydensigh/sa_lg_councillors export MORPH_API_KEY=your-api-key The API key lets this scraper download from other scrapers.

Internally it processes all sqlite rows as json objects. Rows are merged in from the state databases if they have the following fields: * councillor or name * council_name or council * council_website or council_url * optionally email

The councillor and council names are used to 'uniquely' identify a person. The council website is used to search for their email. Some state databases already contain emails.

It can handle basic error cases like empty fields, but ideally the input data should be as clean as possible.

It takes many runs to build up the database as it only searches for a limited number of councillors per run.

Contributors kaydensigh

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Read database (4098 rows). Current memory usage: 25.20 MB no-council-website for Hansford, Nathan (null) no-councillor-name for null (Central Darling Shire Council) no-council-website for COATES, Stephanie (Bayswater) no-council-website for KENYON, Terry (Bayswater) no-council-website for McKENNA, Barry (Bayswater) no-council-website for RADFORD, Alan (Bayswater) no-council-website for SUTHERLAND, Michelle (Bayswater) no-council-website for RIFICI, John (Bayswater) no-council-website for TOLDO, Martin (Bayswater) no-council-website for SABATINO, Michael (Bayswater) no-council-website for CORNISH, Chris (Bayswater) no-council-website for ANDERTON, Mike (Bayswater) no-council-website for ALBERT, Sylvan (Bayswater) Fetching state databases: [ 'kaydensigh/sa_lg_councillors', 'kaydensigh/nsw_lg_directory_councillors', 'kaydensigh/vic_lg_directory_councillors', 'openaustralia/qld_lg_directory_councillors', 'kaydensigh/wa_lg_election_results' ] Merging 855 rows from: kaydensigh/sa_lg_councillors Added 0 new rows. Merging 1463 rows from: kaydensigh/nsw_lg_directory_councillors Added 0 new rows. Merging 624 rows from: kaydensigh/vic_lg_directory_councillors Added 0 new rows. Merging 569 rows from: openaustralia/qld_lg_directory_councillors Added 0 new rows. Merging 593 rows from: kaydensigh/wa_lg_election_results Added 0 new rows. Writing new rows to database. Writing database to disk. Finished. Memory usage: 29.12 MB

Data

Downloaded 5 times by dotnetnoobie MikeRalphson kaydensigh otherchirps

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (653 KB) Use the API

rows 10 / 4098

councillor position council_name ward council_website email
Knight, Kevin
Mayor
City of Tea Tree Gully
kevin.knight@cttg.sa.gov.au
Harbinson, Matthew
Councillor
City of Tea Tree Gully
Drumminor Ward
matthew.harbinson@cttg.sa.gov.au
Barbaro, Paul
Councillor
City of Tea Tree Gully
Hillcott Ward
paul.barbaro@cttg.sa.gov.au
Keane, Bernie
Councillor
City of Tea Tree Gully
Pedare Ward
bernie.keane@cttg.sa.gov.au
Jones JP, Lucas
Councillor
City of Tea Tree Gully
Steventon Ward
lucas.jones@cttg.sa.gov.au
McLafferty, Jim
Councillor
City of Tea Tree Gully
Steventon Ward
jim.mclafferty@cttg.sa.gov.au
Massey, Brian
Councillor
City of Tea Tree Gully
Water Gully Ward
brian.massey@cttg.sa.gov.au
Moran, Anne
Adelaide City Council
Area Councillor
dcw.adelaidecitycouncil.com
city@adelaidecitycouncil.com
Haese, Martin
Adelaide City Council
Lord Mayor
dcw.adelaidecitycouncil.com
city@adelaidecitycouncil.com
Hender, Megan
Adelaide City Council
Central Ward
dcw.adelaidecitycouncil.com
city@adelaidecitycouncil.com

Statistics

Average successful run time: 9 minutes

Total run time: about 5 hours

Total cpu time used: 15 minutes

Total disk space used: 763 KB

History

  • Auto ran revision 57a64e1f and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    102 records added, 102 records removed in the database
    238 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    200 records added, 200 records removed in the database
    553 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    140 records added, 140 records removed in the database
    328 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    140 records added, 140 records removed in the database
    291 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    593 records added in the database
    5 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    47 records added, 47 records removed in the database
    191 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    190 records added, 190 records removed in the database
    719 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    120 records added, 120 records removed in the database
    316 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    150 records added, 150 records removed in the database
    456 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    150 records added, 150 records removed in the database
    397 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    200 records added, 200 records removed in the database
    788 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    200 records added, 200 records removed in the database
    631 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    200 records added, 200 records removed in the database
    650 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    160 records added, 160 records removed in the database
    436 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    200 records added, 200 records removed in the database
    671 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    200 records added, 200 records removed in the database
    686 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    170 records added, 170 records removed in the database
    509 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    160 records added, 160 records removed in the database
    490 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    160 records added, 160 records removed in the database
    489 pages scraped
  • Auto ran revision 57a64e1f and completed successfully .
    190 records added, 190 records removed in the database
    573 pages scraped
  • Manually ran revision 57a64e1f and completed successfully .
    90 records added, 90 records removed in the database
    260 pages scraped
  • Manually ran revision 57a64e1f and completed successfully .
    140 records added, 140 records removed in the database
    450 pages scraped
  • Manually ran revision 57a64e1f and completed successfully .
    3505 records added in the database
    4 pages scraped
  • Manually ran revision 5c545d4e and completed successfully .
    160 records updated in the database
    462 pages scraped
  • Manually ran revision 19f0aea3 and completed successfully .
    100 records updated in the database
    317 pages scraped
  • Manually ran revision 8cd078dd and failed .
    nothing changed in the database
    65 pages scraped
  • Manually ran revision 8cd078dd and completed successfully .
    2656 records added in the database
    4 pages scraped
  • Manually ran revision 8cd078dd and completed successfully .
    nothing changed in the database
    1 page scraped
  • Manually ran revision 8cd078dd and completed successfully .
    72 records updated in the database
    244 pages scraped
  • Manually ran revision 8cd078dd and completed successfully .
    849 records added in the database
    1 page scraped
  • Manually ran revision f5ceef5d and failed .
    nothing changed in the database
  • Manually ran revision 014519cc and failed .
    nothing changed in the database
    222 pages scraped
  • Manually ran revision 014519cc and failed .
    nothing changed in the database
    336 pages scraped
  • Manually ran revision 921e4c09 and completed successfully .
    849 records added in the database
    1 page scraped
  • Manually ran revision fc8f73aa and failed .
    nothing changed in the database
    1 page scraped
  • Created on morph.io