LoveMyData / onkaparinga

Onkaparinga SA Development Applications


City Of Onkaparinga Council Scraper

Note - This scraper will only scrape the primary DA and any subsequent DAs will not be scrape. eg. 683/2017/2 will not be scraped but 683/2017 will

  • Server - .NET - ePathway
  • Cookie tracking - Yes
  • Pagination - No
  • Javascript - Yes
  • Clearly defined data within a row - No but acceptable
  • Force scan one record at a time - Yes
  • User-Agent - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • year value (default is this year)

Enjoy

Contributors emikulic henare LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Getting data in year `2019`, changable via MORPH_PERIOD environment Skipping already saved record 854/2019 Skipping already saved record 6007/2019 Skipping already saved record 8049/2019

Data

Downloaded 670 times by LoveMyData equivalentideas

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (18.1 MB) Use the API

rows 10 / 54056

council_reference address description info_url comment_url date_scraped date_received
1/2017
11 Glenloth Drive, HAPPY VALLEY SA 5159
Verandah
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-02
2/2017
5 Tiller Drive, SEAFORD SA 5169
Single storey dwelling (dwelling 2)
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
3/2017
13 Kingsbridge Drive, MORPHETT VALE SA 5162
Single storey dwelling
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
4/2017
1/1927 Main South Road, HACKHAM SA 5163
Demolition of dwelling
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
5/2017
12 Semillon Circuit, REYNELLA SA 5161
Verandah
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
6/2017
28 Angle Road, SEAFORD MEADOWS SA 5169
Verandah
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
7/2017
13 Fabian Crescent, MORPHETT VALE SA 5162
Shed
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
8/2017
6A Sturt Lane, HACKHAM SA 5163
Shed
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
9/2017
30 Meadowvale Road, COROMANDEL VALLEY SA 5051
Retaining wall
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
10/2017
20 Daintree Drive, SEAFORD RISE SA 5169
Carport
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-04

Statistics

Average successful run time: 6 minutes

Total run time: 3 days

Total cpu time used: about 1 hour

Total disk space used: 18.1 MB

History

  • Auto ran revision 00a893f0 and completed successfully .
    nothing changed in the database
  • Auto ran revision 00a893f0 and completed successfully .
    11 records added in the database
  • Auto ran revision 00a893f0 and completed successfully .
    8 records added in the database
  • Auto ran revision 00a893f0 and completed successfully .
    12 records added in the database
  • Auto ran revision 00a893f0 and completed successfully .
    12 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

onkaparinga / scraper.rb