LoveMyData / onkaparinga

Onkaparinga SA Development Applications

Scrapes pathway.onkaparinga.sa.gov.au

IIS7


City Of Onkaparinga Council Scraper

Note - This scraper will only scrape the primary DA and any subsequent DAs will not be scrape. eg. 683/2017/2 will not be scraped but 683/2017 will

  • Server - .NET - ePathway
  • Cookie tracking - Yes
  • Pagination - No
  • Javascript - Yes
  • Clearly defined data within a row - No but acceptable
  • Force scan one record at a time - Yes
  • User-Agent - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • year value (default is this year)

Enjoy

Contributors emikulic henare LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.1  Fetching httpclient 2.6.0.1  Fetching mime-types 2.6.1  Installing httpclient 2.6.0.1  Installing unf_ext 0.0.7.1 with native extensions  Installing mime-types 2.6.1  Fetching net-http-digest_auth 1.4  Installing net-http-digest_auth 1.4  Fetching net-http-persistent 2.9.4  Fetching mini_portile 0.6.2  Installing net-http-persistent 2.9.4  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Installing mini_portile 0.6.2  Fetching webrobots 0.1.1  Fetching sqlite3 1.3.10  Installing webrobots 0.1.1  Fetching nokogiri 1.6.6.2  Installing sqlite3 1.3.10 with native extensions  Installing nokogiri 1.6.6.2 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.24  Installing domain_name 0.5.24  Fetching http-cookie 1.0.2  Installing http-cookie 1.0.2  Fetching sqlite_magic 0.0.5  Installing sqlite_magic 0.0.5  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.3  Installing mechanize 2.7.3  Bundle complete! 2 Gemfile dependencies, 17 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (28.20s)  Cleaning up the bundler cache. -----> Detecting rake tasks   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.3.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in year `2018`, changable via MORPH_PERIOD environment Skipping already saved record 1700/2018 Saving record 1701/2018, 6 Barcelona Drive, HAPPY VALLEY SA 5159 Saving record 1702/2018, 302 Pimpala Road, WOODCROFT SA 5162 Saving record 1704/2018, Allot 1009 Commercial Road, MOANA SA 5169 Saving record 1705/2018, 8 Haven Avenue, SEAFORD SA 5169 Saving record 1706/2018, 37 Elanora Avenue, MORPHETT VALE SA 5162 Saving record 1707/2018, 6 Narina Way, ABERFOYLE PARK SA 5159 Saving record 1708/2018, 34 Vendale Drive, FLAGSTAFF HILL SA 5159 Saving record 1709/2018, 30 Jennifer Drive, HAPPY VALLEY SA 5159 Saving record 1710/2018, 62 Victor Harbor Road, OLD NOARLUNGA SA 5168 Saving record 1711/2018, 35 Chappel Avenue, MORPHETT VALE SA 5162 Saving record 1712/2018, 64 Hunter Road, CHRISTIES BEACH SA 5165 Saving record 1713/2018, 431 States Road, MORPHETT VALE SA 5162 Saving record 1714/2018, 16 Corn Street, OLD REYNELLA SA 5161 Skipping already saved record 6028/2018 Skipping already saved record 8109/2018 Saving record 8110/2018, 88 Main South Road, MORPHETT VALE SA 5162 Saving record 8111/2018, 58 McLaren Flat Road, MCLAREN FLAT SA 5171 Saving record 8112/2018, 329 Aldinga Road, WHITES VALLEY SA 5172

Data

Downloaded 402 times by LoveMyData equivalentideas

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (17.3 MB) Use the API

rows 10 / 51822

council_reference address description info_url comment_url date_scraped date_received
1/2017
11 Glenloth Drive, HAPPY VALLEY SA 5159
Verandah
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-02
2/2017
5 Tiller Drive, SEAFORD SA 5169
Single storey dwelling (dwelling 2)
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
3/2017
13 Kingsbridge Drive, MORPHETT VALE SA 5162
Single storey dwelling
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
4/2017
1/1927 Main South Road, HACKHAM SA 5163
Demolition of dwelling
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
5/2017
12 Semillon Circuit, REYNELLA SA 5161
Verandah
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
6/2017
28 Angle Road, SEAFORD MEADOWS SA 5169
Verandah
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
7/2017
13 Fabian Crescent, MORPHETT VALE SA 5162
Shed
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
8/2017
6A Sturt Lane, HACKHAM SA 5163
Shed
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
9/2017
30 Meadowvale Road, COROMANDEL VALLEY SA 5051
Retaining wall
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-03
10/2017
20 Daintree Drive, SEAFORD RISE SA 5169
Carport
mailto:mail@onkaparinga.sa.gov.au
2017-05-19
2017-01-04

Statistics

Average successful run time: 8 minutes

Total run time: 2 days

Total cpu time used: about 1 hour

Total disk space used: 17.4 MB

History

  • Auto ran revision 00a893f0 and completed successfully .
    16 records added in the database
    103 pages scraped
  • Auto ran revision 00a893f0 and completed successfully .
    29 records added in the database
    127 pages scraped
  • Auto ran revision 00a893f0 and completed successfully .
    3 records added in the database
    75 pages scraped
  • Auto ran revision 00a893f0 and completed successfully .
    nothing changed in the database
    69 pages scraped
  • Auto ran revision 00a893f0 and completed successfully .
    nothing changed in the database
    69 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

onkaparinga / scraper.rb