planningalerts-scrapers / saplanningportal

South Australia Development Applications - SA Planning Portal


South Australia (SA) Planning Portal scraper

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

  • Server - .NET
  • Cookie tracking - No
  • Pagnation - Yes
  • Javascript - No
  • Ajax - Yes
  • Clearly defined data within a record - Yes
  • Scrape detail page - No

Enjoy

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching rubysl-base64 2.0.0  Installing rubysl-base64 2.0.0  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 4 Gemfile dependencies, 21 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.52s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scraping page 1 of 38 Scraping page 2 of 38 Scraping page 3 of 38 Scraping page 4 of 38 Scraping page 5 of 38 Scraping page 6 of 38 Scraping page 7 of 38 Scraping page 8 of 38 Scraping page 9 of 38 Scraping page 10 of 38 Scraping page 11 of 38 Scraping page 12 of 38 Scraping page 13 of 38 Scraping page 14 of 38 Scraping page 15 of 38 Something not right here: {"council_reference"=>"21026046", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026046-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-07"} Scraping page 16 of 38 Something not right here: {"council_reference"=>"21026046", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026046-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-07"} Scraping page 17 of 38 Scraping page 18 of 38 Scraping page 19 of 38 Scraping page 20 of 38 Something not right here: {"council_reference"=>"21026045", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026045-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-03"} Scraping page 21 of 38 Scraping page 22 of 38 Scraping page 23 of 38 Something not right here: {"council_reference"=>"21026065", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026065-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Something not right here: {"council_reference"=>"21026165", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026165-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Scraping page 24 of 38 Something not right here: {"council_reference"=>"21026149", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026149-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Something not right here: {"council_reference"=>"21026156", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026156-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Something not right here: {"council_reference"=>"21026159", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026159-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Something not right here: {"council_reference"=>"21026065", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026065-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Something not right here: {"council_reference"=>"21026165", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026165-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Scraping page 25 of 38 Something not right here: {"council_reference"=>"21026065", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026065-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Something not right here: {"council_reference"=>"21026165", "address"=>"", "description"=>"Detached dwelling", "info_url"=>"https://plan.sa.gov.au/development_application_register#view-21026165-DAP", "comment_url"=>"https://plan.sa.gov.au/development_application_register", "date_scraped"=>"2021-09-17", "date_received"=>"2021-09-01"} Scraping page 26 of 38 Scraping page 27 of 38 Scraping page 28 of 38 Scraping page 29 of 38 Scraping page 30 of 38 Scraping page 31 of 38 Scraping page 32 of 38 Scraping page 33 of 38 Scraping page 34 of 38 Scraping page 35 of 38 Scraping page 36 of 38 Scraping page 37 of 38 Scraping page 38 of 38

Data

Downloaded 1677 times by openaustralia dcalde jesse-a-reilly DevAppProject softgrow tobiasoberrauch Jedwards157 MichaelBone

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (10.1 MB) Use the API

rows 10 / 30498

id council_reference address description info_url comment_url date_scraped date_received
1
252/D341/17
Wattle Street Brompton, SA
Boundary Realignment. Torrens Title division. 2 semi-detached dwellings. 1 detached dwelling.
2017-10-17
2017-10-15
2
040/D295/17
17-23 Ashburton Street Blair Athol, SA
Land Division
2017-10-17
2017-10-13
3
145/D202/17
Valley View Drive McLaren Vale, SA
1 in to 2 land division
2017-10-17
2017-10-13
4
170/D195/17
Andrea Avenue Newton, SA
Torrens Title land division. One allotment into three. Existing dwelling to be demolished.
2017-10-17
2017-10-13
5
170/D196/17
8-10 Gilbert Street NEWTON, SA
1 into 4 Torrens division
2017-10-17
2017-10-13
6
211/D172/17
37 and 39 Malurus Avenue Lockleys, SA
division of land to create 3 new allotments from the existing two.
2017-10-17
2017-10-13
7
361/D188/17
19 Uranbo Street Salisbury North, SA
Torrens Land Division. One Allotment into Two Allotments.
2017-10-17
2017-10-13
8
361/D189/17
7 Gilbert Street Ingle Farm, SA
Land Division
2017-10-17
2017-10-13
9
100/D241/17
9 Rosedale Avenue Morphettville, SA
Division of land to create two new Torrens title allotments
2017-10-17
2017-10-12
10
100/D242/17
19 Lawson Avenue Morphettville, SA
Division of land to create two new Torrens title allotments
2017-10-17
2017-10-12

Statistics

Average successful run time: 2 minutes

Total run time: 2 days

Total cpu time used: 32 minutes

Total disk space used: 10.2 MB

History

  • Auto ran revision 67f31762 and completed successfully .
    2761 records added, 2555 records removed, 1 record updated in the database
  • Auto ran revision 67f31762 and completed successfully .
    2802 records added, 2613 records removed, 1 record updated in the database
  • Auto ran revision 67f31762 and completed successfully .
    2783 records added, 2586 records removed, 1 record updated in the database
  • Auto ran revision 67f31762 and completed successfully .
    2802 records added, 2802 records removed, 1 record updated in the database
  • Auto ran revision 67f31762 and completed successfully .
    2929 records added, 2757 records removed, 1 record updated in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

saplanningportal / scraper.rb