planningalerts-scrapers / saplanningportal

South Australia Development Applications - SA Planning Portal


South Australia (SA) Planning Portal scraper

  • Server - .NET
  • Cookie tracking - No
  • Pagnation - Yes
  • Javascript - No
  • Ajax - Yes
  • Clearly defined data within a record - Yes
  • Scrape detail page - No

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default)
  • thismonth
  • lastmonth
  • ~~year (ie. 2017, min 2003)~~

Enjoy

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching json 2.2.0  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching rubysl-base64 2.0.0  Installing rubysl-base64 2.0.0  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 4 Gemfile dependencies, 21 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (30.25s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Scraping page 1 of 2 Saving record 544/D008/19, Lot Section 130 North Coast Road Point Souttar, SA Saving record 100/D094/19, 26 Alawoona Avenue Mitchell Park, SA Saving record 361/D101/19, Lot 38 Martins Road Salisbury Downs, SA Saving record 252/D150/19, Lot 4100 Troubridge Drive West Lakes, SA Saving record 145/D091/19, Johnson Crescent Port Noarlunga, SA Saving record 145/D090/19, Third Ave Moana, SA Saving record 040/D119/19, Murray Avenue Vale Park, SA Saving record 894/C007/19, Lot 276 in F191648 Chruch Street & Young Street Penola, SA Saving record 170/D093/19, Gorge Road Athelstone, SA Saving record 580/V007/19, 43 Secker Road, Mount Barker, SA Saving record 155/C029/19, 8-14 Briar Road Felixstow, SA Saving record 822/D003/19, Old Naracoorte Road Mount Benson, SA Saving record 050/C012/19, 81 Churchill Road Prospect, SA Saving record 674/D007/19, Lot Block 7A Williams Road & Black Range Road Lucindale, SA Saving record 361/C100/19, 51 Kings Road Salisbury Down, SA Something not right here: {"council_reference"=>"020/A047/15 V6", "address"=>"", "description"=>"", "info_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "comment_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "date_scraped"=>"2019-06-24", "date_received"=>"2019-06-18"} Saving record 020/A039/19, 274 North Terrace, Adelaide, SA Saving record 070/D071/19, 10 Sunhaven Road, Redwood Park, SA Saving record 361/D099/19, 8 Binnowie Street Ingle Farm, SA Saving record 080/D043/19, 26 / 23 Richmond Rd / Monmouth Rd Westbourne Park, SA Saving record 252/C149/19, 236-238 Trimmer Parade Grange, SA Saving record 020/A023/18 V1, 2-56 Franklin Street, Adelaide, SA Saving record 400/D006/19, 205 GOLF COURSE ROAD BORDERTOWN, SA Saving record 110/C033/19, 611 BRIGHTON SEACLIFF, SA Saving record 100/D092/19, 13 Shearing Street Oaklands Park, SA Saving record 100/C093/19, 13 Shearing Street Oaklands Park, SA Saving record 080/D042/19, 12 Dudley Avenue Daw Park, SA Saving record 252/C148/19, 15 Bower Street Woodville, SA Saving record 070/D070/19, 10 Highbury Drive Highbury, SA Saving record 155/D028/19, 87 Ann Street Stepney, SA Saving record 040/D118/19, 12 Austral Avenue Clearview, SA Saving record 100/D091/19, 60 Seymour Terrace Ascot Park, SA Saving record 252/D147/19, Flinders Parade Flinders Park, SA Saving record 361/D098/19, 25 Beovich Road Ingle Farm, SA Saving record 170/C091/19, 70 Arthur Street Tranmere, SA Saving record 170/D092/19, 101 Shakespeare Avenue Magill, SA Saving record 211/C061/19, 531 South Road Keswick , SA Saving record 170/C090/19, 62 Darley Road Paradise, SA Saving record 080/D041/19, 42 Quinlan Avenue St Marys, SA Saving record 020/L038/19, West Terrace Cemetery, SA Saving record 020/D015/19, 88 & 90 Hill Street North Adelaide, SA Saving record 211/D062/19, 65-85 Deeds Road North Plympton, SA Saving record 180/C022/19, 14 Conyngham Glenside, SA Saving record 145/D089/19, 5 MARSTON MORPHETT VALE, SA Saving record 070/D069/19, 4 Loral Street Modbury, SA Saving record 490/D029/19, 48 Athol Road Kudla, SA Saving record 080/D040/19, 13 Ayr Avenue Torrens Park, SA Saving record 145/D088/19, 17 Kenney Street Christies Beach, SA Saving record 711/D008/19, Mackintosh Road Cadell, SA Saving record 490/D026/19, Lot 2,1,9010,9011, Balmoral Road Gawler East , SA Scraping page 2 of 2 Saving record 960/D023/19, 981 & 993c Light Pass Road Vine Vale, SA Saving record 490/D025/19, Lot 9000 Calton Road Gawler East , SA Saving record 252/D145/19, 41 Fredrick Street Welland, SA Saving record 490/C024/19, 45 Alexander Avenue Evanston Park, SA Saving record 252/D144/19, 4 Wharfe Street WOODVILLE SOUTH, SA Saving record 960/D025/19, Lot 1,2,9010,9011 Balmoral Road Gawler East & Kalbeeba, SA Saving record 490/D028/19, Lot 9201,9001,9003 Calton Road Gawler East , SA Saving record 145/D088/19, 17 Kenney Street Christies Beach, SA Saving record 711/D008/19, Mackintosh Road Cadell, SA Saving record 490/D026/19, Lot 2,1,9010,9011, Balmoral Road Gawler East , SA Saving record 960/D024/19, 44 WASHINGTON ST ANGASTON, SA Saving record 490/D027/19, Lot 9002 Calton Road Gawler East , SA Saving record 252/D146/19, 133 Hanson Road Woodville North, SA Saving record 010/U016/19, 137 Fredrickstad Road, Penong, SA Saving record 422/V004/19, 192 Lower Bright Road, Bright, SA Saving record 110/D032/19, 57 Rossall Road Somerton Park, SA Saving record 145/D087/19, Pridmore Ave McLaren Vale, SA Saving record 110/M004/19, 2 Canning Street, Glenelg North, SA Saving record 145/D086/19, Compass Drive Seaford, SA Saving record 252/D143/19, 90 Frogmore Road Kidman Park, SA Saving record 155/D027/19, Lot 82 Lewis Rd Glynde, SA Saving record 170/D088/19, 11 Norman Street Paradise , SA Something not right here: {"council_reference"=>"010/U015/19", "address"=>"", "description"=>"", "info_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "comment_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "date_scraped"=>"2019-06-24", "date_received"=>"2019-06-12"} Saving record 381/C004/19, 238 Wehl Street North Mount Gambier, SA Saving record 490/C022/19, 60 Hillier Road Evanston, SA Saving record 170/D089/19, 8 Wicks Avenue Campbelltown, SA Something not right here: {"council_reference"=>"361/V001/19 V1", "address"=>"", "description"=>"", "info_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "comment_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "date_scraped"=>"2019-06-24", "date_received"=>"2019-06-11"} Saving record 894/V003/19, Lot 1 Pettman Terrace, Mount Burr, SA Saving record 894/D006/19, 508 Belt Road Millient, SA Saving record 252/D142/19, 7 Cavendish Street West Beach, SA Saving record 100/D090/19, 66 Waterhouse Road South Plympton, SA Saving record 292/D048/19, Lot Piece 501 Frisby Road Angle Vale, SA Saving record 490/V003/19, 81 Calton Road, Gawler East, SA Saving record 145/C084/19, 14 Taunton Parade Christies Beach, SA Something not right here: {"council_reference"=>"080/L016/16 V1", "address"=>"", "description"=>"", "info_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "comment_url"=>"http://www.saplanningportal.sa.gov.au/public_register", "date_scraped"=>"2019-06-24", "date_received"=>"2019-06-11"} Saving record 100/L067/19, 30 Hannah Road, Tonsley, SA Saving record 211/L026/18 V1, 2 May Terrace, Lockleys, SA Saving record 100/L066/19, 16 Hannah Road, Tonsley, SA Saving record 040/C116/19, 12 ISLINGTON COURT DUDLEY PARK, SA Saving record 040/D117/19, 315 Victoria Road Largs North, SA Saving record 145/D085/19, 230 Old South Road Old Reynella, SA

Data

Downloaded 746 times by openaustralia jesse-a-reilly tobiasoberrauch Jedwards157

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.63 MB) Use the API

rows 10 / 5865

id council_reference address description info_url comment_url date_scraped date_received
1
252/D341/17
Wattle Street Brompton, SA
Boundary Realignment. Torrens Title division. 2 semi-detached dwellings. 1 detached dwelling.
2017-10-17
2017-10-15
2
040/D295/17
17-23 Ashburton Street Blair Athol, SA
Land Division
2017-10-17
2017-10-13
3
145/D202/17
Valley View Drive McLaren Vale, SA
1 in to 2 land division
2017-10-17
2017-10-13
4
170/D195/17
Andrea Avenue Newton, SA
Torrens Title land division. One allotment into three. Existing dwelling to be demolished.
2017-10-17
2017-10-13
5
170/D196/17
8-10 Gilbert Street NEWTON, SA
1 into 4 Torrens division
2017-10-17
2017-10-13
6
211/D172/17
37 and 39 Malurus Avenue Lockleys, SA
division of land to create 3 new allotments from the existing two.
2017-10-17
2017-10-13
7
361/D188/17
19 Uranbo Street Salisbury North, SA
Torrens Land Division. One Allotment into Two Allotments.
2017-10-17
2017-10-13
8
361/D189/17
7 Gilbert Street Ingle Farm, SA
Land Division
2017-10-17
2017-10-13
9
100/D241/17
9 Rosedale Avenue Morphettville, SA
Division of land to create two new Torrens title allotments
2017-10-17
2017-10-12
10
100/D242/17
19 Lawson Avenue Morphettville, SA
Division of land to create two new Torrens title allotments
2017-10-17
2017-10-12

Statistics

Average successful run time: less than a minute

Total run time: about 6 hours

Total cpu time used: 4 minutes

Total disk space used: 1.66 MB

History

  • Auto ran revision 77f2429f and completed successfully .
    84 records added, 84 records removed, 1 record updated in the database
  • Auto ran revision 77f2429f and completed successfully .
    84 records added, 84 records removed, 1 record updated in the database
  • Auto ran revision 77f2429f and completed successfully .
    86 records added, 86 records removed, 1 record updated in the database
  • Auto ran revision 77f2429f and completed successfully .
    102 records added, 102 records removed, 1 record updated in the database
  • Auto ran revision 77f2429f and completed successfully .
    114 records added, 100 records removed, 1 record updated in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

saplanningportal / scraper.rb