planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching json 2.2.0  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Installing json 2.2.0 with native extensions  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (30.57s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record POS19/0229, Birtinya Lot 900 Saving record REC15/0067.03, 16 Sutherland St BUDERIM Saving record REC15/0067.02, 16 Sutherland St BUDERIM Saving record REC09/0031.05, The Ridge 2 Bronze Wing Cct PEREGIAN SPRINGS Saving record REC09/0030.03, The Ridge Ridges Bvd PEREGIAN SPRINGS Saving record OPW19/0537, 11 Snapper St LITTLE MOUNTAIN Saving record MCU17/2092.01, 18 Doolan St NAMBOUR Saving record OPW19/0536, 92 Aerodrome Rd MAROOCHYDORE Saving record OPW19/0538, 27 Kooringal Cres BUDDINA Saving record OPW19/0535, Maroochy River Motel 361 Bradman Ave MAROOCHYDORE Saving record OPW18/0770.02, Coral Sea Dr PELICAN WATERS Saving record POS19/0239, 1 O'Connor Street Maroochydore Saving record POS19/0245, Stage 2 Unitywater Depot Saving record RAL19/0112, 1 & 37 Godfreys Road Bli Bli Saving record OPW19/0539, 1 Teske Dr PALMWOODS Saving record MCU19/0246, 247 Dales Road Chevallum Saving record RAL19/0092.01, 31, 37 and 43 Owen Creek Rd FOREST GLEN Saving record MCU12/0082.02, Lavarack Crescent 1/16 Lavarack Cres BUDERIM Saving record OPW19/0542, 4 Blackwing St PELICAN WATERS Saving record MCU01/0125.01, First Avenue Mooloolaba 19/39 First Ave MOOLOOLABA Saving record RAL18/0085.04, 7 Pinkwood St PALMVIEW Saving record MCU19/0248, 21 Queen of Colonies Pde MOFFAT BEACH Saving record MCU19/0249, 12 Ellora Ct ROSEMOUNT Saving record OPW19/0540, 432 Paynters Creek Rd ROSEMOUNT Saving record OPW17/2174.01, 42 and 44 Gloucester Rd BUDERIM Saving record MCU19/0247, 111 Anderson Rd EERWAH VALE Saving record RAL19/0008.06, 30 Tranquility Way PALMVIEW Saving record OPW19/0544, 12 Garden Ct BUDERIM Saving record MCU19/0252, 2 Vyner Ct FOREST GLEN Saving record MCU19/0251, 12 and 14 Maud St NAMBOUR Saving record MCU05/0213.01, Vista Estate 3/14-18 Bade St NAMBOUR Saving record OPW19/0545, 10 Crestmont Dr BUDERIM Saving record MCU19/0250, 55 Ball Rd EUMUNDI Saving record MCU19/0253, 31, 37 and 43 Owen Creek Rd FOREST GLEN Saving record OPW19/0543, 26 Mount Pleasant Rd NAMBOUR Saving record REC16/0085.02, 47 Petrie Creek Road ROSEMOUNT Saving record OPW19/0551, 70 and 72 Memorial Ave MAROOCHYDORE Saving record OPW19/0550, 18 Maleny St LANDSBOROUGH Saving record OPW19/0549, 7 Paynter Park Dr and 49 Countryview St WOOMBYE Saving record OPW19/0548, 16 Dulin St MAROOCHYDORE Saving record OPW19/0546, 31, 37 and 43 Owen Creek Rd FOREST GLEN Saving record OPW19/0547, 12 Hideaway Lane GLENVIEW Saving record MCU19/0255, 16 Dulin St MAROOCHYDORE Saving record MCU19/0254, 892 Eumundi Noosa Rd DOONAN Saving record RAL18/0012.11, Lot 929 Walling St PALMVIEW Saving record OPW19/0555, 32 Kondalilla Falls Rd MONTVILLE Saving record OPW19/0554, 169 Glenmount Rd TANAWHA Saving record OPW19/0553, 95-123 Glenmount Rd TANAWHA Saving record MCU11/0208.04, 1 Development Ct CALOUNDRA WEST Saving record 2006/510039.01, 740 Nicklin Way CURRIMUNDI Saving record OPW19/0552, 40 Menzies Dr PACIFIC PARADISE Saving record RAL19/0113, Lot 347 Laxton Rd PALMVIEW Saving record OPW19/0556, 244 Wilson Rd ILKLEY Saving record OPW19/0557, Kurrajong Cl MOOLOOLAH VALLEY Saving record RAL19/0114, FASTRACK 406 and 408 Nicklin Way BOKARINA Saving record OPW19/0558, 28 Birdsong Dr MOOLOOLAH VALLEY Saving record REC16/0069.28, 4 Banfield Pl MAROOCHYDORE Saving record RAL19/0115, 50 Chants Rd VALDORA Saving record MCU19/0257, 28 Ron Parkenson Cres CORBOULD PARK Saving record MCU17/0148.03, 1 Mantra Esp BIRTINYA

Data

Downloaded 6034 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas ajoh3691 Jedwards157

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.39 MB) Use the API

rows 10 / 3566

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 1.42 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    60 records added, 54 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    66 records added, 59 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    69 records added, 69 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    64 records added, 64 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    64 records added, 64 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb