planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.89s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record MCU21/0244, 9 Bayonne Cl LITTLE MOUNTAIN Saving record OPW21/0376, FAST TRACK 10 Paulger Flat Rd YANDINA Saving record MCU21/0246, FAST TRACK 10 Paulger Flat Rd YANDINA Saving record REC16/0069.49, 20 Bedarra Cct MAROOCHYDORE Saving record MPC18/0004.05, Lot 905 Bokarina Blvd BOKARINA Saving record REC11/0041.02, DH 2a Stewart Way SHELLY BEACH Saving record MCU18/0205.01, 19-27 Fred Chaplin Cct CORBOULD PARK Saving record POS21/0166, 8 Reflection Ct NAMBOUR Saving record MCU21/0247, FASTRACK 25 Shady Grove TANAWHA Saving record OPW21/0380, 159 Hodgens Rd BALD KNOB Saving record OPW21/0378, 70-82 Crescent Rd EUMUNDI Saving record MCU14/0163.03, 138-174 Camp Flat Road BLI BLI Saving record OPW21/0379, 10 Yulunga Pl MOOLOOLABA Saving record RAL21/0083, 29-31 Crescent Rd EUMUNDI Saving record MCU21/0248, Lot 3291 Koel Cct PEREGIAN SPRINGS Saving record OPW21/0377, 87A Glenmount Rd MONS Saving record MCU17/2190.01, 311 Petrie Creek Rd ROSEMOUNT Saving record OPW20/0040.02, 743 Old Maroochydore Rd FOREST GLEN Saving record OPW21/0381, 23 Princess Cres NAMBOUR Saving record POS21/0169, 8 Eliza Way Maroochydore Saving record OPW21/0383, 6 Nambour Mapleton Rd NAMBOUR Saving record RAL18/0058.01, 131 Memorial Dr EUMUNDI Saving record OPW19/0220.02, 10 Picnic Point Esp MAROOCHYDORE Saving record OPW21/0382, 32 Amarina Ave MOOLOOLABA Saving record MPC20/0002.01, Amendment to Master Plan No. 117 (site Development Plan Saving record MCU21/0249, 9 Century Ct MOUNT COOLUM Saving record RAL17/2032.09, DH Azure Forest Glen 16 Cycad Lane FOREST GLEN Saving record MCU21/0251, Lot 1 Commercial Rd KULUIN Saving record OPW21/0385, 45 Sippy Downs Dr SIPPY DOWNS Saving record OPW21/0384, 13 Laurel Ave NAMBOUR Saving record REC15/0114.07, 43 Nyes Cres BUDERIM Saving record MCU21/0250, The Ridge 34 Honeyeater Cres PEREGIAN SPRINGS Saving record MCU20/0172.01, 22 Wharf St, 47 & 51 Central Park Dr YANDINA Saving record MCU16/0145.02, The Hampton Residences 2/8 Culbara St MOOLOOLABA Saving record OPW21/0386, 1505 Nambour North Connection Rd YANDINA Saving record OPW21/0387, 30 Rita Ct TANAWHA Saving record RAL18/0042.01, 22 Wharf St, 47 & 51 Central Park Dr YANDINA Saving record POS21/0171, ** Note on MCU20/0115 to send referral to Compliance when Plan Seal app lodged **10 Boorook St BUDDINA Saving record OPW21/0388, Lot 1 Commercial Rd KULUIN Saving record M987680.01, 49-51 River Esp MOOLOOLABA Saving record MCU21/0252, 55 Running Creek Rd NORTH ARM Saving record MCU04/0172.01, 13 Mari St ALEXANDRA HEADLANDS Saving record RAL21/0084, FASTRACK 42 Vernon St NAMBOUR Saving record OPW21/0391, 18 Colemans Road YANDINA Saving record OPW21/0389, 49 Coolibah St MUDJIMBA Saving record OPW20/0498.03, 56 Sugar Bag Rd LITTLE MOUNTAIN Saving record MPC21/0007, Bokarina Bvd BOKARINA Saving record MCU21/0253, 3 Post Office Rd MAPLETON Saving record OPW21/0393, 15 Grammar School Way FOREST GLEN & 354 Mons Road FOREST GLEN Saving record OPW21/0392, 1 Greentree Pl DOONAN Saving record OPW21/0390, 102 Mooloolah Drive Mooloolaba Saving record MCU21/0254, FASTRACK 81 Lapoinya Cres WARANA Saving record 2007/56R0019.23, 58 Frankland Ave MERIDAN PLAINS and 31 Prudence Place MERIDAN PLAINS Saving record OPW21/0396, 23 Roderick St MOFFAT BEACH Saving record OPW21/0394, FASTRACK 81 Lapoinya Cres WARANA Saving record OPW21/0397, 185 Palmview Forest Dr PALMVIEW Saving record MCU21/0256, DH 668 Glenview Rd GLENVIEW Saving record OPW21/0398, 39-41 Old Gympie Rd YANDINA Saving record MCU20/0157.01, 17 Lomandra Pl COOLUM BEACH -Minor Change to increase gross floor area within 2 existing tenancies- tenancy 3 and 6 Saving record OPW21/0139.01, Pelican Waters Southern Lakes Stage 4E Saving record OPW21/0395, 1 Lorikeet Pl BURNSIDE Saving record OPW21/0400, 40 Menzies Drive PACIFIC PARADISE Saving record MCU21/0258, 194 Old Palmwoods Rd PALMWOODS Saving record 2001/570020.02, 11 Maui Court PARREARRA Saving record REC15/0043.02, 16 Jorl Cout BUDERIM Saving record MCU21/0257, 24-26 Keil St WOOMBYE Saving record MCU21/0261, 7 Gregson Pl CALOUNDRA Saving record MCU21/0260, Kawana Way BIRTINYA Saving record RAL18/0125.02, Maddock Ave, Brandenburg Rd, King Rd and 170 King Rd MOOLOOLAH VALLEY Saving record MCU21/0259, 184-186 Flaxton Dr FLAXTON Saving record RAL20/0025.04, 8 Emerald Street PALMVIEW Saving record OPW21/0401, Kawana Way BIRTINYA Saving record OPW21/0402, Maddock Ave MOOLOOLAH VALLEY

Data

Downloaded 9670 times by slav123 openaustralia LoveMyData dcalde DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas yourneighbourhood ajoh3691 Jedwards157 EMBLAPROJECTS Corris224

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.78 MB) Use the API

rows 10 / 7540

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 2.81 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 73 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 73 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 64 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    75 records added, 62 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    67 records added, 62 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb