LoveMyData / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Contributors LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Installing connection_pool 2.2.2  Fetching json 2.2.0  Fetching mime-types-data 3.2018.0812  Installing json 2.2.0 with native extensions  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.85s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Skipping already saved record 2005/520077.07 Skipping already saved record MCU15/0101.02 Skipping already saved record OPW22/0241 Skipping already saved record OPW22/0242 Skipping already saved record OPW22/0243 Skipping already saved record RAL21/0021.02 Skipping already saved record OPW22/0244 Skipping already saved record OPW22/0245 Skipping already saved record POS22/0075 Skipping already saved record OPW22/0246 Skipping already saved record OPW22/0247 Skipping already saved record POS21/0308 Skipping already saved record RAL21/0021.03 Skipping already saved record OPW22/0248 Skipping already saved record OPW21/0214.01 Skipping already saved record MCU18/0358.01 Skipping already saved record MCU22/0157 Skipping already saved record OPW22/0249 Skipping already saved record RAL22/0056 Skipping already saved record OPW22/0250 Skipping already saved record MCU22/0158 Skipping already saved record OPW22/0251 Skipping already saved record REC04/0219.32 Skipping already saved record OPW22/0252 Skipping already saved record OPW19/0115.01 Skipping already saved record POS22/0001 Skipping already saved record POS22/0026 Skipping already saved record MCU20/0154.01 Skipping already saved record MCU22/0159 Skipping already saved record MCU18/0281.01 Skipping already saved record MCU22/0160 Skipping already saved record OPW22/0253 Saving record POS22/0044, 23 Kombi St BOKARINA Skipping already saved record POS22/0065 Skipping already saved record POS22/0074 Skipping already saved record POS22/0082 Skipping already saved record 2000/500055.01 Skipping already saved record OPW22/0254 Skipping already saved record MCU17/2035.01 Skipping already saved record MCU18/0286.03 Skipping already saved record MCU20/0116.02 Skipping already saved record RAL22/0057 Skipping already saved record MCU22/0161 Skipping already saved record OPW22/0255 Skipping already saved record MCU22/0162 Skipping already saved record OPW22/0256 Skipping already saved record MCU16/0113.06 Skipping already saved record OPW22/0257 Skipping already saved record MCU22/0163 Skipping already saved record OPW22/0258 Skipping already saved record MCU22/0164 Skipping already saved record OPW22/0259 Skipping already saved record OPW22/0260 Skipping already saved record OPW22/0261 Skipping already saved record OPW22/0262 Skipping already saved record OPW22/0264 Skipping already saved record RAL22/0059 Skipping already saved record MCU22/0167 Skipping already saved record 1999/160001.05 Skipping already saved record OPW22/0266 Skipping already saved record MCU22/0169 Skipping already saved record MCU22/0170 Skipping already saved record OPW22/0267 Skipping already saved record OPW21/0062.03 Skipping already saved record MCU22/0166 Skipping already saved record OPW22/0265 Skipping already saved record RAL22/0058 Skipping already saved record OPW22/0263 Skipping already saved record MCU22/0165 Skipping already saved record MCU22/0171 Skipping already saved record OPW22/0269 Skipping already saved record MCU14/0153.01 Skipping already saved record OPW22/0270 Saving record REC13/0074.01, 257 Burnside Road BURNSIDE Skipping already saved record OPW22/0268 Saving record RAL21/0074.01, 811 Eumundi Noosa Rd DOONAN Saving record MCU22/0172, 112 Maleny-Kenilworth Rd MALENY Saving record OPW22/0271, 92 Marina View Dr PELICAN WATERS Saving record MCU22/0173, 75 Oceanic Drive WARANA Saving record OPW22/0272, 75 Oceanic Drive WARANA Saving record OPW22/0273, 56 Pine Grove Road WOOMBYE error to parse council_reference's address: OPW22/0274 Saving record MCU22/0174, 24 Conebush Street MUDJIMBA Saving record OPW22/0275, 24 Conebush Street MUDJIMBA

Data

Downloaded 463 times by dcalde MichaelBone EMBLAPROJECTS

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.29 MB) Use the API

rows 10 / 7312

council_reference address description info_url comment_url date_scraped date_received
REC15/0004.01
110 North Maleny Rd NORTH MALENY
Generally In Accordance - Staged Reconfiguration of 1 Lot into 5 - Just Sheds
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
RAL18/0076.01
21 and 31 Coonowrin Rd GLASS HOUSE MOUNTAINS
Change to Development Approval - Reconfiguring 2 Lots into 24 Lots - Glasshouse Developments Pty Ltd - Innovative Planning Solutions
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
POS19/0030
183 Glenview Road Glenview
Survey Plan SP299300 (2 Lots) - DK & D Langdon - Subdivisions (Qld) Pty Ltd
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
POS19/0028
Habitat Palmwoods Stage 2
Abbotts Road Palmwoods - Survey Plan SP307684 (15 Lots, 14 created and balance Lot 500) - Sunshine Beach Constructions Pty Ltd
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW19/0045
10 Donovan Pl MAROOCHY RIVER
Vegetation Clearing - Tree Fix
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW19/0044
FAST TRACK 165 Oceanic Dve WARANA
Landscaping - Dual Occupancy - Shane and Laura Blake - Adams and Sparkes Town Planning and Development COMBINED MCU19/0019
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW19/0043
Bruce Hwy PALMVIEW
Electrical and Street Lighting - AVID Residential Group Pty Ltd - Project Urban
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW18/0620.01
21 and 31 Coonowrin Rd GLASS HOUSE MOUNTAINS
Change to Development Approval - Roadworks, Stormwater, Earthworks, Drainage Works and Landscaping - Glasshouse Developments Pty Ltd - Innovative Planning Solutions
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
MCU19/0021
1-47 Stringybark Rd BUDERIM
Educational Establishment - Matthew Flinders Anglican College - Adamson Town Planning
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
MCU19/0020
1-3 Claude Boyd Pde BELLS CREEK
Service Station and Shopping Complex - Westcon Construction and Development Pty Ltd - FMAAP
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04

Statistics

Average successful run time: 1 minute

Total run time: 1 day

Total cpu time used: 20 minutes

Total disk space used: 2.32 MB

History

  • Auto ran revision f1c531be and completed successfully .
    10 records added in the database
  • Auto ran revision f1c531be and completed successfully .
    9 records added in the database
  • Auto ran revision f1c531be and completed successfully .
    nothing changed in the database
  • Auto ran revision f1c531be and completed successfully .
    12 records added in the database
  • Auto ran revision f1c531be and completed successfully .
    8 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb