LoveMyData / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Contributors LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.76s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Skipping already saved record MCU22/0305 Skipping already saved record OPW22/0487 Skipping already saved record RAL22/0104 Skipping already saved record OPW22/0488 Skipping already saved record OPW22/0489 Skipping already saved record MCU22/0306 Skipping already saved record OPW22/0490 Skipping already saved record 1999/510068.01 Skipping already saved record POS22/0155 Skipping already saved record POS22/0160 Skipping already saved record POS22/0173 Skipping already saved record POS22/0177 Skipping already saved record MCU22/0308 Skipping already saved record RAL21/0102.01 Skipping already saved record RAL22/0105 Skipping already saved record OPW22/0491 Skipping already saved record OPW22/0492 Skipping already saved record OPW22/0493 Skipping already saved record MCU19/0002.01 Skipping already saved record RAL18/0097.01 Skipping already saved record RAL22/0106 Skipping already saved record MCU22/0309 Skipping already saved record 2004/56M0017.01 Skipping already saved record MCU02/0110.02 Skipping already saved record 2001/500025.02 Skipping already saved record REC17/0090.03 Skipping already saved record MCU22/0310 Skipping already saved record OPW22/0494 Skipping already saved record POS22/0166 Skipping already saved record POS22/0187 Skipping already saved record 2009/510022.11 Skipping already saved record OPW22/0495 Skipping already saved record OPW20/0193.04 Skipping already saved record OPW22/0496 Skipping already saved record MCU22/0311 Skipping already saved record OPW22/0497 Skipping already saved record MCU22/0312 Skipping already saved record MCU22/0313 Skipping already saved record MCU22/0314 Skipping already saved record OPW22/0498 Skipping already saved record OPW22/0499 Skipping already saved record MCU22/0315 Skipping already saved record REC02/0291.01 Skipping already saved record MCU15/0270.07 Saving record MCU22/0316, 4 Turner St BEERWAH Saving record RAL22/0107, Nambour Sewerage Treatment #4 and Nambour Sewerage Treatment Bli Bli Rd NAMBOUR Saving record RAL22/0108, 92 Ratcliffe Rd DIAMOND VALLEY Skipping already saved record OPW22/0500 Saving record RAL22/0109, 2781 Eumundi-Kenilworth Rd KENILWORTH Saving record MCU22/0317, 35 Central Park Dr YANDINA Saving record OPW22/0501, 35 Central Park Dr YANDINA Saving record OPW22/0502, Forest Glen Shops 354 Mons Rd and 15 Grammar School Way FOREST GLEN error to parse council_reference's address: OPW22/0503 Saving record MCU18/0310.05, 16 Grammar School Way FOREST GLEN Saving record REC09/0065.41, 1 Lady Elliot St MOUNTAIN CREEK Saving record OPW22/0504, 6 Conebush St MUDJIMBA

Data

Downloaded 598 times by dcalde MichaelBone EMBLAPROJECTS

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.51 MB) Use the API

rows 10 / 8031

council_reference address description info_url comment_url date_scraped date_received
REC15/0004.01
110 North Maleny Rd NORTH MALENY
Generally In Accordance - Staged Reconfiguration of 1 Lot into 5 - Just Sheds
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
RAL18/0076.01
21 and 31 Coonowrin Rd GLASS HOUSE MOUNTAINS
Change to Development Approval - Reconfiguring 2 Lots into 24 Lots - Glasshouse Developments Pty Ltd - Innovative Planning Solutions
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
POS19/0030
183 Glenview Road Glenview
Survey Plan SP299300 (2 Lots) - DK & D Langdon - Subdivisions (Qld) Pty Ltd
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
POS19/0028
Habitat Palmwoods Stage 2
Abbotts Road Palmwoods - Survey Plan SP307684 (15 Lots, 14 created and balance Lot 500) - Sunshine Beach Constructions Pty Ltd
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW19/0045
10 Donovan Pl MAROOCHY RIVER
Vegetation Clearing - Tree Fix
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW19/0044
FAST TRACK 165 Oceanic Dve WARANA
Landscaping - Dual Occupancy - Shane and Laura Blake - Adams and Sparkes Town Planning and Development COMBINED MCU19/0019
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW19/0043
Bruce Hwy PALMVIEW
Electrical and Street Lighting - AVID Residential Group Pty Ltd - Project Urban
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
OPW18/0620.01
21 and 31 Coonowrin Rd GLASS HOUSE MOUNTAINS
Change to Development Approval - Roadworks, Stormwater, Earthworks, Drainage Works and Landscaping - Glasshouse Developments Pty Ltd - Innovative Planning Solutions
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
MCU19/0021
1-47 Stringybark Rd BUDERIM
Educational Establishment - Matthew Flinders Anglican College - Adamson Town Planning
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04
MCU19/0020
1-3 Claude Boyd Pde BELLS CREEK
Service Station and Shopping Complex - Westcon Construction and Development Pty Ltd - FMAAP
mail@sunshinecoast.qld.gov.au
2019-03-08
2019-02-04

Statistics

Average successful run time: 1 minute

Total run time: 1 day

Total cpu time used: 22 minutes

Total disk space used: 2.54 MB

History

  • Auto ran revision f1c531be and completed successfully .
    10 records added in the database
  • Auto ran revision f1c531be and completed successfully .
    nothing changed in the database
  • Auto ran revision f1c531be and completed successfully .
    7 records added in the database
  • Auto ran revision f1c531be and completed successfully .
    nothing changed in the database
  • Auto ran revision f1c531be and completed successfully .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb