planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.55s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record POS22/0162, 38 Bluebird Parade Bokarina Saving record MCU22/0296, The Oasis On Golden Beach 2 Landsborough Pde and Landsborough Pde and Oasis 1/2 Landsborough Pde GOLDEN BEACH Saving record RAL22/0102, Sunshine Coast Airport 50 Airport Dr MARCOOLA Saving record OPW22/0473, Sunshine Coast Airport 50 Airport Dr MARCOOLA Saving record OPW22/0474, 38 Undanbi Pl PELICAN WATERS Saving record MCU19/0287.01, 74 Sugar Road Maroochydore Saving record REC15/0196.16, 31 Chambers Ct GLENVIEW Saving record MCU21/0412.02, 6-14 Fred Chaplin Cct CORBOULD PARK Saving record RAL20/0096.04, 21 Weebill St PALMVIEW Saving record OPW22/0226.01, 6-14 Fred Chaplin Circuit CORBOULD PARK Saving record RAL21/0020.02, 113 Carnarvon Dr BEERWAH Saving record 2004/520007.38, 37 Marmont St PELICAN WATERS Saving record MCU22/0297, 17 Nichols Ave BEERWAH Saving record OPW22/0475, 17 Nichols Ave BEERWAH Saving record OPW22/0476, Lake Kawana Bvd BIRTINYA Saving record MCU22/0298, 2 Princess Dr BLI BLI Saving record RAL21/0027.06, DH Saving record OPW22/0477, 2 Princess Dr BLI BLI Saving record OPW19/0283.01, 47 and 49 Owen Creek Rd FOREST GLEN Saving record MCU22/0299, Lenco Cres LANDSBOROUGH Saving record ToAdviseOPW01, 2 Bilenda Cl BEERWAH Saving record RAL22/0103, 128 and 172 Toolborough Rd YANDINA CREEK Saving record OPW22/0478, Sunshine Coast Airport 790-818 David Low Way MARCOOLA Saving record MCU22/0300, 17 Karawatha Dr MOUNTAIN CREEK Saving record OPW22/0479, 17 Karawatha Dr MOUNTAIN CREEK Saving record MCU20/0006.04, 1-3 and 5 The Esplanade MAROOCHYDORE Saving record OPW22/0480, 1-5 Mons School Rd MONS Saving record OPW22/0481, 79 Hardwood Rd LANDSBOROUGH Saving record MCU22/0301, Bruce Hwy PALMVIEW Saving record MCU22/0302, 18 Werona St BUDDINA Saving record OPW22/0482, 18 Werona St BUDDINA Saving record RAL21/0138.01, 3490 & 3492 & 3494 & 3496-3512 Maleny-Kenilworth Road, KENILWORTH Saving record MCU16/0111.03, 211, 227 Saving record OPW22/0483, 30 Currie St NAMBOUR Saving record OPW22/0484, Forest Glen Shops 354 Mons Rd and 370 Mons Rd and 15 Grammar School Way FOREST GLEN Saving record OPW20/0090.01, 37 Smith Street MOOLOOLABA Saving record MCU22/0303, Combined OPW22/0485 Saving record OPW22/0485, Combined MCU22/0303 Saving record OPW22/0486, Bruce Hwy PALMVIEW Saving record RAL20/0115.01, Lot 3000 Sunshine Cove Way MAROOCHYDORE Saving record MCU21/0023.01, 31 Chambers Ct GLENVIEW Saving record MCU22/0304, 89 Wilson Lane and 51-85 Wilson Lane WOOMBYE Saving record MCU22/0305, 3 Greber Rd BEERWAH Saving record OPW22/0487, 3 Greber Rd BEERWAH Saving record RAL22/0104, Bruce Hwy PALMVIEW Saving record OPW22/0488, Bruce Hwy PALMVIEW Saving record OPW22/0489, 18 Silky Oak Ct NINDERRY Saving record MCU22/0306, 111 Sunridge Rd EUDLO Saving record OPW22/0490, 111 Sunridge Rd EUDLO Saving record 1999/510068.01, Red Road Beerburrum Saving record MCU22/0308, 85 Memorial Drive EUMUNDI Saving record RAL21/0102.01, 55A Arthur St CALOUNDRA Saving record RAL22/0105, FASTTRACK 426 Glenview Rd GLENVIEW Saving record OPW22/0491, 426 Glenview Rd GLENVIEW Saving record OPW22/0492, 341 Diddillibah Road Diddillibah Saving record OPW22/0493, 33 Harbourlights Way PELICAN WATERS

Data

Downloaded 12308 times by slav123 openaustralia LoveMyData dcalde DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas yourneighbourhood ajoh3691 Jedwards157 EMBLAPROJECTS Corris224

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.67 MB) Use the API

rows 10 / 10064

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 30 days

Total cpu time used: about 1 hour

Total disk space used: 3.7 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    56 records added, 56 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    74 records added, 59 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    66 records added, 66 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 73 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 73 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb