planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (26.45s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record OPW20/0627, 96 Crohamhurst Rd CROHAMHURST Saving record POS20/0301, Stage 3 Saving record OPW20/0629, 8 Village Pl BUDERIM Saving record REC14/0110.03, 3/84 Taylors Road TANAWHA Saving record POS20/0300, 5 William Street Alexandra Headland Saving record MCU20/0330, Chevallum Industrial Park 19 Kikuyu Rd CHEVALLUM Saving record MCU19/0003.01, Mooloolah Community Centre 43 Bray Rd MOOLOOLAH VALLEY Saving record OPW20/0630, Chevallum Industrial Park 19 Kikuyu Rd CHEVALLUM Saving record MCU20/0329, 168 Hillcrest Rd KIAMBA Saving record RAL18/0164.01, 91 and 97A Memorial Dr & 1 Etheridge St EUMUNDI Saving record OPW20/0631, 1 Clithero Ave BUDERIM Saving record MCU03/0042.03, Chancellor Park Tavern 20 Chancellor Village Bvd SIPPY DOWNS Saving record OPW20/0632, 1396 Eumundi Noosa Rd EUMUNDI Saving record MCU19/0246.01, 247 Dales Rd CHEVALLUM Saving record POS20/0294, 22 Mount Pleasant Road Nambour Saving record POS20/0307, 1 Tamarind Court Woombye Saving record OPW20/0634, 115 Eudlo Flats Rd DIDDILLIBAH Saving record OPW20/0635, 39 Buderim Ave MOOLOOLABA Saving record MCU08/0176.10, Vantage 1 Vantage Dr YAROOMBA Saving record MCU17/2092.02, 18 Doolan St NAMBOUR Saving record MCU17/2153.04, St Andrews Anglican College 40 Peregian Springs Dr PEREGIAN SPRINGS Saving record POS20/0297, William Street Coolum Beach Saving record OPW20/0633, 5 Meadow Ct DOONAN Saving record MCU20/0331, 1-11 Hinkabooma Lane GHEERULLA Saving record POS20/0298, William Street Coolum Beach Saving record OPW18/0716.01, 247 Dales Rd CHEVALLUM Saving record REC06/0078.11, The Pavilion Peregian 6 Pavilion Dr PEREGIAN SPRINGS Saving record OPW20/0641, 8 and 10 Clifton Pl EUMUNDI Saving record REC16/0038.01, 145 Ninderry Rd NINDERRY Saving record RAL18/0009.09, Pelican Waters Southern Lakes Stage 4B, 4C(1), 4C(2), 4D(1), 4D(2), 4E, 4F, 4G, 4H(1), 4H(2), 4H(3) & 4I Saving record OPW20/0637, 9 Regent Pl BURNSIDE Saving record OPW20/0638, 23 Ronald Ct PEACHESTER Saving record MCU03/0094.04, Park Avenue Residences 24/66 The Avenue PEREGIAN SPRINGS Saving record MCU20/0332, 189 Brisbane Rd MOOLOOLABA Saving record MCU20/0333, 99 Lower Mount Mellum Rd LANDSBOROUGH Saving record OPW20/0636, 1286 Steve Irwin Way BEERWAH Saving record MCU20/0006.01, 1-3 and 5 The Esplanade MAROOCHYDORE Saving record REC17/0019.04, 27 Greenock Pl KIELS MOUNTAIN Saving record REC16/0069.46, 34 Banfield Pl MAROOCHYDORE Saving record OPW20/0639, 96 Marina View Dr PELICAN WATERS Saving record MCU06/0102.04, 10 and 40 Peregian Springs Drive Peregian Springs Saving record OPW20/0644, BP Service Station Kawana 22 Wyanda Dr BOKARINA Saving record POS20/0302, 39 Woombye-Palmwoods Road Woombye Saving record MCU20/0334, 64 Yvonne St HIGHWORTH Saving record MCU20/0335, Maroochy Sands 11/110 Sixth Ave MAROOCHYDORE Saving record RAL17/2034.06, 14 Amber Pl PALMVIEW Saving record OPW20/0640, 54 Elizabeth St NAMBOUR Saving record OPW20/0642, 2 Akeringa Pl MOOLOOLABA Saving record OPW20/0643, 405 Mooloolaba Rd BUDERIM Saving record MCU20/0336, 222-226 Quanda Rd COOLUM BEACH Saving record OPW20/0339.01, 82 Steggalls Rd YANDINA Saving record OPW17/0307.05, Pelican Waters Southern Lakes Stages 4 and 5 Saving record OPW20/0645, 44 Carwoola Cres Mooloolaba Saving record MCU19/0329.01, 1820 David Low Way COOLUM BEACH Saving record OPW20/0646, Flintwood Cres & 22 Twlight Avenue Palmview Saving record MCU20/0337, 44 Carwoola Cres Mooloolaba Saving record OPW20/0650, 1137 Nambour Connection Road Parklands Saving record OPW19/0650.01, 3293 Old Gympie Rd LANDSBOROUGH Saving record MCU20/0338, 310 University Way Sippy Downs Saving record OPW20/0647, 14 Rifle Range Rd NAMBOUR Saving record OPW20/0648, 220 Petrie Creek Road Rosemount Saving record OPW20/0649, 34 Woorilla Cres Mountain Creek Saving record RAL20/0141, Spitfire Banks Drive & Coral Sea Drive Pelican Waters Saving record POS20/0303, 143 and 165 Old Palmwoods Rd WEST WOOMBYE Saving record REC16/0085.04, 47 Petrie Creek Road ROSEMOUNT Saving record REC15/0140.02, 8 Carol Court Buderim Saving record MCU18/0149.02, 668 Bli Bli Rd NAMBOUR Saving record RAL18/0009.10, MacLamond Saving record MCU18/0112.01, 54 School Rd BLI BLI Saving record REC07/0016.23, 41/66 The Avenue PEREGIAN SPRINGS Saving record REC06/0019.11, Duke Rd Doonan Saving record OPW19/0152.02, Harmony District Collector Road Phase 4 Saving record OPW20/0651, Operational Works Application Saving record MPC20/0007, Kawana Way Birtinya

Data

Downloaded 8591 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb yourneighbourhood dcalde equivalentideas Jedwards157 EMBLAPROJECTS Corris224 ajoh3691

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.27 MB) Use the API

rows 10 / 6088

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 2.3 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    74 records added, 60 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    69 records added, 61 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    68 records added, 60 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    79 records added, 79 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    84 records added, 84 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb