planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (28.67s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record MCU18/0111.01, 21 & 19 Bunya St MALENY Saving record OPW20/0311, 60 Redwood Rd DOONAN Saving record RAL20/0076, Tranquility Way PALMVIEW Saving record RAL20/0075, 281 and 325 Image Flat Rd IMAGE FLAT Saving record OPW20/0313, Tranquility Way PALMVIEW Saving record RAL18/0148.07, 27 Greenmount Dr PALMVIEW Saving record POS20/0146, 52A Glouceser Road Buderim Saving record POS20/0149, Stage 4B Saving record OPW20/0312, 34 Thompson Rd BEERWAH Saving record OPW20/0315, 864 Yandina Bli Bli Rd BLI BLI Saving record MCU19/0019.01, 165 Oceanic Dve WARANA Saving record MCU19/0024.01, 103 Oceanic Drive WARANA Saving record RAL18/0063.01, 23 Solandra St and Lot 1 Parsons St NAMBOUR Saving record REC17/0026.04, Sunrise Dr MAROOCHYDORE Saving record RAL20/0079, 13-15 Upper Gay Tce KINGS BEACH Saving record OPW20/0318, 1 Affinity Pl BIRTINYA Saving record MCU20/0181, 13 Teak St MALENY Saving record RAL20/0080, 26 Moorhouse Rd and 350 Nambour Connection Rd WOOMBYE Saving record POS20/0153, 4 Cooma Terrace Caloundra Saving record POS20/0151, Stage 1 Saving record RAL20/0077, Lot 500 Taintons Rd WOOMBYE Saving record OPW20/0316, Lot 500 Taintons Rd WOOMBYE Saving record RAL20/0078, 65 William St BUDERIM Saving record MCU09/0077.04, 64 Burnett St BUDERIM Saving record OPW20/0317, 65 William St BUDERIM Saving record MCU12/0173.02, 2 Malones Rd KIELS MOUNTAIN Saving record MCU15/0129.04, 84 & 86 Caloundra Road Little Mountain Saving record MCU04/0219.01, Units 63 and 65/62-66 Sixth Ave MAROOCHYDORE Saving record MCU20/0180, 1 Affinity Pl BIRTINYA Saving record POS20/0144, 35 Jenyor Street Coolum Beach Saving record OPW20/0320, 26 Saving record MCU20/0182, 980 Nambour Connection Rd NAMBOUR Saving record POS20/0145, 28 Horizon Way Woombye Saving record REC11/0127.03, 66 Taintons Rd WOOMBYE Saving record OPW20/0319, 38-68 Balmoral Rd MONTVILLE Saving record MCU20/0183, 3 Tagera St WARANA Saving record OPW20/0323, 2B Page St KUNDA PARK Saving record OPW20/0321, 4 Myoora Ct MINYAMA Saving record OPW20/0322, 3 Tagera St WARANA Saving record EXE20/0002, 1/73 Parkyn Pde MOOLOOLABA Saving record OPW20/0324, 7 Treetop Pl LITTLE MOUNTAIN Saving record REC15/0205.02, 10A Dress Circle Ct BUDERIM Saving record MCU20/0184, 2B Page St KUNDA PARK Saving record MCU20/0185, 22 Swan St BEERWAH Saving record OPW20/0328, Sedge St and Central Dr SIPPY DOWNS Saving record MPC11/0004.19, 24 Tailslide Cres BOKARINA Saving record MCU20/0187, Sedge St and Central Dr SIPPY DOWNS Saving record OPW20/0327, 16 Nunkeri Ct FOREST GLEN Saving record OPW20/0326, 1 Beerburrum St LANDSBOROUGH Saving record RAL20/0081, Sedge St and Central Dr SIPPY DOWNS Saving record MCU20/0186, 41 Vantage Dr YAROOMBA Saving record MCU18/0319.01, 3 Nagari Pl WARANA Saving record C937095.01, 274 Tanawha Tourist Dr TANAWHA Saving record OPW20/0325, 5 Ormonde Tce KINGS BEACH Saving record OPW20/0329, 23A Nandewar Dr BUDERIM Saving record MCU20/0188, 36 Egret Ave WOOMBYE Saving record RAL18/0173.02, Panorama Estate Saving record OPW20/0330, 384 Mawsons Rd BEERWAH Saving record MCU20/0190, 23 Roderick St MOFFAT BEACH Saving record MCU11/0039.02, 2 Link Cres COOLUM BEACH Saving record OPW20/0331, 17 Durham Place GLENVIEW Saving record MCU20/0191, 8 Knolls Ct MALENY Saving record OPW20/0333, 322 Mossy Bank Rd EUDLO Saving record OPW20/0335, 13 Evergreen Way BEERWAH Saving record MCU08/0055.05, 46 Paulger Flat Rd YANDINA Saving record 1994/10021.01, 14/28 Pelican Waters Blvd PELICAN WATERS Saving record OPW20/0332, Central Dr SIPPY DOWNS Saving record OPW20/0334, 14 Jarrah Way LANDSBOROUGH Saving record MCU15/0263.01, 20 Kondalilla Falls Road MONTVILLE Saving record OPW20/0336, 17 Bangalow St MINYAMA Saving record CDA06/0044, Cilento Saving record MCU20/0192, Central Dr SIPPY DOWNS Saving record OPW20/0339, 82 Steggalls Rd YANDINA Saving record OPW20/0337, 6-8 Bond Ct DOONAN Saving record C977493.05, 30 Moorings Cct TWIN WATERS Saving record MCU17/2108.02, 118 Alexandra Pde ALEXANDRA HEADLAND error to parse council_reference's address: OPW20/0340 Saving record OPW20/0338, 167 Sippy Creek Rd TANAWHA

Data

Downloaded 7672 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas ajoh3691 Jedwards157 EMBLAPROJECTS Corris224

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.94 MB) Use the API

rows 10 / 5127

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 1.97 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    77 records added, 77 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    77 records added, 77 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    77 records added, 69 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    81 records added, 81 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    92 records added, 78 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb