planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.62s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record RAL20/0076.04, 4 Harmony Blvd PALMVIEW Saving record OPW21/0601, FASTTRACK 2 Finch Ct BOKARINA Saving record OPW21/0600, 12 Wyanda Dr BOKARINA Saving record MCU21/0378, 12 and 14 Wyanda Dr BOKARINA Saving record OPW21/0597, 6 Flinders Lane MAROOCHYDORE Saving record RAL21/0117, 41 Glenbrook Dr and 21 Strathford Ave NAMBOUR Saving record REC15/0226.07, 82 Anning Road FOREST GLEN Saving record OPW21/0596, 12 Oak Ct MINYAMA Saving record RAL20/0076.03, 6 Harmony Blvd PALMVIEW Saving record MCU21/0380, 2 Finch Ct BOKARINA Saving record OPW21/0598, 265 Diamond Valley Road DIAMOND VALLEY Saving record OPW21/0599, 6-8 Settlers Ridge BUDERIM Saving record MPC14/0027.07, Part of Lot 462 Innovation Pkwy BIRTINYA Saving record POS21/0226, 9 Nichols Ave BEERWAH Saving record OPW21/0602, 10 Piat Place BEERWAH Saving record OPW21/0608, 119 Pioneer Rd BEERBURRUM Saving record MCU20/0257.01, 145 Palmwoods-Montville Rd, 38 and 50 Landershute Rd PALMWOODS Saving record MCU21/0379, 1, 3 and 5 Commercial Rd KULUIN Saving record OPW21/0606, 37 Upper Hutton Rd KIAMBA Saving record MCU21/0383, FASTTRACK 21 Whalley Creek Cl BURNSIDE Saving record OPW21/0607, 1 Eden St MINYAMA Saving record RAL20/0067.09, 29 Sitella Pl PALMVIEW Saving record OPW21/0605, 30 Cootamundra Drive MOUNTAIN CREEK Saving record MCU21/0382, 10 & 40 Peregian Springs Drive Peregian Springs Saving record MCU20/0151.01, 26 Mayfield Street Alexandra Headland Saving record OPW21/0609, FASTRACK 21 Whalley Creek Cl BURNSIDE Saving record OPW21/0603, 154-164 Horton Parade MAROOCHYDORE Saving record REC14/0038.05, 34 Heritage Place MONS Saving record OPW21/0604, 115-119 Quanda Road COOLUM BEACH Saving record MCU21/0381, 36 Roys Rd LANDERS SHOOT Saving record OPW21/0610, Bokarina Boulevard BIRTINYA Saving record POS21/0251, Stage 2 Saving record REC17/0026.17, 16-30 Eton Lane MAROOCHYDORE Saving record MCU21/0384, 780 Eumundi-Kenilworth Rd BELLI PARK Saving record RAL17/2032.11, 81 Palm Forest Cct FOREST GLEN Saving record RAL17/2032.12, 73 Palm Forest Cct FOREST GLEN Saving record POS21/0237, 68 Witta Rd and 266 Maleny Kenilworth Rd WITTA Saving record RAL17/2023.05, 14 Eagles Lane WOOMBYE Saving record REC13/0135.06, 11 Maracas Bay Cl YAROOMBA Saving record MCU21/0388, Future Lot 30 within 27 Corporate Pl LANDSBOROUGH Saving record MCU21/0387, Future Lot 16 within 27 Corporate Pl LANDSBOROUGH Saving record OPW21/0614, 7 Finley Rd EUMUNDI Saving record RAL19/0139.01, 55 Orme Rd BUDERIM Saving record POS21/0260, 403-413 Upper Rosemount Rd ROSEMOUNT Saving record MCU21/0390, 17 and 19 Bradman Ave MAROOCHYDORE Saving record OPW18/0328.04, Stage 3D Saving record OPW21/0613, Future Lot 30 within 27 Corporate Pl LANDSBOROUGH Saving record POS21/0258, 58, 60 & 70-72 Negus Road Montville Saving record OPW21/0611, Future Lot 16 within 27 Corporate Pl LANDSBOROUGH Saving record OPW21/0615, 185-205 Diddillibah Rd WOOMBYE Saving record MCU21/0385, St Andrews Anglican College 40 and 10 Peregian Springs Dr PEREGIAN SPRINGS Saving record 2001/570010.08, 404/13 Nicklin Way MINYAMA Saving record OPW21/0612, 7 Aldinga Pl MOOLOOLABA Saving record MCU21/0389, 23 Caroline Crescent BUDERIM Saving record POS21/0233, Pelican Waters Southern Lakes Stage 4E & 4G LEASES Saving record OPW21/0617, 7 Greenock Place KIELS MOUNTAIN Saving record POS21/0243, 80 & 76 Eudlo Rd MOOLOOLAH VALLEY Saving record RAL18/0009.15, 107 Bells Esp PELICAN WATERS Saving record OPW21/0616, 17 Buckle St YANDINA Saving record MCU17/0154.01, 58 Harmony Bvd PALMVIEW Saving record ToAdviseOPW00, Hillside Mooloolah Valley Stage 5 170 King Rd MOOLOOLAH VALLEY Saving record RAL21/0118, 17 Buckle St YANDINA Saving record RAL21/0119, 14 Karanne Dr MOOLOOLAH VALLEY Saving record OPW21/0618, 221 Storrs Rd PEACHESTER Saving record MCU16/0190.02, Honey Farm Rd MERIDAN PLAINS Saving record RAL17/2032.14, DH 83 Palm Forest Cct FOREST GLEN Saving record MCU21/0045.01, 10 Goongilla St YAROOMBA Saving record OPW21/0622, 67 Taylor Ave GOLDEN BEACH Saving record OPW21/0621, FASTRACK 19 Empire Cres CHEVALLUM Saving record RAL17/2032.13, DH 75 Palm Forest Cct FOREST GLEN Saving record RAL17/2032.15, DH 79 Palm Forest Cct FOREST GLEN Saving record OPW21/0619, 32 Wises Rd BUDERIM Saving record MCU16/0224.02, 7 Bradman Ave MAROOCHYDORE Saving record OPW21/0620, 35 Moondarra Cres MOOLOOLABA Saving record 2002/570012.09, 14 Crete Ct PARREARRA Saving record MCU21/0391, FASTRACK 19 Empire Cres CHEVALLUM Saving record POS21/0246, 16 Kuthar St PELICAN WATERS Saving record MCU21/0392, 55-59 Obi Vale NORTH MALENY Saving record MCU21/0010.01, 20 Beach Rd MAROOCHYDORE Saving record MCU21/0393, 14 Dray Pl PALMWOODS Saving record MCU08/0149.01, 1/5 Montage Court BUDERIM Saving record RAL20/0051.01, 7 Buzaki Rd GLASS HOUSE MOUNTAINS Saving record OPW21/0624, 543 Diddillibah Rd DIDDILLIBAH Saving record MCU14/0205.01, 3 Mari Street ALEXANDRA HEADLAND Saving record MCU21/0061.01, 88, 90 & 94 Duporth Avenue MAROOCHYDORE Saving record OPW21/0623, 125 Memorial Dr EUMUNDI Saving record POS21/0249, 51 Oogar Street Alexandra Headland Saving record REC02/0316.55, 119 The Avenue PEREGIAN SPRINGS Saving record MPC14/0028.06, 60 Prime Pde BIRTINYA Saving record MCU21/0394, 853 Browns Creek Rd EERWAH VALE Saving record MCU21/0395, FASTRACK 3A Lynd Ct BUDERIM Saving record OPW21/0625, 65 Mapleton Falls Rd MAPLETON Saving record OPW21/0626, 405 Mooloolaba Rd BUDERIM

Data

Downloaded 10209 times by slav123 openaustralia LoveMyData dcalde DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas yourneighbourhood ajoh3691 Jedwards157 EMBLAPROJECTS Corris224

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.99 MB) Use the API

rows 10 / 8154

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 3.02 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    93 records added, 88 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    97 records added, 90 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    128 records added, 128 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    128 records added, 118 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    129 records added, 129 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb