planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (29.93s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record POS19/0179, 239 Tanawha Tourist Drive Tanawha Saving record REC03/0144.21, 158 Balgownie Dr PEREGIAN SPRINGS Saving record POS19/0184, 11 Ridge Road Maroochydore Saving record MCU09/0025.05, 66 & 110 Cooney Rd Bli Bli and State Forest Reserve Nambour Connection Rd BLI BLI Saving record OPW19/0424, 3 Mountain View Rd BALD KNOB Saving record MCU19/0192, 35 Watson St CURRIMUNDI Saving record MCU19/0191, Lot 34 Citron Place PALMWOODS Saving record MCU19/0190, 64 Anzac Ave MAROOCHYDORE Saving record POS19/0180, 41 & 45 King Street Buderim Saving record POS19/0187, 2 Hargreaves Street Currimundi Saving record MCU16/0033.03, 19 Coral St MALENY Saving record POS19/0186, 12 Grove Boulevard Mooloolah Valley Saving record OPW13/0762.03, David Low Way YAROOMBA Saving record OPW19/0428, 48 Lysaght St COOLUM BEACH Saving record MCU19/0194, 35 Citron Pl PALMWOODS Saving record OPW19/0426, 2 Caloundra Rd and Aldi 2 Baldwin St CALOUNDRA Saving record OPW19/0425, 2 Caloundra Rd and Aldi 2 Baldwin St CALOUNDRA Saving record OPW17/2421.02, 65-71 Coolum Tce, 1718-1720, 1722 David Low Way COOLUM BEACH Saving record MCU19/0193, Citron Pl PALMWOODS Saving record OPW19/0427, 26 Mt Pleasant Road Nambour Saving record REC14/0159.03, 32 Citron Pl PALMWOODS Saving record MCU19/0195, 48 Lysaght St COOLUM BEACH Saving record REC16/0004.02, 31 Sippy Downs Dr SIPPY DOWNS Saving record OPW19/0429, 137 Jones Road Buderim Saving record OPW17/2040.01, 137 Jones Road BUDERIM Saving record MCU19/0198, Ocean Dr TWIN WATERS Saving record MCU19/0197, 396 Western Ave MONTVILLE Saving record MCU19/0196, 1 Steggalls Rd YANDINA Saving record MCU18/0308.01, 22-24 Industrial Pl YANDINA Saving record MCU14/0187.01, 27 Mayfield St ALEXANDRA HEADLAND Saving record POS19/0178, 13 Dulin Street Maroochydore Saving record RAL19/0095, FAST TRACK 681 and 725 Hunchy Rd HUNCHY Saving record OPW19/0432, 197 Winston Rd South CHEVALLUM Saving record OPW19/0430, 12 First Ave COOLUM BEACH Saving record RAL19/0093, 89 and lot 4 Railway Pde GLASSHOUSE MOUNTAINS Saving record MCU17/0040.04, Baden Powell St MAROOCHYDORE Saving record REC09/0129.03, David Low Way YAROOMBA Saving record RAL19/0096, 166 and 240 Browns Rd KIDAMAN CREEK Saving record RAL19/0094, 4 Mount Pleasant Rd NAMBOUR Saving record OPW19/0435, 56-62 Whites Rd BUDERIM Saving record MCU19/0201, 110 Delicia Rd MAPLETON Saving record MCU19/0200, 20-22 Geo Hawkins Cres CORBOULD PARK Saving record MCU19/0199, 185 Parklakes Dr BLI BLI Saving record OPW19/0433, 20-24 Whyandra Close Doonan Saving record 2003/56M0010.01, 20 Gipps St CALOUNDRA WEST Saving record 2004/510019.03, Nicklin Way Parreara Saving record OPW19/0434, 20-22 Geo Hawkins Cres CORBOULD PARK Saving record MCU18/0234.01, 57 and 63 Owen Creek Rd FOREST GLEN Saving record OPW19/0436, 100 Duporth Ave MAROOCHYDORE Saving record OPW19/0431, 12 Carwoola Crescent Mooloolaba Saving record OPW19/0437, 50 Leigha Pl KUREELPA Saving record MPC16/0004.06, Lot 903 Bright Pl BIRTINYA Saving record MCU17/0146.01, 54 Junction Drive COOLUM BEACH Saving record OPW18/0727.02, 56-62 Whites Rd BUDERIM Saving record RAL18/0078.03, 36 Flintwood Cres PALMVIEW Saving record RAL19/0097, 9 Churchill St PALMWOODS Saving record OPW19/0438, 65 Dunning St PALMWOODS Saving record REC15/0196.04, 17 Chambers Ct GLENVIEW Saving record OPW19/0439, 16 Dulin St MAROOCHYDORE Saving record MCU19/0203, 16 Bulcock St CALOUNDRA Saving record RAL19/0098, FAST TRACK 616 Petrie Creek Rd ROSEMOUNT Saving record MCU19/0202, 81 Balmoral Rd MONTVILLE Saving record OPW19/0441, 100 Duporth Ave MAROOCHYDORE Saving record OPW19/0440, 37 Eckersley Ave BUDERIM Saving record MCU11/0208.03, 1 Development Crt CALOUNDRA WEST Saving record MCU17/2007.01, Old Peachester Rd PEACHESTER Saving record MCU17/0107.03, 31 Claude Boyd Parade BELLS CREEK Saving record 1989/10068.02, 174/2 Melody Ct WARANA Saving record RAL19/0099, Lake Kawana Bvd BIRTINYA Saving record REC01/0081.02, 46 Prestwick Dr TWIN WATERS Saving record 2000/510045.01, Australia Zoo 41 Fraser Rd BEERWAH Saving record RAL17/2026.02, 170 King Rd MOOLOOLAH VALLEY Saving record OPW19/0443, 56-62 Whites Rd BUDERIM Saving record MCU16/0119.01, 35 Edward St ALEXANDRA HEADLAND Saving record RAL18/0085.03, 16 Silverleaf St PALMVIEW Saving record OPW19/0442, FAST TRACK 53-55 Lysaght St COOLUM BEACH Saving record MCU19/0205, FAST TRACK 53-55 Lysaght St COOLUM BEACH Saving record RAL19/0100, 86 Countryview Street Woombye Saving record REC04/0194.01, 16 Clearwater Cct BLI BLI -Generally in Accordance Reconfig (Major Urban Subdiv x 32 Lots) Saving record OPW19/0446, 50 Cooloolabin Rd YANDINA error to parse council_reference's address: OPW19/0445 Saving record RAL19/0008.03, 34 Tranquility Way PALMVIEW Saving record OPW19/0444, 937 Maleny Stanley River Rd BOOROOBIN

Data

Downloaded 5719 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas Jedwards157 ajoh3691

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.28 MB) Use the API

rows 10 / 3238

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 1.31 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    82 records added, 82 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    82 records added, 73 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    80 records added, 71 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    83 records added, 64 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    71 records added, 71 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb