planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (27.87s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record MCU20/0249, 18 & 20 Wright Street Maroochydore Saving record OPW20/0474, 4 Spring Street SIPPY DOWNS Saving record OPW20/0472, 50 Silvertop Road DOONAN Saving record RAL20/0025.01, Village Green Palmview Stage 5 Saving record OPW20/0475, 28-40 Savilles Road NAMBOUR Saving record OPW20/0473, Laxton Road Palmview Saving record OPW20/0471, 91 and 97A Memorial Drive and 1 Etheridge Street Eumundi Saving record MCU20/0250, Chevallum Industrial Park 6 Empire Cres CHEVALLUM Saving record OPW20/0470, 96 Karawatha St BUDERIM Saving record MCU18/0028.01, 35 Dahlia Rd VERRIERDALE Saving record OPW20/0479, 41 Jessica Boulevard MINYAMA Saving record OPW20/0481, 2-4 Kate Court BEERWAH Saving record OPW20/0480, 67 Beerwah Parade Saving record MCU20/0012.01, 28 Saffron Drive CURRIMUNDI Saving record OPW20/0478, 47 Baanya Street WURTULLA Saving record POS20/0226, 8 Pandanus Street Mudjimba Saving record OPW19/0140.01, Peter Crosby Way PALMVIEW Saving record MCU20/0251, 18 Regent Pl BURNSIDE Saving record 2002/520039.35, 12 Rosella Lane PALMVIEW Saving record OPW20/0477, 113-121 Atkinson Rd Bli Bli Saving record MCU14/0040.02, 233 Maleny-Stanley River Rd WOOTHA Saving record OPW20/0484, 21 Harbourlights Way PELICAN WATERS Saving record OPW20/0482, 27 Inalls Road OBI OBI Saving record MCU20/0256, FASTRACK 6 Palm St NAMBOUR Saving record OPW20/0483, 433 Yandina-Bli Bli Road MAROOCHY RIVER Saving record MCU20/0253, FASTRACK 230 Godfreys Road PACIFIC PARADISE Saving record MCU20/0252, 11 Sydney Street NAMBOUR Saving record 2009/56M0015.01, 150 Fred Chaplin Circuit COURBOULD PARK Saving record MCU17/0071.01, 44 Lysaght St COOLUM BEACH Saving record MCU20/0254, 2/4 Lower Neill St DICKY BEACH Saving record OPW20/0486, Lot 1000 Village Green Bvd PALMVIEW Saving record RAL17/2089.01, 31 Cobbs Rd WOOMBYE Saving record MCU20/0255, FASTRACK 8 Honey Myrtle Pl WOOMBYE Saving record OPW20/0487, FASTRACK 8 Honey Myrtle Pl WOOMBYE Saving record RAL20/0106, FASTRACK 57 and 65 Regency Rd DOONAN Saving record 2002/520039.36, 15 Kookaburra Dr PALMVIEW Saving record MCU18/0310.03, 16 Grammar School Way FOREST GLEN Saving record MCU20/0257, 145 Palmwoods-Montville Rd, 38 and 50 Landershute Rd PALMWOODS Saving record OPW19/0589.02, Pelican Waters Southern Lakes Stages 4D(2) and 4F Saving record OPW20/0488, FASTRACK 6 Palm St NAMBOUR Saving record OPW20/0485, 11 Citrus Rd and 20 Rifle Range Rd PALMWOODS Saving record POS20/0223, 56 Abbotts Road Palmwoods Saving record POS20/0214, 8 Aragorn Street Maroochydore Saving record OPW20/0494, 58 Evans Grove Rd GLENVIEW Saving record OPW20/0489, 132 Whites Rd BUDERIM Saving record OPW20/0490, 78 Arthur St CALOUNDRA Saving record OPW20/0491, 953-971 David Low Way MARCOOLA Saving record OPW20/0492, 18 Regent Pl BURNSIDE Saving record OPW20/0493, 102 Hardwood Rd LANDSBOROUGH Saving record POS20/0213, 6 Aragorn Street Maroochydore Saving record POS20/0212, Revive Oceanside Stage 2 Saving record MCU20/0258, FASTRACK 53 Dacmar Rd COOLUM BEACH Saving record OPW20/0499, 2 Allonga St CURRIMUNDI Saving record MCU13/0192.04, 4 Saving record MCU20/0115.02, 10 Boorook St BUDDINA Saving record MCU20/0260, 85 Obi Obi Rd MAPLETON Saving record OPW20/0496, FASTRACK 79 Elizabeth St WARANA Saving record OPW20/0497, 46 Toulambi Drive BUDERIM Saving record OPW19/0212.01, 18-20, 22-24 and 26-28 Toral Dr BUDERIM Saving record OPW20/0495, FASTRACK 53 Dacmar Rd COOLUM BEACH Saving record OPW20/0498, 56 Sugar Bag Rd LITTLE MOUNTAIN Saving record REC15/0102.34, 10 Langsford Cres MAROOCHYDORE Saving record MCU20/0259, FASTRACK 79 Elizabeth St NAMBOUR Saving record OPW20/0500, 11 Dinmore St WOOMBYE Saving record RAL18/0009.06, 82 Marina View Dr PELICAN WATERS Saving record RAL20/0067.01, Harmony Release 14 Saving record RAL20/0107, FASTRACK 60 Towen Mount Rd TOWEN MOUNTAIN Saving record SPS20/0004, 42 and 44 Gloucester Rd BUDERIM Saving record SPS20/0005, Green Turtle Pl MARCOOLA Saving record MCU20/0261, 700 Bli Bli Rd NAMBOUR Saving record OPW20/0501, 62 Lamerough Pde PELICAN WATERS Saving record 2006/520014.08, 14 Foreshore Ct DICKY BEACH Saving record OPW20/0503, 19 Mudjimba Beach Rd MUDJIMBA Saving record OPW20/0502, 363 Duke Rd DOONAN Saving record OPW20/0504, 28-44 Glenmount Rd MONS Saving record MCU20/0263, 89-97 Currie St NAMBOUR Saving record 2004/520007.31, 11 The Passage PELICAN WATERS Saving record OPW20/0505, 28-30 Balkin Rd EUMUNDI Saving record OPW19/0567.01, 101 Thomson Valley Rd VALDORA -change to Earthworks Saving record MCU20/0262, 9 Cherry St MALENY Saving record MCU20/0266, 155 Clarendon Road Peregian Beach Saving record RAL20/0108, 25 Fig Tree Drive Beerwah Saving record OPW20/0509, 1-3 and 5 The Esplanade MAROOCHYDORE Saving record OPW20/0506, 25 Fig Tree Drive Beerwah Saving record MCU20/0264, 13 & 22 Peachester Road Beerwah Saving record OPW20/0508, 45 Coomaroo Crescent Minyama Saving record MCU20/0265, 51 Reids Road West Woombye Saving record OPW20/0507, 13 and 15 Rigby St NAMBOUR

Data

Downloaded 8129 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb yourneighbourhood equivalentideas Jedwards157 EMBLAPROJECTS Corris224 ajoh3691

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.1 MB) Use the API

rows 10 / 5607

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 2.13 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    88 records added, 77 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    89 records added, 89 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    93 records added, 93 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    93 records added, 93 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    94 records added, 74 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb