planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.17s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record 2002/520039.37, 32 Kookaburra Drive PALMVIEW- Change to Condition Saving record OPW20/0436.01, 83 Parker St MAROOCHYDORE Saving record OPW21/0074, 70-82 Crescent Rd EUMUNDI Saving record MCU21/0037, FAST TRACK 35 Steggalls Rd YANDINA Saving record MCU21/0038, 125 Jacksons Rd WEST WOOMBYE Saving record MCU18/0336.01, 918 David Low Way MARCOOLA Saving record OPW21/0076, Mantra Esp BIRTINYA Saving record OPW21/0077, 599 Blackall Range Road WOOMBYE Saving record OPW21/0075, FAST TRACK 35 Steggalls Rd YANDINA Saving record OPW18/0148.01, 43-45 Primary School Ct MAROOCHYDORE Saving record POS21/0043, 10 Honeysuckle Lane Maleny Saving record POS21/0036, Harmony DBPC (Dedicated Bicycle & Pedestrian Corridor) Saving record POS21/0044, 235 Racecourse Road Corbould Park Saving record OPW21/0080, 14 Oramba St CURRIMUNDI Saving record MCU21/0040, Lot 41 & 42 Green Turtle Place MARCOOLA Saving record OPW21/0078, 17 Royal Close WURTULLA Saving record OPW21/0079, 13 Petrel Cres MOUNTAIN CREEK Saving record MCU21/0039, 23-29 Nature Place BUDERIM Saving record MCU17/0010.01, 30 Access Cres COOLUM BEACH Saving record OPW21/0081, 3 & 5 Lang St COOLUM BEACH Saving record RAL20/0096.01, Flintwood Cres & 22 Twilight Ave PALMVIEW Saving record OPW21/0082, 6 Bridie Place MOUNTAIN CREEK Saving record MCU17/0071.02, Unit 6 & 7 44 Lysaght St COOLUM BEACH Saving record REC02/0143.01, 5 Warrener Place MAROOCHY RIVER Saving record OPW21/0083, 19 Pine Edge Dr LANDSBOROUGH Saving record POS21/0047, 32 Plantation Circuit Landsborough Saving record OPW20/0159.01, 34 Millwell Rd MAROOCHYDORE Saving record OPW21/0084, 192 Dulong Rd DULONG Saving record MCU18/0234.03, 57 & 63 Owen Creek Rd FOREST GLEN Saving record 2001/570010.07, Unit 404 13 Nicklin Way MINYAMA Saving record RAL21/0013, 89 Whites Rd BUDERIM Saving record REC12/0108.04, 12 and 14 Hideaway Lane GLENVIEW Saving record MCU07/0129.02, 22 Palmwoods Montville Road Palmwoods Saving record 2002/520056.01, 27 Peony Cct LITTLE MOUNTAIN Saving record MCU21/0041, 21 South Coolum Rd COOLUM BEACH Saving record OPW19/0589.03, Pelican Waters Southern Lakes Stages 4D(1), 4D(2) & 4F Saving record OPW21/0085, Coral Sea Dr PELICAN WATERS Saving record OPW13/0177.03, 22 Palmwoods-Montville Rd PALMWOODS Saving record OPW21/0087, 1-7 Island View Ct BUDERIM Saving record REC17/0010.01, 33 William St BUDERIM Saving record MCU10/2108.02, 21 Whalley Creek Cl Burnside Saving record MCU20/0156.01, 39 Allen St MOFFAT BEACH Saving record OPW21/0086, Skyline Cct PALMWOODS Saving record RAL18/0069.02, 149 Kirra Rd MAROOCHY RIVER Saving record OPW21/0088, 13 Blackwood St MAROOCHYDORE Saving record OPW21/0092, Stringybark Rd SIPPY DOWNS Saving record MCU20/0277.01, 44 Lysaght St COOLUM BEACH Saving record REC08/0102.07, 21 & 27 Jenyor Street Coolum Beach Saving record OPW21/0093, Harmony Bvd PALMVIEW Saving record OPW21/0090, 40 Peregian Springs Dr PEREGIAN SPRINGS Saving record OPW21/0089, 10 Stephen Burton Way PELICAN WATERS Saving record RAL21/0014, Harmony Bvd PALMVIEW Saving record RAL21/0016, 19 Coonowrin Rd GLASS HOUSE MOUNTAINS Saving record OPW21/0094, 96 Bellflower Rd SIPPY DOWNS Saving record OPW21/0096, 121-125 Quanda Rd COOLUM BEACH Saving record REC15/0226.06, 44 and 42 Anning Rd FOREST GLEN Saving record MCU21/0042, 109-113 Quanda Rd COOLUM BEACH Saving record RAL21/0015, 47B King St SHELLY BEACH Saving record OPW21/0095, 82 Steggalls Rd YANDINA Saving record MCU21/0043, 121-125 Quanda Rd COOLUM BEACH Saving record MCU21/0045, 10 Goongilla St YAROOMBA Saving record OPW21/0098, 10 Goongilla St YAROOMBA Saving record OPW21/0099, 6 Marina Wk ALEXANDRA HEADLAND Saving record MCU21/0044, 17 First Ave COOLUM BEACH Saving record OPW21/0101, 145 Palmwoods-Montville Rd, 38 and 50 Landershute Rd PALMWOODS Saving record RAL18/0200.03, 83 Cutters Way BLI BLI Saving record MCU10/0034.02, 16 Briggs St PALMWOODS Saving record RAL21/0017, 8 Clifton Pl EUMUNDI Saving record OPW20/0498.01, 56 Sugar Bag Rd LITTLE MOUNTAIN Saving record OPW21/0097, 17 First Ave COOLUM BEACH Saving record MCU19/0253.01, 35 and 37 Owen Creek Rd FOREST GLEN Saving record MCU00/0132.01, 61 Jubilee Dr PALMWOODS Saving record OPW21/0100, 28 Cash Rd EUMUNDI Saving record MCU21/0046, 6 Marina Wk ALEXANDRA HEADLAND

Data

Downloaded 8954 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly dcalde TheZepto spatialsteve dsccramb yourneighbourhood equivalentideas Jedwards157 EMBLAPROJECTS Corris224 ajoh3691

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.44 MB) Use the API

rows 10 / 6561

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 2.48 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    74 records added, 74 records removed in the database
  • Auto ran revision 6a6ca4ed and failed .
    nothing changed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    85 records added, 85 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    86 records added, 56 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    64 records added, 59 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb