planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching json 2.2.0  Fetching mime-types-data 3.2018.0812  Installing json 2.2.0 with native extensions  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (29.67s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record OPW19/0667, 1 Plimsol Ct WURTULLA Saving record POS19/0308, 1, 2, 3 & 4 / 22 Queen of Colonies Parade Moffat Beach Saving record MPC11/0004.13, 21 Tailside Cres BOKARINA Saving record POS19/0267, 107A King Street Buderim Saving record MCU17/2061.03, Botanica Dr SIPPY DOWNS Saving record MCU17/0040.06, 32 Baden Powell St MAROOCHYDORE Saving record OPW19/0672, 195 Parklands Blvd LITTLE MOUNTAIN Saving record OPW19/0668, 2 Caloundra St and 2 Baldwin St CALOUNDRA Saving record MCU16/0205.01, 2 Old Gympie Rd YANDINA Saving record MCU15/0262.01, 193 Mooloolaba Rd BUDERIM Saving record MCU03/0042.01, Chancellor Park Tavern 20 Chancellor Village Bvd SIPPY DOWNS Saving record MCU19/0316, 302 Bridge Creek Rd MALENY Saving record OPW19/0670, FAST TRACK 1 McKinnon Dr BURNSIDE Saving record OPW19/0669, 42 Bahdilli Cres DIDDILLIBAH Saving record RAL19/0131, FAST TRACK 18 Sutherland St BUDERIM Saving record MCU19/0315, FAST TRACK 1 McKinnon Dr BURNSIDE Saving record POS19/0259, Tide Residences Stage 2 Saving record POS19/0298, 8 Glenalpin Court Buderim Saving record OPW19/0674, Kirra Rd, Halcyon Dr, Parklakes Dr, Kingfisher Dr BLI BLI Saving record MCU19/0318, 51 Zgrajewski Rd YANDINA CREEK Saving record RAL19/0133, 13 Reflection Ct NAMBOUR Saving record OPW19/0675, 13 Reflection Ct NAMBOUR Saving record OPW19/0671, 14 Washington St NAMBOUR Saving record MCU19/0317, 28 Ackerman Rd KULANGOOR Saving record RAL19/0132, 1-59 Vise Rd and 759 Old Maroochydore Rd BUDERIM Saving record OPW19/0673, 563 Petrie Creek Rd ROSEMOUNT Saving record POS19/0306, 16 Wilson Avenue Dicky Beach Saving record OPW19/0677, 8 Mark St BUDERIM Saving record OPW19/0676, 8 Richards St MAROOCHYDORE Saving record MCU14/0142.03, Saffire Mooloolaba Beach 16 Marjorie St MOOLOOLABA Saving record OPW19/0678, 44-48 and 54 Savilles Rd and 35 Creekside Cct HIGHWORTH Saving record MCU17/2061.04, Botanica Dr SIPPY DOWNS Saving record MCU19/0320, 226-228 Balmoral Rd MONTVILLE Saving record OPW17/2424.03, 121 Bli Bli Rd BLI BLI Saving record OPW19/0680, 52 Shaws Rd BEERWAH Saving record OPW19/0679, Lot 429 Imagination Dr NAMBOUR Saving record OPW19/0683, 5 Hughes Pl VALDORA Saving record OPW17/2264.01, 5 Pavilion Dr PEREGIAN SPRINGS Saving record OPW19/0681, 82-84 Dales Rd CHEVALLUM Saving record MCU19/0321, 3 and 15 Tectonic Cres KUNDA PARK Saving record MCU19/0319, 5 Hughes Pl VALDORA Saving record OPW19/0682, 3 and 15 Tectonic Cres KUNDA PARK Saving record MCU14/0098.03, Steve Irwin Way GLENVIEW Saving record OPW19/0685, 25-27 Geo Hawkins Cres BELLS CREEK Saving record MCU09/0015.04, 5 Pavillion Dr PEREGIAN SPRINGS Saving record OPW19/0684, 7 William St LANDSBOROUGH Saving record OPW19/0686, 1 Coghlan Ct MAROOCHYDORE Saving record MCU15/0270.04, 24 and 26 Box St BUDERIM Saving record RAL18/0186.01, Laxton Rd PALMVIEW Saving record OPW19/0688, Laxton Rd PALMVIEW Saving record MCU12/0188.01, St Andrews Anglican College 40 Peregian Springs Dr PEREGIAN SPRINGS Saving record MCU08/0152.07, 916 David Low Way MARCOOLA Saving record OPW19/0687, 3-31 Abbotts Rd PALMWOODS Saving record MCU19/0323, 108-112 Quanda Rd COOLUM BEACH Saving record OPW19/0689, 108-112 Quanda Rd COOLUM BEACH Saving record MCU15/0139.02, 42-60 Menzies Drive, PACIFIC PARADISE Saving record MCU19/0322, 8/1 Tolman Court Maroochydore Saving record OPW19/0691, 24 Cocas Court Doonan Saving record OPW19/0690, 39-41 Monarch Place Mons Saving record MCU19/0324, 29 Mooloolah Island Minyama Saving record MCU13/0023.01, 5 Pioneer Rd YANDINA Saving record MCU13/0096.02, Lot 5 Woodlands Blvd MERIDAN PLAINS Saving record MCU16/0201.05, 96 & 98 Burnett Street Buderim error to parse council_reference's address: ToAdviseMCU00 Saving record MCU19/0325, Units 2 & 14, 316-318 Alexandra Parade Alexandra Headland Saving record OPW19/0692, 16 Hocking Street Nambour Saving record MCU19/0326, 71-75 Mill Hill Road Montville Saving record OPW19/0244.01, 18 Elizabeth Street BUDERIM

Data

Downloaded 6424 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas ajoh3691 Jedwards157

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.53 MB) Use the API

rows 10 / 3967

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 1.56 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    67 records added, 67 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    82 records added, 66 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 73 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    94 records added, 84 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    101 records added, 101 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb