planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.............  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching json 2.2.0  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Installing json 2.2.0 with native extensions  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (27.95s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record POS19/0231, 19 Kurrajong Circuit Peregian Springs Saving record REC16/0069.32, 9 Tully Lane MAROOCHYDORE Saving record OPW20/0002, 28 Willis Rd BLI BLI Saving record OPW20/0004, 1 Beryl Tce MOUNT COOLUM Saving record MCU20/0002, 8 Petrie Ave MARCOOLA Saving record OPW20/0003, 10 Brae Crt BUDERIM Saving record OPW20/0006, Lot 800 Racecourse Rd CORBOULD PARK Saving record OPW20/0005, 455-463 Paynters Creek Rd ROSEMOUNT Saving record MPC16/0005.02, Part of Lot 1000 Birtinya Bvd BIRTINYA QLD Saving record MPC14/0027.04, Mantra Esplanade, BIRTINYA Saving record OPW20/0008, 405 Mooloolaba Rd BUDERIM Saving record OPW20/0007, 141 Jones Rd BUDERIM Saving record MCU20/0003, 5 Currawong St MUDJIMBA Saving record RAL20/0001, 143 and 165 Old Palmwoods Rd WEST WOOMBYE Saving record MCU20/0004, 7 Meditation Ct NAMBOUR error to parse council_reference's address: OPW20/0009 Saving record REC15/0209.02, 7 Paynter Park Drive WOOMBYE Saving record RAL20/0002, 774 Eudlo Rd PALMWOODS Saving record MCU10/2108.01, 21 Whalley Creek Cl BURNSIDE Saving record MCU08/0016.01, 3/26-28 Link Cres COOLUM BEACH Saving record POS19/0327, 40-42 Junction Drive Coolum Beach Saving record OPW16/0273.02, 1 Vantage Drive YAROOMBA Saving record MCU20/0005, 4 Honey Myrtle Pl WOOMBYE Saving record OPW20/0010, 6 Berrima St MAROOCHYDORE Saving record OPW20/0012, 20 Second Ave MAROOCHYDORE Saving record REC13/0157.10, 1-13 Flinders Lane Maroochydore Saving record SPS20/0001, 56 Bonato Rd GLASS HOUSE MOUNTAINS Saving record MCU18/0310.02, 134-158 Parsons Rd and 286 Mons Rd FOREST GLEN Saving record OPW20/0011, 26 Gattera Road Landsborough Saving record MCU20/0006, 1-3 and 5 The Esplanade MAROOCHYDORE Saving record RAL20/0003, 104 Obi Obi Rd and 31 Sommer Rd MAPLETON Saving record OPW20/0013, Service St MAROOCHYDORE Saving record MCU20/0007, Service St MAROOCHYDORE Saving record MCU20/0008, 53 Quanda Rd COOLUM BEACH Saving record MCU15/0305.04, 96 Memorial Ave MAROOCHYDORE- Change to Condition Saving record OPW20/0015, 20 Burys Rd BEERWAH Saving record OPW20/0017, 56-62 Whites Rd BUDERIM Saving record REC17/0026.03, Sunshine Cove Precinct 6 Saving record OPW20/0016, 3 Karnu Dr NINDERRY Saving record OPW20/0014, 22 Myoora Ct MINYAMA Saving record RAL20/0004, Sedge St SIPPY DOWNS Saving record OPW20/0021, 375 Kiamba Rd COOLOOLABIN Saving record REC16/0069.33, 14 Banfield St, MAROOCHYDORE Saving record REC15/0060.03, 1-7 Dixon Rd Buderim Saving record RAL20/0005, 163 and 201 Windsor Rd BURNSIDE Saving record RAL19/0008.08, 24 Tranquility Way PALMVIEW Saving record OPW20/0018, Sunshine Cove Way MAROOCHYDORE Saving record REC15/0060.02, 1-7 Dixon Rd BUDERIM Saving record OPW20/0019, 18 Bury St NAMBOUR Saving record OPW20/0020, 12 Midden Pl PELICAN WATERS Saving record MPC20/0001, Bombora Dr BOKARINA Saving record MCU11/0231.01, Howard St NAMBOUR Saving record 2007/520037.05, Lot 10 Albyn Pl GLASS HOUSE MOUNTAINS Saving record OPW20/0022, 6/20 Avocado Lane MALENY Saving record MCU11/0231.02, Howard St NAMBOUR

Data

Downloaded 6596 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas Jedwards157 ajoh3691

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.57 MB) Use the API

rows 10 / 4085

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 1.6 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    54 records added, 48 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    48 records added, 48 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    48 records added, 32 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    36 records added, 29 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    29 records added, 22 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb