planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.92s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record POS22/0085, 38A Stringybark Road and 38 Central Drive Sippy Downs Saving record POS22/0096, 171 Oceanic Dr Warana Saving record OPW22/0306, 29 Casuarina Dr LITTLE MOUNTAIN Saving record MCU22/0193, 85 Tantula Rd West ALEXANDRA HEADLAND Saving record OPW22/0307, 92 Lake Kawana Blvd BIRTINYA Saving record MCU22/0194, 61 Lara Dr BUDERIM Saving record OPW22/0308, 2/28 Culbara St MOOLOOLABA Saving record OPW22/0309, 192 Neill Rd MOOLOOLAH VALLEY Saving record MCU22/0196, FASTRACK 19 Lomandra Pl COOLUM BEACH Saving record OPW22/0310, FASTRACK 19 Lomandra Pl COOLUM BEACH Saving record OPW22/0311, 25 Neerim Drive MOOLOOLABA Saving record POS22/0070, 864 Yandina Bli Bli Road Bli Bli Saving record MCU21/0107.01, 38 Bluebird Pde BOKARINA Saving record RAL22/0065, FASTTRACK 281 Eumundi Range Rd EUMUNDI and 48-52 Sunrise Rd EUMUNDI Saving record RAL22/0066, 27 Hilltop Cres MALENY Saving record OPW22/0312, 1 Power Rd BUDERIM Saving record OPW22/0313, 39 Barns Lane COOLUM BEACH Saving record OPW22/0084.01, 139 Bradman Ave MAROOCHYDORE Saving record OPW22/0314, 19 Wagtail Cl BUDERIM Saving record OPW22/0315, 44 Schirrmann Dr MAROOCHYDORE Saving record REC09/0030.09, 7 Rosella Tce PEREGIAN SPRINGS Saving record RAL17/2076.01, 167A Vise Rd MONS Saving record MCU20/0147.01, 66 Cooney Rd BLI BLI Saving record POS22/0100, 11 Dinmore St WOOMBYE Saving record POS22/0101, 45 Learg St Coolum Beach Saving record RAL22/0067, 170 and 186 Burnside Rd PERWILLOWEN Saving record MCU22/0197, Palmetto St CHEVALLUM Saving record OPW22/0316, Palmetto St CHEVALLUM Saving record MCU10/2061.01, 66 Nicklin Way PARREARRA Saving record 2009/56M0015.03, 150 Fred Chaplin Cct CORBOULD PARK Saving record OPW22/0317, 11 Sydney St NAMBOUR Saving record RAL22/0068, 50 Airport Dr MARCOOLA Saving record OPW22/0318, 85 Picnic Point Esplanade MAROOCHYDORE Saving record OPW22/0319, 1, 3 and 5 Commercial Rd KULUIN Saving record MCU22/0199, 34 and 36 Rigby St NAMBOUR Saving record MCU15/0233.01, 6/38 Lysaght Street COOLUM BEACH Saving record MCU22/0200, 47 Carwoola Cre MOOLOOLABA Saving record MCU22/0201, 18 Newing Way CALOUNDRA WEST Saving record OPW22/0320, 18 Newing Way CALOUNDRA WEST Saving record OPW22/0321, 41 Glenbrook Dr NAMBOUR Saving record MCU22/0202, 6 Jeanette Ave NAMBOUR Saving record OPW22/0322, 924 Eumundi Noosa Road DOONAN Saving record RAL19/0116.01, Bells Creek Rd COOCHIN CREEK Saving record MCU22/0203, 14 Curbarra St BUDDINA Saving record MCU22/0204, 404 Oceanic Drive WURTULLA Saving record RAL17/2006.02, 55 William Road EUMUNDI Saving record OPW19/0369.04, 55 William Road Eumundi Saving record POS22/0084, 41 Smith Dr MOOLOOLAH VALLEY Saving record MPC22/0006, Lake Kawana Bvd BIRTINYA Saving record MCU22/0205, 54 and 56 Washington St NAMBOUR Saving record OPW22/0323, 54 and 56 Washington St NAMBOUR Saving record MCU21/0145.01, 1 Biggs Rd DULONG Saving record 2002/520039.49, 81 Palmview Forest Dr PALMVIEW Saving record MCU22/0206, 9 Wyanda Dr WARANA Saving record OPW22/0324, 9 Wyanda Dr WARANA Saving record MCU22/0207, 101 Stoney Wharf Rd BLI BLI Saving record MPC11/0003.16, Bokarina Bvd BOKARINA Saving record RAL22/0069, McDonalds Rd BLI BLI Saving record MCU22/0208, 24 Bosky Pl NAMBOUR Saving record OPW22/0325, Harmony Blvd PALMVIEW Saving record OPW22/0326, 774 Eudlo Rd PALMWOODS Saving record MCU22/0209, 35-55 Gobberts Rd YANDINA Saving record OPW22/0327, 36 Wategos Cct PELICAN WATERS Saving record OPW22/0328, 37-41 Chants Rd VALDORA Saving record MCU22/0210, 20 Mulholland Pl NAMBOUR Saving record OPW22/0329, 16 Honeysuckle Pl YANDINA Saving record OPW22/0330, 17 Blackwood St MAROOCHYDORE Saving record MCU22/0211, 6 Conebush St MUDJIMBA Saving record OPW22/0331, 171 Bradman Ave MAROOCHYDORE Saving record OPW22/0332, 1 Power Rd BUDERIM Saving record MCU22/0212, 1 Teske Drive PALMWOODS Saving record OPW22/0333, 1 Teske Drive PALMWOODS Saving record MCU16/0085.02, Harmony Bvd and Bruce Hwy PALMVIEW

Data

Downloaded 11746 times by slav123 openaustralia LoveMyData dcalde DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb yourneighbourhood equivalentideas Jedwards157 EMBLAPROJECTS Corris224 ajoh3691

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.49 MB) Use the API

rows 10 / 9568

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 3.52 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 73 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 63 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    64 records added, 45 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    53 records added, 46 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    52 records added, 41 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb