planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.3 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.............  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching json 2.2.0  Installing json 2.2.0 with native extensions  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.0  Installing sqlite3 1.4.0 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 3 Gemfile dependencies, 20 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (31.42s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record MCU10/0025.02, 30 Henebery Rd BURNSIDE Saving record POS20/0071, 15 & 17 Yinni Street Maroochydore Saving record MCU20/0088, 22 Ellis Way AND Hamilton St MERIDAN PLAINS Saving record OPW19/0625.01, Laxton Rd PALMVIEW Saving record REC13/0088.08, 112 Panorama Dr EUMUNDI Saving record RAL20/0037, 22 Ellis Way and Hamilton St MERIDAN PLAINS Saving record MCU15/0080.02, 14 Henebery Rd Burnside Saving record MCU20/0090, 54-64 Dynes Rd VALDORA Saving record MCU20/0092, Pummelo Cct PALMWOODS Saving record OPW20/0148, Pummelo Cct PALMWOODS Saving record REC14/0095.03, 83 Malones Road KIELS MOUNTAIN Saving record MCU19/0011.01, 26 Cook Rd BLI BLI Saving record POS20/0073, Forest Edge & Botanica Saving record OPW20/0147, 48 Lavender Cl GLENVIEW Saving record MCU20/0089, 81 Flaxton Dr and 2 Ringwood Lane MAPLETON Saving record MCU16/0115.02, 78 Simpson St BEERWAH Saving record MCU18/0149.01, 668 Bli Bli Rd NAMBOUR Saving record MCU20/0091, Hamptons At Montville 20 Hamptons Pl MONTVILLE Saving record MCU14/0174.02, 93 & 125 David Low Way DIDDILLIBAH Saving record POS20/0075, Walsh Rd HIGHWORTH and Nambour-Mapleton Rd BURNSIDE Saving record OPW20/0152, Kawana Way BIRTINYA Saving record MCU20/0094, 62 Edmund St SHELLEY BEACH Saving record RAL20/0038, 1-7 Dixon Rd BUDERIM Saving record OPW20/0149, 56-62 Whites Rd BUDERIM Saving record MCU20/0095, 18 Watson Street Currimundi Saving record 2007/510003.01, 68 Nicklin Way PARREARRA Saving record REC00/0027.22, 17 Saving record SPS20/0003, 18 and 20 Wright St MAROOCHYDORE Saving record MCU20/0100, 51 Roderick St and 19 Buccleugh St MOFFAT BEACH Saving record MCU17/0049.03, 13 and 15 The Esplanade MAROOCHYDORE Saving record MCU20/0096, 112 Warrack St COOLUM BEACH Saving record MCU20/0097, 38 Kingfisher Cres PALMVIEW Saving record MCU19/0211.02, 127-133 Quanda Rd COOLUM BEACH Saving record MCU20/0099, 20 Tantula Rd East ALEXANDRA HEADLAND Saving record OPW20/0153, Springs Dr MERIDAN PLAINS Saving record REC15/0237.01, 42 Coolum View Tce BUDERIM Saving record REC16/0049.03, Pelican Waters Golf Club 40 Mahogany Dr PELICAN WATERS Saving record OPW20/0150, 20 Tantula Rd East ALEXANDRA HEADLAND Saving record MCU19/0347, 18 Paulger Flat Rd YANDINA Saving record MCU14/0039.02, 787 Landsborough-Maleny Rd Bald Knob Saving record MCU15/0216.08, 10 -12 Duporth Ave MAROOCHYDORE Saving record MCU15/0283.02, 3 Island Ct MINYAMA Saving record OPW20/0155, Lot 906 Bokarina Blvd BOKARINA Saving record MCU20/0098, 100 Duporth Avenue Maroochydore Saving record REC16/0026.03, 10-18 Pikki St MAROOCHYDORE Saving record RAL20/0040, 618 Petrie Creek Rd ROSEMOUNT Saving record OPW20/0154, Burran Ct, 31 Martins Dr and 6 Charlston Pl KULUIN Saving record REC15/0172.01, 6 Bean St MALENY Saving record MCU18/0107.01, 102-104 Coolum Terrace COOLUM BEACH Saving record RAL18/0148.01, 7 Clearwing Dr PALMVIEW Saving record OPW19/0050.02, 102-104 Coolum Terrace COOLUM BEACH Saving record RAL18/0085.08, 36 Velvey Way Palmview Saving record MCU19/0305.01, 41 Empire Cres and 5 Palmetto St CHEVALLUM Saving record MPC11/0004.17, Wurley Dr BOKARINA Saving record MPC11/0003.10, Wurley Dr BOKARINA Saving record MCU18/0089.02, 5 Meridian St BOKARINA and 408 Nicklin Way BOKARINA Saving record RAL20/0041, 5 Gosling Street Caloundra Saving record RAL20/0042, 65 Stirling Rd PEACHESTER Saving record OPW20/0156, 5 Gosling Street Caloundra Saving record MPC13/0013.03, 7 Splendor Place Birtinya Saving record MCU20/0101, 5 Gosling Street Caloundra Saving record MCU15/0214.05, 943-951, 953-971, 973-1005 David Low Way MARCOOLA Saving record OPW20/0157, 49 Landsborough-Maleny Rd LANDSBOROUGH Saving record MCU20/0102, 155 Clarendon Rd PEREGIAN BEACH

Data

Downloaded 7099 times by slav123 openaustralia LoveMyData DevAppProject WilliamYun jesse-a-reilly TheZepto spatialsteve dsccramb equivalentideas EMBLAPROJECTS Corris224 ajoh3691 Jedwards157

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.75 MB) Use the API

rows 10 / 4593

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 29 days

Total cpu time used: about 1 hour

Total disk space used: 1.78 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    64 records added, 63 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    74 records added, 62 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    67 records added, 59 records removed in the database
  • Auto ran revision 6a6ca4ed and failed .
    nothing changed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    79 records added, 79 records removed in the database
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb