planningalerts-scrapers / sunshine_coast

Sunshine Coast Council Development Applications


Sunshine coast council scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • JSON - Yes
  • Structured HTML - No (especially address is a nightmare)

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, last 14 days)
  • thismonth
  • lastmonth

Enjoy

Contributors LoveMyData MichaelBone mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.5.3 [1G-----> Installing dependencies using bundler version 1.15.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. [1G Fetching gem metadata from https://rubygems.org/...... [1G Fetching version metadata from https://rubygems.org/. [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Fetching connection_pool 2.2.2 [1G Using bundler 1.15.2 [1G Fetching unf_ext 0.0.7.5 [1G Fetching httpclient 2.8.3 [1G Installing connection_pool 2.2.2 [1G Installing httpclient 2.8.3 [1G Installing unf_ext 0.0.7.5 with native extensions [1G Fetching json 2.2.0 [1G Installing json 2.2.0 with native extensions [1G Fetching mime-types-data 3.2018.0812 [1G Installing mime-types-data 3.2018.0812 [1G Fetching net-http-digest_auth 1.4.1 [1G Installing net-http-digest_auth 1.4.1 [1G Fetching mini_portile2 2.4.0 [1G Installing mini_portile2 2.4.0 [1G Fetching ntlm-http 0.1.1 [1G Installing ntlm-http 0.1.1 [1G Fetching webrobots 0.1.2 [1G Installing webrobots 0.1.2 [1G Fetching sqlite3 1.4.0 [1G Installing sqlite3 1.4.0 with native extensions [1G Fetching net-http-persistent 3.0.0 [1G Installing net-http-persistent 3.0.0 [1G Fetching mime-types 3.2.2 [1G Installing mime-types 3.2.2 [1G Fetching nokogiri 1.10.1 [1G Installing nokogiri 1.10.1 with native extensions [1G Fetching unf 0.1.4 [1G Installing unf 0.1.4 [1G Fetching sqlite_magic 0.0.6 [1G Installing sqlite_magic 0.0.6 [1G Fetching domain_name 0.5.20180417 [1G Installing domain_name 0.5.20180417 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching http-cookie 1.0.3 [1G Installing http-cookie 1.0.3 [1G Fetching mechanize 2.7.6 [1G Installing mechanize 2.7.6 [1G Bundle complete! 3 Gemfile dependencies, 20 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into ./vendor/bundle. [1G Bundle completed (30.52s) [1G Cleaning up the bundler cache. [1G Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Getting 'Last 14 Days' data, changable via MORPH_PERIOD environment Saving record POS23/0091, 27 Hilltop Crescent MALENY Saving record MCU23/0130, 25 Link Crescent Coolum Beach Saving record OPW23/0225, 25 Link Crescent Coolum Beach Saving record MCU23/0131, FASTTRACK 268 Main Rd KULUIN Saving record MCU22/0030.01, 22 Empire Cres CHEVALLUM Saving record MCU23/0132, FASTTRACK 20 April Ct MAROOCHYDORE Saving record OPW23/0226, FASTTRACK 20 April Ct MAROOCHYDORE Saving record OPW23/0229, FASTTRACK 268 Main Rd KULUIN Saving record 2002/570012.11, 1 Barbuda Cct PARREARRA Saving record OPW23/0227, 325 Red Rd BEERBURRUM Saving record REC16/0049.06, 40 Mahogany Dr PELICAN WATERS Saving record OPW23/0228, 797-809 David Low Way MUDJIMBA Saving record OPW23/0230, 32 Bell St DICKY BEACH Saving record POS23/0085, 17 Greber Street Beerwah Saving record POS23/0097, 111 Millwell Road Maroochydore Saving record RAL23/0044, 145 Broadwater Ave MAROOCHYDORE Saving record OPW22/0488.01, *notes *Lot 5 Bruce Highway PALMVIEW HARMONY Linear Park 7 to 10 Saving record MCU21/0238.01, 114-116 Fred Chaplin Cct CORBOULD PARK Saving record OPW23/0231, 584 Valdora Rd VALDORA Saving record OPW23/0232, 54 Schirrmann Dr MAROOCHYDORE Saving record MCU21/0137.02, 372 Mons Rd FOREST GLEN Saving record OPW23/0233, 17 Sunshine Grove Pl YANDINA Saving record POS23/0001, 1 Camelot Ct BLI BLI Saving record RAL19/0116.02, Bells Creek Rd COOCHIN CREEK Saving record OPW23/0234, 145 Broadwater Ave, MAROOCHYDORE Saving record MCU23/0133, 145 Broadwater Ave MAROOCHYDORE Saving record MCU18/0126.01, 7/47 Lysaght St COOLUM BEACH Saving record OPW23/0235, 3 East Coolabine Rd COOLABINE Saving record MCU23/0134, 6 and 8 Lenco Cres LANDSBOROUGH Saving record OPW23/0236, 6 and 8 Lenco Cres LANDSBOROUGH Saving record MCU23/0135, 372 Mons Rd FOREST GLEN Saving record OPW23/0237, 372 Mons Rd FOREST GLEN Saving record MCU21/0293.02, 372 Mons Road FOREST GLEN Saving record OPW23/0238, 167 Lindeman Rd BEERWAH Saving record POS23/0095, 1 Flowers Rd & 271 Yandina Bli Bli Road MAROOCHY RIVER Saving record OPW23/0239, 111 & 115 Main St MONTVILLE Saving record MCU23/0136, FASTRACK 5 Kyeema St BUDDINA Saving record OPW23/0240, FASTRACK 5 Kyeema St BUDDINA Saving record REC16/0098.13, Pelican Waters Southern Lakes Stage 3 Saving record MCU19/0097.01, 27 and 29 Enterprise St CALOUNDRA WEST Saving record POS23/0098, 2771 & 2781 Eumundi-Kenilworth Rd KENILWORTH Saving record OPW23/0241, MOTEL Saving record OPW23/0242, 11 Lone Hand Lane EUMUNDI Saving record OPW23/0243, 1121 Cooloolabin Road Cooloolabin Saving record MCU23/0137, 69 Valley Drive Doonan Saving record RAL23/0045, 242 & 248 Highlands Rd EUDLO Saving record MCU23/0138, 5/4 Lawyer Street Maleny Saving record POS23/0060, 55A Arthur St CALOUNDRA Saving record MCU21/0251.01, Lot 1 Commercial Rd KULUIN Saving record MCU23/0139, FASTRACK 26 Mayfield St ALEXANDRA HEADLAND Saving record OPW23/0244, Harmony Display Village 3 Temporary Carpark Saving record OPW23/0245, FASTRACK 26 Mayfield St ALEXANDRA HEADLAND Saving record OPW23/0246, 10 Brushbox Pl EUMUNDI Saving record OPW23/0247, 8-12 Turner St BEERWAH Saving record REC16/0133.02, 8-16 Sorensen Ave BUDERIM Saving record OPW21/0520.02, 49 The Esplanade MAROOCHYDORE Saving record MCU23/0140, 97 Mill Hill Rd MONTVILLE Saving record OPW23/0036.01, 35 Vernon St NAMBOUR Saving record POS23/0089, 15-33, 35-53 Brandons Rd YANDINA Saving record REC02/0316.56, 5 Inverness Pl PEREGIAN SPRINGS Saving record ToAdviseMCU01, Maroochy Bvd MAROOCHYDORE Saving record OPW23/0248, 51 Main St MONTVILLE Saving record POS23/0074, Honeysuckle Place Yandina Saving record POS23/0079, Azure Forest Glen Stages 5 & 6 Saving record POS23/0102, 627 and 649 Maleny-Kenilworth Rd WITTA Saving record POS23/0105, 64 Brindabella Avenue PEREGIAN SPRINGS Saving record RAL23/0046, FASTRACK 77 Power Rd BUDERIM Saving record MCU19/0003.03, Mooloolah Community Centre 43 Bray Rd MOOLOOLAH VALLEY Saving record MCU20/0306.02, 49 The Esplanade MAROOCHYDORE Saving record OPW23/0249, 58 Nambour Mapleton Rd NAMBOUR Saving record MCU21/0303.01, 26 Second Ave MAROOCHYDORE Saving record POS23/0107, 8 Tagera Street Warana Saving record MCU23/0141, 1730-1732 David Low Way COOLUM BEACH Saving record OPW23/0250, Waterside Rtt BIRTINYA Saving record OPW23/0251, Ayliffe Park Esplanade GOLDEN BEACH

Data

Downloaded 13794 times by slav123 openaustralia LoveMyData dcalde mlandauer DevAppProject jesse-a-reilly WilliamYun TheZepto spatialsteve dsccramb yourneighbourhood equivalentideas BrendanFrameless ajoh3691 Jedwards157 EMBLAPROJECTS Corris224

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (4.14 MB) Use the API

rows 10 / 11364

date_scraped description info_url date_received council_reference address comment_url
2012-08-27
Indoor Recreation Sports Centre - Studdock Investments Pty Ltd & Cricket Generation Pty Ltd C/
2012-08-01
MCU12/0111
84 Sandalwood Lane, FOREST GLEN, QLD
2012-08-27
Extension to Relevant Period
2012-08-01
132006.1360.03
8 William St, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
132007.1841.04
26 Eenie Creek Rd, NOOSAVILLE, QLD
2012-08-27
Generally in Accordance
2012-08-01
2008/56M0014.02
476 Nicklin Way, WURTULLA, QLD
2012-08-27
45 Primary School Crt - Conversion of Dual Occupancy to Offices - Martin Kurvink
2012-08-02
MCU12/0112
43-45 Primary School Ct, MAROOCHYDORE, QLD
2012-08-27
Warehouse - Uptons Salvage Trading Pty Ltd
2012-08-02
MCU12/0113
31 Moroney Pl, BEERWAH, QLD
2012-08-27
62 Duporth Ave MAROOCHYDORE - Generally In Accordance
2012-08-03
MCU02/0088.05
52 Duporth Ave, MAROOCHYDORE, QLD
2012-08-27
Extension to Relevant Period - Furno's Pty Ltd
2012-08-03
MCU07/0214.03
88 Burnett St, BUDERIM, QLD
2012-08-27
Development Application for Material Change of Use to establish an Sport and Recreation (Naval Cadet Unit)
2012-08-03
1998/510053.01
Maleny-Stanley River Rd, MALENY, QLD
2012-08-27
Infrastructure Agreement
2012-08-06
MCU10/0052.01
0 Parklakes Dr, BLI BLI, QLD

Statistics

Average successful run time: less than a minute

Total run time: 30 days

Total cpu time used: about 1 hour

Total disk space used: 4.17 MB

History

  • Auto ran revision 6a6ca4ed and completed successfully .
    75 records added, 66 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    66 records added, 61 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    69 records added, 65 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    73 records added, 64 records removed in the database
  • Auto ran revision 6a6ca4ed and completed successfully .
    74 records added, 65 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

sunshine_coast / scraper.rb