LoveMyData / noosa_council

Noosa Council Development Applications

Scrapes noo-web.t1cloud.com


Noosa Council Scraper

  • Server - Microsoft
  • System - eProperty
  • Cookie tracking - No
  • Pagination - Yes
  • Javascript - Yes __doPostBack
  • Detail Page - Yes
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default)
  • thismonth
  • lastmonth

Enjoy

Contributors otherchirps henare LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0-p648 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Using bundler 1.15.2  Fetching unf_ext 0.0.7.4  Installing unf_ext 0.0.7.4 with native extensions  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.84s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1 of 1 Skipping already saved record RAB18/0194 Skipping already saved record RAB18/0195 Skipping already saved record PC18/1169 Skipping already saved record PC18/1167 Skipping already saved record PC18/1163 Skipping already saved record 51992.213.02 Skipping already saved record 51987.2951.01 Skipping already saved record 132007.738.03 Skipping already saved record MCU18/0156 Skipping already saved record MCU18/0154 Skipping already saved record OPW18/0244 Skipping already saved record OPW18/0243 Skipping already saved record PC18/1162 Skipping already saved record PC18/1159 Skipping already saved record PC18/1158 Skipping already saved record PC18/1157 Skipping already saved record PC18/1161 Skipping already saved record PC18/1160 Skipping already saved record PC18/1166 Skipping already saved record RAP18/0042 Skipping already saved record DBW18/0021 Skipping already saved record DBW18/0020

Data

Downloaded 446 times by LoveMyData

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (599 KB) Use the API

rows 10 / 1291

council_reference address description info_url comment_url date_scraped date_received
OPW17/0192
Jalee 19 Henderson St SUNSHINE BEACH QLD 4567
19 Henderson Stret, Sunshine Beach - Vegetation Removal - Diamond Tre Services
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
RAB17/0080
22 Mcanally Dr SUNSHINE BEACH QLD 4567
2 Mcanaly Dr SUNSHINE BEACH - Carport & Storerom in Front Boundary
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
OPW17/0191
Bayona 30 Alderly Tce NOOSA HEADS QLD 4567
30 Alderly Terace, Nosa Heads - Vegetation Clearing - Profesional Tre Surgery
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
RAB17/0079
4 Pipi Cres NOOSAVILLE QLD 4566
4 Pipi Cres NOSAVILE - Carport & Covered Deck within boundary setbacks
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
OPW17/0190
The Emerald Noosa 42 Hastings St NOOSA HEADS QLD 4567
42 Hastings Stret, Nosa Heads - Vegetation Clearing - Nosa Tre Man
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-31
PC17/0537
52 Seaview Tce SUNSHINE BEACH QLD 4567
Alterations & Aditions to Dweling - 52 Seaview Tce SUNSHINE BEACH
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-31
PS17/0019
24 Ann St NOOSAVILLE QLD 4566
24 An St NOSAVILE - Building Format lots 1-2 & Comon Property, EMT A&B in Comon Property (Canceling Lot 9RP8159) - Skylines Surveyors
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30
PS17/0018
38 Pavilion St POMONA QLD 4568
38 Pavilion St POMONA - Standard Format Lots 1 & 2, EMT A in Lot 2 SP29373 (Canceling Lot 1 RP196984) - Max Waterson & Asoc
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30
PS17/0017
Lake Macdonald Dr LAKE MACDONALD QLD 4563
Botanical Gardens Lake Macdonald Dr LAKE MACDONALD - Standard Format - EMT B in Lot 1 RP167258 - Max Waterson & Asoc
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30
RAB17/0078
89 Lake Weyba Dr NOOSAVILLE QLD 4566
89 Lake Weyba Dr NOSAVILE - Carport within boundary setback
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30

Statistics

Average successful run time: 1 minute

Total run time: about 10 hours

Total cpu time used: 9 minutes

Total disk space used: 638 KB

History

  • Auto ran revision dfddafc5 and completed successfully .
    nothing changed in the database
    24 pages scraped
  • Auto ran revision dfddafc5 and completed successfully .
    8 records added in the database
  • Auto ran revision dfddafc5 and completed successfully .
    13 records added in the database
    16 pages scraped
  • Auto ran revision dfddafc5 and completed successfully .
    nothing changed in the database
    3 pages scraped
  • Auto ran revision dfddafc5 and completed successfully .
    1 record added in the database
    3 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

noosa_council / scraper.rb