LoveMyData / noosa_council

Noosa Council Development Applications

Scrapes noo-web.t1cloud.com


Noosa Council Scraper

  • Server - Microsoft
  • System - eProperty
  • Cookie tracking - No
  • Pagination - Yes
  • Javascript - Yes __doPostBack
  • Detail Page - Yes
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default)
  • thismonth
  • lastmonth

Enjoy

Contributors otherchirps henare LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0-p648 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Using bundler 1.15.2  Fetching unf_ext 0.0.7.4  Installing unf_ext 0.0.7.4 with native extensions  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (23.62s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1 of 1 Saving record RAB19/0033 - 42 The Anchorage NOOSAVILLE QLD 4566 Saving record RAB19/0032 - 5 Werin St TEWANTIN QLD 4565 Saving record RAB19/0031 - 73 Gympie St TEWANTIN QLD 4565 Saving record PC19/0164 - Noosa Lions Football Club 100 Eenie Creek Rd NOOSA HEADS QLD 4567 Saving record PC19/0163 - Library 9 Maple St COOROY QLD 4563

Data

Downloaded 590 times by LoveMyData

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (764 KB) Use the API

rows 10 / 1628

council_reference address description info_url comment_url date_scraped date_received
OPW17/0192
Jalee 19 Henderson St SUNSHINE BEACH QLD 4567
19 Henderson Stret, Sunshine Beach - Vegetation Removal - Diamond Tre Services
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
RAB17/0080
22 Mcanally Dr SUNSHINE BEACH QLD 4567
2 Mcanaly Dr SUNSHINE BEACH - Carport & Storerom in Front Boundary
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
OPW17/0191
Bayona 30 Alderly Tce NOOSA HEADS QLD 4567
30 Alderly Terace, Nosa Heads - Vegetation Clearing - Profesional Tre Surgery
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
RAB17/0079
4 Pipi Cres NOOSAVILLE QLD 4566
4 Pipi Cres NOSAVILE - Carport & Covered Deck within boundary setbacks
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-06-02
OPW17/0190
The Emerald Noosa 42 Hastings St NOOSA HEADS QLD 4567
42 Hastings Stret, Nosa Heads - Vegetation Clearing - Nosa Tre Man
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-31
PC17/0537
52 Seaview Tce SUNSHINE BEACH QLD 4567
Alterations & Aditions to Dweling - 52 Seaview Tce SUNSHINE BEACH
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-31
PS17/0019
24 Ann St NOOSAVILLE QLD 4566
24 An St NOSAVILE - Building Format lots 1-2 & Comon Property, EMT A&B in Comon Property (Canceling Lot 9RP8159) - Skylines Surveyors
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30
PS17/0018
38 Pavilion St POMONA QLD 4568
38 Pavilion St POMONA - Standard Format Lots 1 & 2, EMT A in Lot 2 SP29373 (Canceling Lot 1 RP196984) - Max Waterson & Asoc
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30
PS17/0017
Lake Macdonald Dr LAKE MACDONALD QLD 4563
Botanical Gardens Lake Macdonald Dr LAKE MACDONALD - Standard Format - EMT B in Lot 1 RP167258 - Max Waterson & Asoc
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30
RAB17/0078
89 Lake Weyba Dr NOOSAVILLE QLD 4566
89 Lake Weyba Dr NOSAVILE - Carport within boundary setback
mailto:mail@noosa.qld.gov.au
2017-06-02
2017-05-30

Statistics

Average successful run time: 1 minute

Total run time: about 13 hours

Total cpu time used: 12 minutes

Total disk space used: 803 KB

History

  • Auto ran revision dfddafc5 and completed successfully .
    5 records added in the database
    7 pages scraped
  • Auto ran revision dfddafc5 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision dfddafc5 and completed successfully .
    nothing changed in the database
    19 pages scraped
  • Auto ran revision dfddafc5 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision dfddafc5 and completed successfully .
    7 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

noosa_council / scraper.rb