StashProperty / wollondilly

Wollondilly Shire Council Development Applications

README.md

Wollondilly now has a custom web application which has an API. So, we're using that.

Contributors LoveMyData mlandauer AndreM101

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.6  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2019.0331  Installing unf_ext 0.0.7.6 with native extensions  Installing httpclient 2.8.3  Installing mime-types-data 3.2019.0331  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Fetching net-http-persistent 3.0.1  Installing sqlite3 1.3.13 with native extensions  Installing net-http-persistent 3.0.1  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.3  Installing nokogiri 1.10.3 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (23.32s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in `initialize': execution expired (Net::OpenTimeout) from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in `open' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in `block in connect' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/timeout.rb:103:in `timeout' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:935:in `connect' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:920:in `do_start' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:915:in `start' from /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-3.0.1/lib/net/http/persistent.rb:710:in `start' from /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-3.0.1/lib/net/http/persistent.rb:640:in `connection_for' from /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-3.0.1/lib/net/http/persistent.rb:945:in `request' from /app/vendor/bundle/ruby/2.5.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:280:in `fetch' from /app/vendor/bundle/ruby/2.5.0/gems/mechanize-2.7.6/lib/mechanize.rb:464:in `get' from scraper.rb:17:in `<main>'

Data

Downloaded 156 times by dcalde

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (353 KB) Use the API

rows 10 / 1043

council_reference address description info_url date_scraped date_received on_notice_from on_notice_to lat lng
DA/2021/90/1
119 Braddocks Road WEROMBI, NSW
Dwelling house and shed
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000090.001
2021-03-13
2021-02-11
2021-02-16
2021-02-23
-34.0124237973208
150.544820064762
DA/2021/91/1
119 Braddocks Road WEROMBI, NSW
S68 AWTS with surface spray irrigation
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000091.001
2021-03-13
2021-02-11
-34.0124237973208
150.544820064762
DA/2021/92/1
3 Mount Bethel Circuit ORANGEVILLE, NSW
Equestrian arena, associated earthworks and retaining works (including legitimise use of existing unauthorised works)
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000092.001
2021-03-13
2021-02-11
-34.0117128155565
150.62815668369
DA/2021/93/1
21 Moore Road OAKDALE, NSW
Secondary dwelling
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000093.001
2021-03-13
2021-02-11
2021-02-23
2021-03-09
-34.0772745524875
150.509846105654
DA/2021/94/1
8 Cecilia Place THIRLMERE, NSW
Dwelling house
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000094.001
2021-03-13
2021-02-11
-34.204377192893
150.58015298769
DA/2021/95/1
155 Lyrebird Road PHEASANTS NEST, NSW
Shed
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000095.001
2021-03-13
2021-02-11
2021-02-16
2021-02-23
-34.2336402001306
150.623829357986
CDC/2021/55/1
51B Kader Street BARGO, NSW
Dwelling House
https://tracking.wollondilly.nsw.gov.au/detail/018.2021.00000055.001
2021-03-13
2021-02-11
-34.2897148023669
150.574805997948
DA/2018/610/4
55 Bronzewing Street West TAHMOOR, NSW
Modification to DA/2018/610/1 proposing a minor lot boundary adjustment and conditions that nominates lots burdened by inter-allotment drainage and rectify the anomalies in relation to lot numbering
https://tracking.wollondilly.nsw.gov.au/detail/010.2018.00000610.004
2021-03-14
2021-02-12
-34.2246989419475
150.580484405743
DA/2021/96/1
140 Eltons Road SILVERDALE, NSW
Construction of an earth dam for use as a sediment basin
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000096.001
2021-03-14
2021-02-12
-33.9275401256705
150.608201223985
DA/2021/97/1
21 Chapman Street TAHMOOR, NSW
Proposed detached dual occupancy & two lot torrens title subdivision
https://tracking.wollondilly.nsw.gov.au/detail/010.2021.00000097.001
2021-03-14
2021-02-12
2021-02-16
2021-03-02
-34.2198040546689
150.58699880774

Statistics

Average successful run time: less than a minute

Total run time: about 12 hours

Total cpu time used: 9 minutes

Total disk space used: 694 KB

History

  • Auto ran revision 784543d9 and failed .
    nothing changed in the database
  • Auto ran revision 784543d9 and failed .
    nothing changed in the database
  • Auto ran revision 784543d9 and failed .
    nothing changed in the database
  • Auto ran revision 784543d9 and failed .
    nothing changed in the database
  • Auto ran revision 784543d9 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

wollondilly / scraper.rb