planningalerts-scrapers / wollondilly

Wollondilly Shire Council Development Applications

README.md

Wollondilly now has a custom web application which has an API. So, we're using that.

Contributors LoveMyData mlandauer

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.6  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2019.0331  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.6 with native extensions  Installing mime-types-data 3.2019.0331  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing webrobots 0.1.2  Installing sqlite3 1.3.13 with native extensions  Fetching net-http-persistent 3.0.1  Installing net-http-persistent 3.0.1  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.3  Installing nokogiri 1.10.3 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (23.10s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in `initialize': execution expired (Net::OpenTimeout) from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in `open' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in `block in connect' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/timeout.rb:103:in `timeout' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:935:in `connect' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:920:in `do_start' from /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:915:in `start' from /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-3.0.1/lib/net/http/persistent.rb:710:in `start' from /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-3.0.1/lib/net/http/persistent.rb:640:in `connection_for' from /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-3.0.1/lib/net/http/persistent.rb:945:in `request' from /app/vendor/bundle/ruby/2.5.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:280:in `fetch' from /app/vendor/bundle/ruby/2.5.0/gems/mechanize-2.7.6/lib/mechanize.rb:464:in `get' from scraper.rb:17:in `<main>'

Data

Downloaded 338 times by openaustralia dcalde muzzie86

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (312 KB) Use the API

rows 10 / 904

council_reference address description info_url date_scraped date_received on_notice_from on_notice_to lat lng
DA/2015/605/2
21 Station Master Avenue THIRLMERE, NSW
Modification to condition 24(5) for purpose of installing dog-proof fencing to prevent dogs from entering the conservation area
https://tracking.wollondilly.nsw.gov.au/detail/010.2015.00000605.002
2020-07-29
2020-07-09
-34.2015815866067
150.57718975772
DA/2017/7/2
40 Burragorang Street THE OAKS, NSW
Modification - design changes
https://tracking.wollondilly.nsw.gov.au/detail/010.2017.00000007.002
2020-07-29
2020-07-09
2020-07-15
2020-07-29
-34.0785271073153
150.566013656916
DA/2018/755/3
18 Torumba Circuit SILVERDALE, NSW
Modification - Modify condition 13 (1)
https://tracking.wollondilly.nsw.gov.au/detail/010.2018.00000755.003
2020-07-29
2020-07-10
-33.9125745023159
150.604898799426
DA/2019/597/3
85 Eagles Road RAZORBACK, NSW
Modification - design changes
https://tracking.wollondilly.nsw.gov.au/detail/010.2019.00000597.003
2020-07-29
2020-07-06
2020-07-10
2020-07-24
-34.1315649588424
150.672930057128
DA/2019/658/2
2 Kalangara Road SILVERDALE, NSW
Modification - design changes
https://tracking.wollondilly.nsw.gov.au/detail/010.2019.00000658.002
2020-07-29
2020-07-03
-33.911332578772
150.604277567584
DA/2020/170/2
120 Burrells Road MENANGLE, NSW
Modification - include temporary occupation
https://tracking.wollondilly.nsw.gov.au/detail/010.2020.00000170.002
2020-07-29
2020-07-07
-34.1561082189654
150.691282209498
DA/2020/231/2
8 Hassall Road BUXTON, NSW
Modification - Design changes
https://tracking.wollondilly.nsw.gov.au/detail/010.2020.00000231.002
2020-07-29
2020-07-10
-34.2487649560524
150.540986009254
DA/2020/315/1
529 Old Razorback Road CAWDOR, NSW
Dwelling additions and s68 connect to existing
https://tracking.wollondilly.nsw.gov.au/detail/010.2020.00000315.001
2020-07-29
2020-06-29
-34.1159615115923
150.668708408434
DA/2020/316/1
5 Camellia Court PICTON, NSW
Secondary dwelling & retaining walls
https://tracking.wollondilly.nsw.gov.au/detail/010.2020.00000316.001
2020-07-29
2020-06-29
-34.1813143878095
150.614690808844
DA/2020/317/1
5 Dillies Lane TAHMOOR, NSW
Dwelling house
https://tracking.wollondilly.nsw.gov.au/detail/010.2020.00000317.001
2020-07-29
2020-06-29
-34.2187083600634
150.604580262409

Statistics

Average successful run time: 4 minutes

Total run time: about 15 hours

Total cpu time used: 8 minutes

Total disk space used: 652 KB

History

  • Auto ran revision 06a19fe9 and failed .
    nothing changed in the database
  • Auto ran revision 06a19fe9 and failed .
    nothing changed in the database
  • Auto ran revision 06a19fe9 and failed .
    nothing changed in the database
  • Auto ran revision 06a19fe9 and failed .
    nothing changed in the database
  • Auto ran revision 06a19fe9 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

wollondilly / scraper.rb