LoveMyData / cumberland

Cumberland Council - Development Applications


Cumberland Council Scraper

Cumberland Council involves the followings * Server - Microsoft * Cookie tracking - No * Pagnation - Yes - But this is mainly for browser not scraper * Javascript - No * Clearly defined data within a row - No * System - XC Track

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default, today -7 days) * thismonth * lastmonth

Enjoy

Contributors LoveMyData

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.2  Fetching httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Installing unf_ext 0.0.7.2 with native extensions  Installing httpclient 2.8.3  Fetching net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing net-http-digest_auth 1.4.1  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Installing mini_portile2 2.1.0  Fetching sqlite3 1.3.13  Fetching mime-types 3.1  Installing sqlite3 1.3.13 with native extensions  Installing mime-types 3.1  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20170223  Installing domain_name 0.5.20170223  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (22.15s)  Cleaning up the bundler cache. -----> Detecting rake tasks   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.3.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scraping for thisweek, changable via MORPH_PERIOD variable /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/net/http.rb:882:in `rescue in block in connect': Failed to open TCP connection to eplanning.cumberland.nsw.gov.au:80 (getaddrinfo: Name or service not known) (SocketError) from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/net/http.rb:879:in `block in connect' from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout' from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/timeout.rb:101:in `timeout' from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/net/http.rb:878:in `connect' from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/net/http.rb:863:in `do_start' from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/net/http.rb:858:in `start' from /app/vendor/bundle/ruby/2.3.0/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:700:in `start' from /app/vendor/bundle/ruby/2.3.0/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:631:in `connection_for' from /app/vendor/bundle/ruby/2.3.0/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:994:in `request' from /app/vendor/bundle/ruby/2.3.0/gems/mechanize-2.7.5/lib/mechanize/http/agent.rb:274:in `fetch' from /app/vendor/bundle/ruby/2.3.0/gems/mechanize-2.7.5/lib/mechanize.rb:464:in `get' from scraper.rb:23:in `<main>'

Data

Downloaded 1576 times by LoveMyData jacroft408 dcalde

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.06 MB) Use the API

rows 10 / 2314

council_reference address description info_url comment_url date_scraped date_received
DA-1060/2004.5 MERRYLANDS
37 Hawksview Street MERRYLANDS NSW 2160
DA's - S96(2) modification (DRAFT)
mailto:council@cumberland.nsw.gov.au
2017-11-01
2017-10-30
DA-443/2015.2 MERRYLANDS
5 Eric Avenue MERRYLANDS NSW 2160
DA's - s96(1a) modification (DRAFT)
mailto:council@cumberland.nsw.gov.au
2017-11-01
2017-10-30
DA-451/2017 MERRYLANDS
3 Holroyd Road MERRYLANDS NSW 2160
DA's - Secondary dwelling/Granny Flat (DRAFT)
mailto:council@cumberland.nsw.gov.au
2017-11-01
2017-10-30
DA-35/2016/A LIDCOMBE
13 Olympic Drive LIDCOMBE NSW 2141
Development Applications - Section 96(1A) application to modify cladding fibro to brick.
mailto:council@cumberland.nsw.gov.au
2017-11-01
2017-10-31
DA-450/2017 AUBURN
7 Cornwall Road AUBURN NSW 2144
Development Applications - Construction of a new secondary dwelling (granny flat) attached Alfresco.
mailto:council@cumberland.nsw.gov.au
2017-11-01
2017-10-31
DA-451/2017 SOUTH GRANVILLE
229 Blaxcell Street SOUTH GRANVILLE NSW 2142
Development Applications - Conversion of a dwelling and outbuilding to a boarding house.
mailto:council@cumberland.nsw.gov.au
2017-11-01
2017-10-31
DA-76/2017.2 GIRRAWEEN
33 Girraween Road GIRRAWEEN NSW 2145
DA's - Child Care Centres and Demolition (DRAFT) (S96) Child Care Centres Demolition
mailto:council@cumberland.nsw.gov.au
2017-11-01
2017-10-31
DA-458/2017 MERRYLANDS
45 Davies Street MERRYLANDS NSW 2160
DA's - Attached Dual Occupancy and Subdivision and Demolition (DRAFT)
mailto:council@cumberland.nsw.gov.au
2017-11-06
2017-11-06
DA-459/2017 PEMULWUY
Butu Wargun Drive PEMULWUY NSW 2145
DA's - Dwelling (DRAFT) (LOT 141)
mailto:council@cumberland.nsw.gov.au
2017-11-06
2017-11-06
DA-460/2017 GIRRAWEEN
5 Mia Mia Street GIRRAWEEN NSW 2145
DA's - Detached Dual Occupancy and Attached Dual Occupancy and Demolition (DRAFT)
mailto:council@cumberland.nsw.gov.au
2017-11-06
2017-11-06

Statistics

Average successful run time: less than a minute

Total run time: 1 day

Total cpu time used: 32 minutes

Total disk space used: 1.08 MB

History

  • Auto ran revision 5c42c5ae and failed .
    nothing changed in the database
  • Auto ran revision 5c42c5ae and failed .
    nothing changed in the database
  • Auto ran revision 5c42c5ae and failed .
    nothing changed in the database
  • Auto ran revision 5c42c5ae and failed .
    nothing changed in the database
  • Auto ran revision 5c42c5ae and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

cumberland / scraper.rb