planningalerts-scrapers / walgett

Walgett Development Applications


This is a scraper that runs on Morph. To get started see the documentation

Contributors CloCkWeRX

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  The git source `git://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching git://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git  Fetching concurrent-ruby 1.0.5  Fetching i18n 0.8.4  Fetching minitest 5.10.2  Installing minitest 5.10.2  Installing i18n 0.8.4  Installing concurrent-ruby 1.0.5  Fetching thread_safe 0.3.6  Fetching multi_json 1.12.1  Installing thread_safe 0.3.6  Installing multi_json 1.12.1  Fetching unf_ext 0.0.7.4  Fetching mime-types-data 3.2016.0521  Fetching netrc 0.11.0  Installing netrc 0.11.0  Installing mime-types-data 3.2016.0521  Fetching rgeo 0.6.0  Fetching httpclient 2.8.3  Installing rgeo 0.6.0 with native extensions  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.4 with native extensions  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Using bundler 1.15.2  Fetching tzinfo 1.2.3  Installing tzinfo 1.2.3  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching activesupport 5.1.1  Installing activesupport 5.1.1  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Fetching activemodel 5.1.1  Installing activemodel 5.1.1  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching rest-client 2.0.2  Installing rest-client 2.0.2  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching scraperwiki 3.0.2  Installing scraperwiki 3.0.2  Fetching scraperwiki-morph 0.1.1  Installing scraperwiki-morph 0.1.1  Fetching rgeo-geojson 0.4.3  Installing rgeo-geojson 0.4.3  Fetching atdis 0.3.13  Installing atdis 0.3.13  Using atdisplanningalertsfeed 0.0.1 from git://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git (at master@7acbbe2)  Bundle complete! 1 Gemfile dependency, 26 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (19.07s)  Cleaning up the bundler cache. -----> Detecting rake tasks   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.3.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response': 502 Bad Gateway (RestClient::BadGateway) from /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!' from /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result' from /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit' from /app/vendor/ruby-2.3.4/lib/ruby/2.3.0/net/http.rb:853:in `start' from /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit' from /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute' from /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute' from /app/vendor/bundle/ruby/2.3.0/gems/rest-client-2.0.2/lib/restclient.rb:67:in `get' from /app/vendor/bundle/ruby/2.3.0/gems/atdis-0.3.13/lib/atdis/model.rb:73:in `read_url' from /app/vendor/bundle/ruby/2.3.0/gems/atdis-0.3.13/lib/atdis/feed.rb:62:in `applications' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/atdisplanningalertsfeed-7acbbe22df71/lib/atdisplanningalertsfeed.rb:34:in `fetch_all_pages' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/atdisplanningalertsfeed-7acbbe22df71/lib/atdisplanningalertsfeed.rb:16:in `save' from scraper.rb:6:in `<main>'

Data

Downloaded 2157 times by openaustralia LoveMyData jesse-a-reilly TheZepto WilliamYun MikeRalphson Jedwards157

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (38 KB) Use the API

rows 10 / 59

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to
020.2016.00000004.001
142 Pandora Street, LIGHTNING RIDGE, NSW 2834
Class 1a -Single Dwelling
2016-03-21
2016-03-07
020.2016.00000003.001
Pitt Street, WALGETT, NSW 2832
Subdivision
2016-03-21
2016-03-04
2016-03-04
2016-03-04
020.2016.00000002.001
Mining, LIGHTNING RIDGE, NSW 2834
Class 1a -Single Dwelling
2016-03-21
2016-03-01
020.2016.00000006.001
17 McNamara Street, CARINDA, NSW 2831
Change of Use
2016-04-14
2016-04-14
2016-04-14
2016-04-14
020.2016.00000005.001
19 Fantasia Street, LIGHTNING RIDGE, NSW 2834
Class 10a - Garage
2016-04-14
2016-04-13
020.2016.00000007.001
4 Morilla Street, LIGHTNING RIDGE, NSW 2834
Class 10b - Fence
2016-04-28
2016-04-27
020.2016.00000008.001
45 Alma Street, BURREN JUNCTION, NSW 2386
Class 10b - Antenna
2016-05-04
2016-05-04
2016-05-04
2016-05-04
020.2011.00000009.002
93 Pandora Street, LIGHTNING RIDGE, NSW 2834
Subdivision
2016-05-10
2016-05-09
020.2016.00000009.001
26 Pandora Street, LIGHTNING RIDGE, NSW 2834
Change of Use
2016-05-23
2016-05-23
020.2011.00000034.002
**** NOT DEFINED ****, UNKNOWN, NSW 2383
Subdivision
2016-05-25
2016-05-25

Statistics

Average successful run time: 1 minute

Total run time: about 16 hours

Total cpu time used: 20 minutes

Total disk space used: 61 KB

History

  • Auto ran revision c8924b9b and failed .
    nothing changed in the database
  • Auto ran revision c8924b9b and failed .
    nothing changed in the database
  • Auto ran revision c8924b9b and completed successfully .
    nothing changed in the database
  • Auto ran revision c8924b9b and completed successfully .
    nothing changed in the database
  • Auto ran revision c8924b9b and completed successfully .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

walgett / scraper.rb