planningalerts-scrapers / multiple_atdis

Scrapes multiple authorities that have ATDIS feed


This is a scraper that runs on Morph. To get started see the documentation

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

Contributors mlandauer dependabot[bot]

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching rake 12.3.3  Fetching concurrent-ruby 1.1.6  Fetching minitest 5.14.1  Installing minitest 5.14.1  Installing rake 12.3.3  Installing concurrent-ruby 1.1.6  Fetching thread_safe 0.3.6  Fetching zeitwerk 2.4.0  Installing zeitwerk 2.4.0  Fetching public_suffix 4.0.5  Installing public_suffix 4.0.5  Installing thread_safe 0.3.6  Fetching ast 2.4.1  Installing ast 2.4.1  Fetching multi_json 1.15.0  Installing multi_json 1.15.0  Fetching http-accept 1.7.0  Installing http-accept 1.7.0  Fetching unf_ext 0.0.7.7  Fetching mime-types-data 3.2020.0512  Fetching netrc 0.11.0  Installing unf_ext 0.0.7.7 with native extensions  Installing netrc 0.11.0  Installing mime-types-data 3.2020.0512  Fetching rgeo 2.1.1  Installing rgeo 2.1.1 with native extensions  Using bundler 1.15.2  Fetching safe_yaml 1.0.5  Installing safe_yaml 1.0.5  Fetching diff-lcs 1.4.4  Installing diff-lcs 1.4.4  Fetching hashdiff 1.0.1  Installing hashdiff 1.0.1  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching parallel 1.19.2  Installing parallel 1.19.2  Fetching rainbow 3.0.0  Installing rainbow 3.0.0  Fetching regexp_parser 1.7.1  Installing regexp_parser 1.7.1  Fetching rexml 3.2.5  Installing rexml 3.2.5  Fetching rspec-support 3.9.3  Installing rspec-support 3.9.3  Fetching ruby-progressbar 1.10.1  Installing ruby-progressbar 1.10.1  Fetching unicode-display_width 1.7.0  Installing unicode-display_width 1.7.0  Fetching sqlite3 1.4.2  Installing sqlite3 1.4.2 with native extensions  Fetching vcr 6.0.0  Installing vcr 6.0.0  Fetching i18n 1.8.5  Installing i18n 1.8.5  Fetching parser 2.7.1.4  Installing parser 2.7.1.4  Fetching addressable 2.7.0  Installing addressable 2.7.0  Fetching tzinfo 1.2.7  Installing tzinfo 1.2.7  Fetching mime-types 3.3.1  Installing mime-types 3.3.1  Fetching crack 0.4.3  Installing crack 0.4.3  Fetching rspec-core 3.9.2  Installing rspec-core 3.9.2  Fetching rspec-expectations 3.9.2  Installing rspec-expectations 3.9.2  Fetching rspec-mocks 3.9.1  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching rgeo-geojson 2.1.1  Installing rgeo-geojson 2.1.1  Fetching rubocop-ast 0.2.0  Installing rubocop-ast 0.2.0  Installing rspec-mocks 3.9.1  Fetching activesupport 6.0.3.2  Fetching webmock 3.8.3  Installing webmock 3.8.3  Installing activesupport 6.0.3.2  Fetching domain_name 0.5.20190701  Installing domain_name 0.5.20190701  Fetching rspec 3.9.0  Installing rspec 3.9.0  Fetching rubocop 0.88.0  Installing rubocop 0.88.0  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching activemodel 6.0.3.2  Installing activemodel 6.0.3.2  Fetching rest-client 2.1.0  Installing rest-client 2.1.0  Fetching atdis 0.5.0  Installing atdis 0.5.0  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching scraperwiki 3.0.2  Installing scraperwiki 3.0.2  Fetching scraperwiki-morph 0.1.1  Installing scraperwiki-morph 0.1.1  Bundle complete! 7 Gemfile dependencies, 51 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  If you are upgrading your Rails application from an older version of Rails:  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  If you are starting a NEW Rails application, you can ignore this notice.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Bundle completed (14.65s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting ATDIS feed data for bega_valley... Collecting ATDIS feed data for ballina... bega_valley: ERROR: Timed out connecting to server /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:749:in `rescue in transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:651:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `block in <main>' scraper.rb:39:in `each' scraper.rb:39:in `<main>' ballina: ERROR: 403 Forbidden Collecting ATDIS feed data for bathurst... /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `block in <main>' scraper.rb:39:in `each' scraper.rb:39:in `<main>' D, [2022-06-27T01:11:12.708839 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=3 D, [2022-06-27T01:11:15.493312 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=4 D, [2022-06-27T01:11:18.248152 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=5 D, [2022-06-27T01:11:20.954579 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=6 D, [2022-06-27T01:11:23.867499 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=7 D, [2022-06-27T01:11:25.599340 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=8 W, [2022-06-27T01:11:26.020784 #18] WARN -- : No/invalid pagination, assuming no records/aborting Storing 018.2022.00000115.001 - 19 Granite RI, KELSO, NSW 2795 Storing 018.2022.00000116.001 - 63 Basalt WAY, KELSO, NSW 2795 Storing 018.2022.00000089.001 - 39 Basalt WAY, KELSO, NSW 2795 Storing 010.2022.00000253.001 - 1/270A Havannah ST, SOUTH BATHURST, NSW 2795 Storing 010.2022.00000269.001 - 71 White Rock RD, WHITE ROCK, NSW 2795 Storing 010.2022.00000258.001 - 5 Church ST, TRUNKEY CREEK, NSW 2795 Storing 010.2022.00000232.001 - 160 Hartwood AVE, ROBIN HILL, NSW 2795 Storing 018.2022.00000106.001 - 70 George Thomas CL, THE LAGOON, NSW 2795 Storing 018.2022.00000103.001 - 46 Wentworth DR, KELSO, NSW 2795 Storing 018.2022.00000108.001 - 11 Peacock ST, EGLINTON, NSW 2795 Storing 018.2022.00000107.001 - 121 Hughes ST, KELSO, NSW 2795 Storing 010.2016.00000255.002 - 555 Limekilns RD, FOREST GROVE, NSW 2795 Storing 010.2022.00000217.001 - 13 Graham DR, KELSO, NSW 2795 Storing 018.2022.00000109.001 - 26 Barina PKY, KELSO, NSW 2795 Storing 010.2022.00000263.001 - 52 Rose ST, SOUTH BATHURST, NSW 2795 Storing 010.2022.00000279.001 - 16 Kite PL, RAGLAN, NSW 2795 Storing 018.2022.00000110.001 - 26 Basalt WAY, KELSO, NSW 2795 Storing 010.2022.00000271.001 - 27 Dovey DR, KELSO, NSW 2795 Storing 010.2022.00000262.001 - 13 Cain DR, KELSO, NSW 2795 Storing 010.2022.00000249.001 - 45 Hamilton ST, EGLINTON, NSW 2795 Storing 010.2022.00000273.001 - 5 Campbell CL, LLANARTH, NSW 2795 Storing 010.2022.00000248.001 - 3453 Limekilns RD, WATTLE FLAT, NSW 2795 Storing 010.2021.00000136.002 - 234 Gilmour ST, KELSO, NSW 2795 Storing 010.2022.00000266.001 - 6 Toronto ST, KELSO, NSW 2795 Storing 010.2022.00000274.001 - 165 Bentinck ST, BATHURST, NSW 2795 Storing 018.2022.00000097.001 - 20 Granite RI, KELSO, NSW 2795 Storing 010.2022.00000268.001 - 19A Alexander ST, EGLINTON, NSW 2795 Storing 010.2022.00000267.001 - 17 Vincent CR, KELSO, NSW 2795 Storing 010.2022.00000237.001 - 411 Yetholme DR, YETHOLME, NSW 2795 Storing 018.2022.00000104.001 - 13 Cain DR, KELSO, NSW 2795 Storing 018.2022.00000101.001 - 34 Basalt WAY, KELSO, NSW 2795 Storing 010.2022.00000265.001 - 26 Locke ST, RAGLAN, NSW 2795 Storing 010.2022.00000246.001 - 3 Poplar PL, KELSO, NSW 2795 Storing 010.2012.00000547.002 - 38 Vittoria ST, WEST BATHURST, NSW 2795 Storing 018.2022.00000098.001 - 201 Peregrine RD, BILLYWILLINGA, NSW 2795 Storing 010.2022.00000264.001 - 168 Saint Anthonys Creek RD, GLANMIRE, NSW 2795 Storing 010.2022.00000211.001 - 25 Ridgeview CL, WHITE ROCK, NSW 2795 Storing 010.2022.00000270.001 - 65 George ST, BATHURST, NSW 2795 Storing 010.2022.00000259.001 - 42 Ecrates PL, KELSO, NSW 2795 Storing 010.2022.00000236.001 - 141 Mount Rankin RD, MOUNT RANKIN, NSW 2795 Storing 010.2022.00000247.001 - 23A Logan ST, EGLINTON, NSW 2795 Storing 010.2022.00000260.001 - 39 Vale RD, SOUTH BATHURST, NSW 2795 Storing 010.2022.00000210.001 - 9 Darvall DR, KELSO, NSW 2795 Storing 010.2022.00000228.001 - 135 Durham ST, BATHURST, NSW 2795 Storing 018.2022.00000087.001 - 1 Cardinia AVE, ROBIN HILL, NSW 2795 Storing 010.2022.00000250.001 - 27 Bradwardine RD, ROBIN HILL, NSW 2795 Storing 010.2022.00000251.001 - 50 Busby ST, SOUTH BATHURST, NSW 2795 Storing 010.2022.00000256.001 - 95 Blue Ridge DR, WHITE ROCK, NSW 2795 Storing 010.2022.00000226.001 - 195 Burges RD, CALOOLA, NSW 2795 Storing 018.2020.00000269.002 - 7 Gell PL, ABERCROMBIE, NSW 2795 Storing 010.2022.00000143.002 - 17 Darvall DR, KELSO, NSW 2795 Storing 010.2022.00000002.002 - 82 Morrow PL, ROBIN HILL, NSW 2795 Storing 010.2022.00000238.001 - 218 Suttor ST, WINDRADYNE, NSW 2795 Storing 010.2022.00000242.001 - 49 George Thomas CL, THE LAGOON, NSW 2795 Storing 018.2022.00000092.001 - 56 Basalt WAY, KELSO, NSW 2795 Storing 010.2022.00000240.001 - 8 Princes ST, BATHURST, NSW 2795 Storing 010.2022.00000243.001 - Porters LA, YETHOLME, NSW 2795 Storing 010.2021.00000419.002 - 82 Hodges RD, BRUINBUN, NSW 2795 Storing 010.2022.00000241.001 - 379 Eleven Mile DR, EGLINTON, NSW 2795 Collecting ATDIS feed data for dubbo... D, [2022-06-27T01:11:38.683699 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=3 D, [2022-06-27T01:11:44.022559 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=4 D, [2022-06-27T01:11:48.111007 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=5 D, [2022-06-27T01:11:52.269117 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=6 D, [2022-06-27T01:11:56.341031 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=7 W, [2022-06-27T01:11:57.185220 #18] WARN -- : No/invalid pagination, assuming no records/aborting Collecting ATDIS feed data for kiama... D, [2022-06-27T01:12:16.462529 #18] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=3 D, [2022-06-27T01:12:20.372358 #18] DEBUG -- : Fetching Storing 010.2022.00000107.001 - 98 Riverside DR, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000014.002 - 17 Riverleigh AV, GERROA, NSW 2534 Storing 010.2021.00000157.002 - 22 Hoolong AV, KIAMA DOWNS, NSW 2533 Storing 010.2017.00000286.002 - 35 Bele PL, KIAMA, NSW 2533 Storing 010.2022.00000102.001 - 110 Burra Creek RD, JAMBEROO, NSW 2533 Storing 010.2022.00000096.001 - 235 Foxground RD, FOXGROUND, NSW 2534 Storing 010.2022.00000079.001 - 4 Rangoon RD, MINNAMURRA, NSW 2533 Storing 010.2022.00000090.001 - 69 South Kiama DR, KIAMA, NSW 2533 Storing 010.2022.00000105.001 - 4 Hollings DR, KIAMA DOWNS, NSW 2533 Storing 010.2022.00000114.001 - 15 Meehan DR, KIAMA DOWNS, NSW 2533 Storing 010.2015.00000303.005 - 5 Sommerville CL, KIAMA, NSW 2533 Storing 010.2022.00000101.001 - 57 Armstrong AV, GERRINGONG, NSW 2534 Storing 010.2022.00000092.001 - 73 Curramore RD, CURRAMORE, NSW 2533 Storing 010.2021.00000249.002 - 6 Brighton ST, KIAMA, NSW 2533 Storing 010.2022.00000091.001 - 72 Barton DR, KIAMA DOWNS, NSW 2533 Storing 010.2020.00000138.002 - 4 Byrne PL, KIAMA DOWNS, NSW 2533 Storing 010.2022.00000094.001 - 59 Oxley AV, KIAMA DOWNS, NSW 2533 Storing 010.2022.00000073.001 - 90 Renfrew RD, WERRI BEACH, NSW 2534 Storing 010.2022.00000097.001 - 15 Wyalla RD, JAMBEROO, NSW 2533 Storing 010.2021.00000140.003 - 2A South Kiama DR, KIAMA HEIGHTS, NSW 2533 Storing 010.2020.00000010.002 - 80 Barton DR, KIAMA DOWNS, NSW 2533 Storing 010.2022.00000083.001 - 5 Young ST, JAMBEROO, NSW 2533 Storing 010.2022.00000095.001 - 50 Riverside DR, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000188.002 - 187 Fern ST, GERRINGONG, NSW 2534 Collecting ATDIS feed data for upper_hunter... W, [2022-06-27T01:12:26.962161 #18] WARN -- : No/invalid pagination, assuming no records/aborting Collecting ATDIS feed data for armidale... Storing 306392 - 174A Allingham Street, ARMIDALE, NSW 2350 Storing 305902 - 15 Grafton Road, ARMIDALE, NSW 2350 Collecting ATDIS feed data for muswellbrook... D, [2022-06-27T01:12:33.419884 #18] DEBUG -- : Fetching https://datracker.muswellbrook.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2022-06-27&lodgement_date_start=2022-05-28&page=3 W, [2022-06-27T01:12:34.861239 #18] WARN -- : No/invalid pagination, assuming no records/aborting Collecting ATDIS feed data for walgett... walgett: ERROR: 500 Internal Server Error E, [2022-06-27T01:12:39.700321 #18] ERROR -- : 500 Internal Server Error /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' D, [2022-06-27T01:12:39.700488 #18] DEBUG -- : /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:233:in `_follow_redirection' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:233:in `_follow_redirection' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:145:in `follow_redirection' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:145:in `follow_redirection' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:121:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:121:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' scraper.rb:45:in `block in <main>' scraper.rb:39:in `each' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:39:in `<main>' scraper.rb:45:in `block in <main>' scraper.rb:39:in `each' scraper.rb:39:in `<main>' scraper.rb:57:in `<main>': There were earlier errors. See output for details (RuntimeError)

Data

Downloaded 12823 times by openaustralia DevAppProject dcalde eduardomontero muzzie86 mattykn vietbaracon WeaponizedBytes

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (5.67 MB) Use the API

rows 10 / 12430

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to authority_label
010.2019.00000164.001
119 Lochview Farm RD, LOCHIEL, NSW 2549
Installation of above ground swimming pool
2019-06-06
2019-05-07
bega_valley
010.2012.00000402.002
5 Bells PL, DOCTOR GEORGE MOUNTAIN, NSW 2550
Modification Erection of a dwelling & detached garage/studio & carport
2019-06-06
2019-05-07
2019-05-23
2019-06-06
bega_valley
018.2019.00005032.001
10 Willie Wagtail PL, ALSTONVILLE, NSW 2477
PCA CDC - New Dwelling - Single Residence
2019-06-06
2019-05-07
ballina
010.2019.00000283.001
2/188-202 Southern Cross DR, BALLINA, NSW 2478
Change of Use and Alterations to Commercial Building Change of use from a high technology industry to general industry involving the assembly, manufacture and fitting of prosthetic limbs and alterations to the building
2019-06-06
2019-05-07
ballina
010.2019.00000284.001
84 Meridian DR, COOLGARDIE, NSW 2478
New Dwelling - Single Residence Construction of a two storey dwelling house and associated studio as depicted in plans by Dean Poile Designs for 64 Meridian Drive Coolgardie dated 2 May 2019
2019-06-06
2019-05-07
ballina
010.2019.00000139.001
141 Keppel ST, BATHURST, NSW 2795
Stage 1 - Five lot torens title subdivision (four residential lots and one development lot) Stage 2 - Seven lot strata title subdivision
2019-06-06
2019-05-07
bathurst
018.2019.00000051.001
10 Newlands CR, KELSO, NSW 2795
Construction of a Single storey dwelling with attached garage
2019-06-06
2019-05-07
bathurst
010.2019.00000138.001
82 Graham DR, KELSO, NSW 2795
Dual Occupancy and two lot residential subdivision
2019-06-06
2019-05-07
bathurst
010.2019.00000078.001
21 Boyd ST, MINNAMURRA, NSW 2533
dwelling & air conditioning unit
2019-06-06
2019-05-07
kiama
010.2019.00000080.001
133 Headland DR, GERROA, NSW 2534
detached storage shed
2019-06-06
2019-05-07
2019-05-23
2019-06-06
kiama

Statistics

Average successful run time: 5 minutes

Total run time: 6 days

Total cpu time used: about 1 hour

Total disk space used: 6.61 MB

History

  • Auto ran revision fdd72316 and failed .
    85 records added, 81 records removed in the database
  • Auto ran revision fdd72316 and failed .
    84 records added, 84 records removed in the database
  • Auto ran revision fdd72316 and failed .
    90 records added, 90 records removed in the database
  • Auto ran revision fdd72316 and failed .
    92 records added, 91 records removed in the database
  • Auto ran revision fdd72316 and failed .
    97 records added, 88 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

multiple_atdis / scraper.rb