planningalerts-scrapers / multiple_atdis

Scrapes multiple authorities that have ATDIS feed


This is a scraper that runs on Morph. To get started see the documentation

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

Contributors mlandauer dependabot[bot]

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching rake 12.3.3  Fetching concurrent-ruby 1.1.6  Fetching minitest 5.14.1  Installing rake 12.3.3  Installing minitest 5.14.1  Installing concurrent-ruby 1.1.6  Fetching thread_safe 0.3.6  Installing thread_safe 0.3.6  Fetching zeitwerk 2.4.0  Installing zeitwerk 2.4.0  Fetching public_suffix 4.0.5  Installing public_suffix 4.0.5  Fetching ast 2.4.1  Installing ast 2.4.1  Fetching multi_json 1.15.0  Installing multi_json 1.15.0  Fetching http-accept 1.7.0  Fetching unf_ext 0.0.7.7  Installing http-accept 1.7.0  Installing unf_ext 0.0.7.7 with native extensions  Fetching mime-types-data 3.2020.0512  Installing mime-types-data 3.2020.0512  Fetching netrc 0.11.0  Installing netrc 0.11.0  Fetching rgeo 2.1.1  Using bundler 1.15.2  Fetching safe_yaml 1.0.5  Installing rgeo 2.1.1 with native extensions  Installing safe_yaml 1.0.5  Fetching diff-lcs 1.4.4  Installing diff-lcs 1.4.4  Fetching hashdiff 1.0.1  Installing hashdiff 1.0.1  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching parallel 1.19.2  Installing parallel 1.19.2  Fetching rainbow 3.0.0  Installing rainbow 3.0.0  Fetching regexp_parser 1.7.1  Installing regexp_parser 1.7.1  Fetching rexml 3.2.5  Installing rexml 3.2.5  Fetching rspec-support 3.9.3  Installing rspec-support 3.9.3  Fetching ruby-progressbar 1.10.1  Installing ruby-progressbar 1.10.1  Fetching unicode-display_width 1.7.0  Installing unicode-display_width 1.7.0  Fetching sqlite3 1.4.2  Installing sqlite3 1.4.2 with native extensions  Fetching vcr 6.0.0  Installing vcr 6.0.0  Fetching tzinfo 1.2.7  Installing tzinfo 1.2.7  Fetching parser 2.7.1.4  Installing parser 2.7.1.4  Fetching addressable 2.7.0  Installing addressable 2.7.0  Fetching i18n 1.8.5  Fetching mime-types 3.3.1  Installing i18n 1.8.5  Installing mime-types 3.3.1  Fetching crack 0.4.3  Fetching rspec-core 3.9.2  Installing rspec-core 3.9.2  Installing crack 0.4.3  Fetching rspec-expectations 3.9.2  Fetching rspec-mocks 3.9.1  Installing rspec-expectations 3.9.2  Installing rspec-mocks 3.9.1  Fetching unf 0.1.4  Fetching rgeo-geojson 2.1.1  Installing rgeo-geojson 2.1.1  Installing unf 0.1.4  Fetching rubocop-ast 0.2.0  Installing rubocop-ast 0.2.0  Fetching activesupport 6.0.3.2  Installing activesupport 6.0.3.2  Fetching webmock 3.8.3  Installing webmock 3.8.3  Fetching rspec 3.9.0  Installing rspec 3.9.0  Fetching domain_name 0.5.20190701  Installing domain_name 0.5.20190701  Fetching rubocop 0.88.0  Fetching http-cookie 1.0.3  Installing rubocop 0.88.0  Installing http-cookie 1.0.3  Fetching activemodel 6.0.3.2  Installing activemodel 6.0.3.2  Fetching rest-client 2.1.0  Installing rest-client 2.1.0  Fetching atdis 0.5.0  Installing atdis 0.5.0  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching scraperwiki 3.0.2  Installing scraperwiki 3.0.2  Fetching scraperwiki-morph 0.1.1  Installing scraperwiki-morph 0.1.1  Bundle complete! 7 Gemfile dependencies, 51 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  If you are upgrading your Rails application from an older version of Rails:  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  If you are starting a NEW Rails application, you can ignore this notice.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Bundle completed (14.40s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting ATDIS feed data for bega_valley... D, [2021-05-15T08:05:53.026384 #17] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=3 D, [2021-05-15T08:06:05.974120 #17] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=4 D, [2021-05-15T08:06:18.623917 #17] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=5 D, [2021-05-15T08:06:31.333696 #17] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=6 D, [2021-05-15T08:06:44.249469 #17] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=7 D, [2021-05-15T08:06:56.730934 #17] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=8 D, [2021-05-15T08:07:04.731364 #17] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=9 W, [2021-05-15T08:07:05.171146 #17] WARN -- : No/invalid pagination, assuming no records/aborting Storing 010.2021.00000180.001 - 60 Stringybark PL, MILLINGANDI, NSW 2549 Storing 010.2021.00000208.001 - 34 Karuka CL, BALD HILLS, NSW 2549 Storing 010.2021.00000185.001 - 12 White Fox RD, BROADWATER, NSW 2549 Storing 010.2021.00000178.001 - 83 Finucanes LA, BEGA, NSW 2550 Storing 010.2021.00000205.001 - 5 Barrumbarra PL, BEGA, NSW 2550 Storing 010.2021.00000181.001 - 32 Rankins RD, COOLAGOLITE, NSW 2550 Storing 010.2021.00000160.001 - 197 Carp ST, BEGA, NSW 2550 Storing 010.2021.00000203.001 - 90-92 Kameruka ST, BEMBOKA, NSW 2550 Storing 010.2021.00000161.001 - 574 Tathra RD, KALARU, NSW 2550 Storing 010.2021.00000179.001 - 97 Monaro ST, MERIMBULA, NSW 2548 Storing 010.2021.00000173.001 - 41 Bay DR, MOGAREEKA, NSW 2550 Storing 010.2021.00000176.001 - 53 Kowara CR, MERIMBULA, NSW 2548 Storing 010.2021.00000177.001 - 22B Elizabeth PDE, TURA BEACH, NSW 2548 Storing 010.2021.00000202.001 - 3 Gubburra LA, SOUTH WOLUMLA, NSW 2550 Storing 010.2021.00000201.001 - 40 Bega ST, TATHRA, NSW 2550 Storing 010.2021.00000200.001 - 285 South Wolumla RD, SOUTH WOLUMLA, NSW 2550 Storing 010.2021.00000199.001 - 125 Toallo ST, PAMBULA, NSW 2549 Storing 010.2021.00000171.001 - 18 West ST, EDEN, NSW 2551 Storing 010.2021.00000194.001 - Shady LA, WALLAGOOT, NSW 2550 Storing 010.2021.00000198.001 - 244 Khandallah RD, KIAH, NSW 2551 Storing 018.2021.00002051.001 - 36 Auckland ST, CANDELO, NSW 2550 Storing 010.2021.00000154.001 - 23 Alexander DR, BERMAGUI, NSW 2546 Storing 010.2021.00000169.001 - 55 Strudwicks RD, BERMAGUI, NSW 2546 Storing 010.2021.00000165.001 - 122 Murrah ST, BERMAGUI, NSW 2546 Storing 018.2021.00002047.001 - 93 Mulloway CCT, MERIMBULA, NSW 2548 Storing 010.2021.00000196.001 - 20 Bellevue PL, EDEN, NSW 2551 Storing 010.2021.00000195.001 - 4305 Princes HWY, BROADWATER, NSW 2549 Storing 010.2021.00000159.001 - 15 Jones LA, BERMAGUI, NSW 2546 Storing 010.2021.00000193.001 - 17 Wumbalwarra DR, BEGA, NSW 2550 Storing 010.2021.00000166.001 - 1/ Kurrajong CR, KALARU, NSW 2550 Storing 010.2021.00000170.001 - 154 East ST, BEGA, NSW 2550 Storing 010.2021.00000183.001 - 90 Jellat WY, KALARU, NSW 2550 Storing 010.2021.00000162.001 - 7 Jacaranda PL, MIRADOR, NSW 2548 Storing 010.2021.00000168.001 - 173 Upper Brogo RD, VERONA, NSW 2550 Storing 010.2021.00000192.001 - 12B Otway CL, MERIMBULA, NSW 2548 Storing 010.2021.00000191.001 - Turingal Head RD, WALLAGOOT, NSW 2550 Storing 010.2021.00000190.001 - 1118 Nethercote RD, NETHERCOTE, NSW 2549 Storing 010.2021.00000189.001 - 189 Mirador DR, MIRADOR, NSW 2548 Storing 010.2021.00000188.001 - 9 Albacore CR, EDEN, NSW 2551 Storing 010.2021.00000157.001 - 26 Princes HWY, SOUTH PAMBULA, NSW 2549 Storing 010.2021.00000187.001 - 68-76 Main ST, MERIMBULA, NSW 2548 Storing 010.2021.00000186.001 - 216-218 Imlay ST, EDEN, NSW 2551 Storing 010.2021.00000184.001 - Gwainurra GR, PAMBULA BEACH, NSW 2549 Storing 010.2020.00000089.002 - Princes HWY, EDEN, NSW 2551 Storing 010.2021.00000182.001 - 15 Edna DR, TATHRA, NSW 2550 Storing 010.2019.00000113.002 - 3A Elizabeth PDE, TURA BEACH, NSW 2548 Storing 010.2021.00000155.001 - 13 Jacaranda PL, MIRADOR, NSW 2548 Storing 010.2018.00000533.002 - 13 Ochre LA, MILLINGANDI, NSW 2549 Storing 010.2021.00000156.001 - 24 Kurrajong CR, KALARU, NSW 2550 Storing 010.2021.00000152.001 - 192 Mirador DR, MIRADOR, NSW 2548 Storing 010.2021.00000163.001 - 215-217 Imlay ST, EDEN, NSW 2551 Storing 010.2005.00000370.004 - 143 Hergenhans LA, ANGLEDALE, NSW 2550 Storing 010.2021.00000153.001 - 42 Wumbalwarra DR, BEGA, NSW 2550 Storing 010.2021.00000175.001 - 11585 Princes HWY, QUAAMA, NSW 2550 Storing 010.2021.00000172.001 - 2866 Mount Darragh RD, WYNDHAM, NSW 2550 Storing 010.2021.00000174.001 - 19 Dandar RD, BEGA, NSW 2550 Storing 010.2020.00000285.002 - Corkhill PL, BEGA, NSW 2550 Storing 010.2021.00000167.001 - 1346 Tarraganda LA, TARRAGANDA, NSW 2550 Storing 010.2021.00000132.001 - 59 Lamont ST, BERMAGUI, NSW 2546 Storing 010.2021.00000164.001 - 24 Mulloway CCT, MERIMBULA, NSW 2548 Storing 010.2020.00000107.002 - 8 Elliot LA, BEGA, NSW 2550 Storing 010.2021.00000151.001 - 18 Hoskins ST, BEMBOKA, NSW 2550 Storing 010.2021.00000158.001 - 1 Yowaka ST, PAMBULA, NSW 2549 Storing 010.2020.00000480.002 - 10457 Princes HWY, COBARGO, NSW 2550 Storing 018.2021.00002046.001 - 6-8 Andy Poole DR, TATHRA, NSW 2550 Collecting ATDIS feed data for ballina... D, [2021-05-15T08:07:19.963371 #17] DEBUG -- : Fetching http://da.ballina.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=3 D, [2021-05-15T08:07:26.420761 #17] DEBUG -- : Fetching http://da.ballina.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=4 D, [2021-05-15T08:07:31.813463 #17] DEBUG -- : Fetching http://da.ballina.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=5 D, [2021-05-15T08:07:38.523753 #17] DEBUG -- : Fetching http://da.ballina.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=6 D, [2021-05-15T08:07:43.947193 #17] DEBUG -- : Fetching http://da.ballina.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=7 D, [2021-05-15T08:07:48.621824 #17] DEBUG -- : Fetching http://da.ballina.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=8 D, [2021-05-15T08:07:53.746676 #17] DEBUG -- : Fetching http://da.ballina.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=9 W, [2021-05-15T08:07:59.028853 #17] WARN -- : No/invalid pagination, assuming no records/aborting Storing 010.2021.00000288.001 - 46 Red LA, ROUS, NSW 2477 Storing 010.2021.00000342.001 - 166 North Creek RD, LENNOX HEAD, NSW 2478 Storing 018.2021.00005073.001 - 42 Seaside AVE, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000369.001 - 17 River ST, BALLINA, NSW 2478 Storing 010.2021.00000368.001 - 69C Teven RD, ALSTONVILLE, NSW 2477 Storing 010.2021.00000367.001 - 123 Willowbank DR, ALSTONVALE, NSW 2477 Storing 010.2021.00000284.001 - 73 Habitat WY, LENNOX HEAD, NSW 2478 Storing 010.2021.00000340.001 - 7 Higgins PL, CUMBALUM, NSW 2478 Storing 010.2021.00000366.001 - 11 Byron ST, EAST WARDELL, NSW 2477 Storing 010.2021.00000365.001 - 1 Simmons ST, BALLINA, NSW 2478 Storing 010.2021.00000314.001 - 62 Teakwood DR, ALSTONVILLE, NSW 2477 Storing 010.2021.00000337.001 - 4 Mellis CCT, ALSTONVILLE, NSW 2477 Storing 010.2021.00000341.001 - 51 Old Byron Bay RD, NEWRYBAR, NSW 2479 Storing 018.2021.00005027.001 - 24 Surf AVE, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000364.001 - 12 Darcey AVE, CUMBALUM, NSW 2478 Storing 010.2021.00000363.001 - 43 Seaswell CR, LENNOX HEAD, NSW 2478 Storing 010.2021.00000287.001 - 71 Stewart ST, LENNOX HEAD, NSW 2478 Storing 010.2021.00000302.001 - Pimlico RD, PIMLICO, NSW 2478 Storing 010.2021.00000279.001 - 40 Surf AVE, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000293.001 - 16 Surf AVE, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000285.001 - 14 Sand ST, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000339.001 - 41 Headlands DR, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000316.001 - 21 Outrigger RD, LENNOX HEAD, NSW 2478 Storing 010.2021.00000323.001 - 239 Hutley DR, SKENNARS HEAD, NSW 2478 Storing 010.2019.00000284.003 - 84 Meridian DR, COOLGARDIE, NSW 2478 Storing 018.2021.00005020.001 - 16A Liffey AVE, CUMBALUM, NSW 2478 Storing 018.2021.00005039.001 - 64 Koellner RD, CUMBALUM, NSW 2478 Storing 010.2021.00000362.001 - 45 Headlands DR, SKENNARS HEAD, NSW 2478 Storing 018.2021.00005057.001 - 1320 Bruxner HWY, WOLLONGBAR, NSW 2477 Storing 010.2021.00000361.001 - 22 Valley DR, ALSTONVILLE, NSW 2477 Storing 010.2021.00000360.001 - 51 Snapper DR, LENNOX HEAD, NSW 2478 Storing 010.2021.00000359.001 - 2/22 Barrett DR, LENNOX HEAD, NSW 2478 Storing 010.2021.00000326.001 - 83 Dees LA, LYNWOOD, NSW 2477 Storing 010.2021.00000313.001 - 17 Fieldcrest PL, LENNOX HEAD, NSW 2478 Storing 010.2021.00000290.001 - 18 Fenwick DR, EAST BALLINA, NSW 2478 Storing 010.2021.00000009.002 - 6 Habitat WY, LENNOX HEAD, NSW 2478 Storing 010.2019.00000779.002 - 13 Sunray CR, LENNOX HEAD, NSW 2478 Storing 010.2021.00000358.001 - 1/15 Stewart ST, LENNOX HEAD, NSW 2478 Storing 018.2021.00005055.001 - 17 Sand ST, SKENNARS HEAD, NSW 2478 Storing 018.2021.00005071.001 - 11 Claremont PL, LENNOX HEAD, NSW 2478 Storing 018.2021.00005065.001 - 57 Stoneyhurst DR, LENNOX HEAD, NSW 2478 Storing 018.2021.00005072.001 - 29 Surf AVE, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000297.001 - 199 Ellis RD, ALSTONVILLE, NSW 2477 Storing 010.2021.00000357.001 - 8 Allens PDE, LENNOX HEAD, NSW 2478 Storing 010.2021.00000356.001 - 51 Dou-jea LA, LYNWOOD, NSW 2477 Storing 010.2021.00000355.001 - 43 Chickiba DR, EAST BALLINA, NSW 2478 Storing 010.2021.00000334.001 - 21 Dodge LA, LENNOX HEAD, NSW 2478 Storing 010.2021.00000333.001 - 18 Angus Kennedy CL, LENNOX HEAD, NSW 2478 Storing 010.2021.00000354.001 - 66 Habitat WY, LENNOX HEAD, NSW 2478 Storing 010.2021.00000353.001 - 1248 Wardell RD, WARDELL, NSW 2477 Storing 010.2021.00000352.001 - 38 Habitat WY, LENNOX HEAD, NSW 2478 Storing 010.2021.00000351.001 - 32 Surf AVE, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000331.001 - 83 Eyles DR, EAST BALLINA, NSW 2478 Storing 010.2021.00000329.001 - 5 William ST, ALSTONVILLE, NSW 2477 Storing 010.2021.00000101.002 - 4 Sand ST, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000349.001 - 64 Habitat WY, LENNOX HEAD, NSW 2478 Storing 010.2021.00000325.001 - 5 Lakefield AVE, LENNOX HEAD, NSW 2478 Storing 010.2021.00000322.001 - 12 Brooklet RD, NEWRYBAR, NSW 2479 Storing 010.2020.00000083.002 - 169 Tamar ST, BALLINA, NSW 2478 Storing 010.2021.00000319.001 - 28 Fig Tree Hill DR, LENNOX HEAD, NSW 2478 Storing 010.2021.00000350.001 - 901 Wardell RD, MEERSCHAUM VALE, NSW 2477 Storing 010.2021.00000317.001 - 21 McLeans ST, SKENNARS HEAD, NSW 2478 Storing 010.2021.00000338.001 - 276 Friday Hut RD, TINTENBAR, NSW 2478 Storing 010.2021.00000315.001 - 59 Habitat WY, LENNOX HEAD, NSW 2478 Storing 010.2013.00000309.002 - 199 Ellis RD, ALSTONVILLE, NSW 2477 Storing 010.2021.00000335.001 - 313 Marom Creek RD, MEERSCHAUM VALE, NSW 2477 Storing 010.2021.00000332.001 - 267 Alcorn RD, KNOCKROW, NSW 2479 Storing 010.2021.00000321.001 - 72 Habitat WY, LENNOX HEAD, NSW 2478 Storing 010.2021.00000296.001 - 9 Presentation CR, CUMBALUM, NSW 2478 Storing 010.2018.00000154.003 - 14 Casuarina RD, EAST BALLINA, NSW 2478 Storing 010.2020.00000670.002 - 23 Henderson DR, LENNOX HEAD, NSW 2478 Storing 010.2021.00000310.001 - 143 Burnet ST, BALLINA, NSW 2478 Storing 010.2020.00000271.002 - 8 Mellis CCT, ALSTONVILLE, NSW 2477 Storing 010.2021.00000348.001 - 13 Jorgensens LA, BROOKLET, NSW 2479 Storing 010.2021.00000347.001 - Wardell RD, WARDELL, NSW 2477 Storing 010.2019.00000674.002 - 16 Sunray CR, LENNOX HEAD, NSW 2478 Storing 018.2021.00005070.001 - 107 Ballina RD, ALSTONVILLE, NSW 2477 Storing 010.2021.00000311.001 - 94 Chickiba DR, EAST BALLINA, NSW 2478 Collecting ATDIS feed data for bathurst... D, [2021-05-15T08:08:07.244015 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=3 D, [2021-05-15T08:08:10.777626 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=4 D, [2021-05-15T08:08:13.447402 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=5 D, [2021-05-15T08:08:16.558361 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=6 D, [2021-05-15T08:08:19.631892 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=7 D, [2021-05-15T08:08:23.766550 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=8 D, [2021-05-15T08:08:27.483067 #17] DEBUG -- : Fetching Storing 010.2021.00000217.001 - 240 Limekilns RD, KELSO, NSW 2795 Storing 010.2021.00000214.001 - 38 Ironstone AVE, WHITE ROCK, NSW 2795 Storing 010.2021.00000212.001 - 33 Rocket ST, BATHURST, NSW 2795 Storing 010.2021.00000208.001 - 13 Bathurst ST, PERTHVILLE, NSW 2795 Storing 010.2021.00000207.001 - 68 Mendel DR, KELSO, NSW 2795 Storing 010.2021.00000174.001 - 53 Parer RD, ABERCROMBIE, NSW 2795 Storing 018.2021.00000114.001 - 100 Graham DR, KELSO, NSW 2795 Storing 010.2021.00000166.001 - 15 Connolly DR, KELSO, NSW 2795 Storing 018.2021.00000133.001 - 13 Darvall DR, KELSO, NSW 2795 Storing 010.2021.00000194.001 - 39 Clairvaux LA, KELSO, NSW 2795 Storing 010.2021.00000206.001 - 169 Blue Ridge DR, WHITE ROCK, NSW 2795 Storing 010.2021.00000171.001 - Boundary RD, MITCHELL, NSW 2795 Storing 010.2021.00000145.001 - 13 Burlington RI, KELSO, NSW 2795 Storing 010.2021.00000170.001 - 43 Logan ST, EGLINTON, NSW 2795 Storing 010.2019.00000312.002 - 16 Russell ST, GORMANS HILL, NSW 2795 Storing 010.2021.00000200.001 - 45 Ridgeview CL, WHITE ROCK, NSW 2795 Storing 010.2021.00000176.001 - 2 Patna ST, WEST BATHURST, NSW 2795 Storing 010.2021.00000216.001 - 190 James White DR, FOSTERS VALLEY, NSW 2795 Storing 010.2021.00000210.001 - 933 Freemantle RD, BILLYWILLINGA, NSW 2795 Storing 010.2021.00000167.001 - 11 Sunbright RD, KELSO, NSW 2795 Storing 010.2021.00000196.001 - 25 Denison ST, SOFALA, NSW 2795 Storing 018.2021.00000120.001 - 139 Hughes ST, KELSO, NSW 2795 Storing 018.2021.00000124.001 - 98 Graham DR, KELSO, NSW 2795 Storing 010.2021.00000149.001 - 100 College RD, SOUTH BATHURST, NSW 2795 Storing 010.2021.00000203.001 - Mitchell HWY, VITTORIA, NSW 2799 Storing 018.2021.00000128.001 - 3 Cain DR, KELSO, NSW 2795 Storing 018.2021.00000140.001 - 9 Darvall DR, KELSO, NSW 2795 Storing 010.2021.00000190.001 - 7 Keppel ST, BATHURST, NSW 2795 Storing 010.2021.00000188.001 - 14 Loren ST, EGLINTON, NSW 2795 Storing 010.2021.00000199.001 - 5 Bullock PL, KELSO, NSW 2795 Storing 010.2018.00000340.002 - 14 Ingersole DR, KELSO, NSW 2795 Storing 010.2021.00000204.001 - 220 Rankin ST, BATHURST, NSW 2795 Storing 010.2021.00000201.001 - 70 Mendel DR, KELSO, NSW 2795 Storing 010.2021.00000183.001 - 350 Molybdonite RD, LOCKSLEY, NSW 2795 Storing 010.2021.00000186.001 - 17 Darvall DR, KELSO, NSW 2795 Storing 010.2021.00000139.001 - 40 William ST, BATHURST, NSW 2795 Storing 010.2021.00000153.001 - 19 Connolly DR, KELSO, NSW 2795 Storing 010.2021.00000187.001 - 48 James Barnet DR, KELSO, NSW 2795 Storing 018.2021.00000121.001 - 20 Darvall DR, KELSO, NSW 2795 Storing 010.2021.00000195.001 - 316 Eusdale RD, YETHOLME, NSW 2795 Storing 018.2021.00000107.001 - 12 Darvall DR, KELSO, NSW 2795 Storing 018.2021.00000119.001 - 71 Swanbrooke ST, WINDRADYNE, NSW 2795 Storing 018.2021.00000129.001 - 40 William ST, BATHURST, NSW 2795 Storing 010.2021.00000168.001 - 12 Darvall DR, KELSO, NSW 2795 Storing 018.2021.00000132.001 - 4 Wallace WAY, KELSO, NSW 2795 Storing 010.2021.00000197.001 - 17 Suttor ST, WEST BATHURST, NSW 2795 Storing 010.2021.00000179.001 - 91 Rankin ST, BATHURST, NSW 2795 Storing 010.2021.00000180.001 - 141 Hughes ST, KELSO, NSW 2795 Storing 018.2021.00000127.001 - 11 Darvall DR, KELSO, NSW 2795 Storing 018.2021.00000126.001 - 149 Hughes ST, KELSO, NSW 2795 Storing 010.2021.00000169.001 - 57 George Thomas CL, THE LAGOON, NSW 2795 Storing 010.2021.00000192.001 - 73 Nile ST, RAGLAN, NSW 2795 Storing 018.2021.00000118.001 - 143 Hughes ST, KELSO, NSW 2795 Storing 010.2021.00000191.001 - 1595 Mitchell HWY, THE ROCKS, NSW 2795 Storing 010.2021.00000181.001 - 392 Howick ST, BATHURST, NSW 2795 Storing 018.2021.00000117.001 - 10 Sunbright RD, KELSO, NSW 2795 Storing 018.2021.00000123.001 - 13 Wellesley CT, RAGLAN, NSW 2795 Storing 010.2021.00000164.001 - 3 Sapphire CR, KELSO, NSW 2795 Storing 010.2021.00000185.001 - 1396 Ophir RD, ROCK FOREST, NSW 2795 Storing 010.2021.00000182.001 - 8 Elphinstone PL, WINDRADYNE, NSW 2795 Storing 010.2021.00000157.001 - Tambaroora ST, HILL END, NSW 2850 Storing 018.2021.00000122.001 - 210 Howick ST, BATHURST, NSW 2795 Storing 018.2021.00000103.001 - 8 Pryce PDE, ABERCROMBIE, NSW 2795 Storing 010.2021.00000184.001 - 37 Keppel ST, BATHURST, NSW 2795 Storing 010.2021.00000156.001 - 30 Hassall GR, KELSO, NSW 2795 Storing 010.2021.00000173.001 - 48 Durham ST, BATHURST, NSW 2795 Storing 018.2021.00000116.001 - 69 Newlands CR, KELSO, NSW 2795 Storing 018.2021.00000104.001 - 12 Sunbright RD, KELSO, NSW 2795 Storing 010.2021.00000178.001 - 2 Kirkland RD, KELSO, NSW 2795 Storing 010.2021.00000177.001 - 297 Howards DR, MOUNT RANKIN, NSW 2795 Storing 018.2021.00000108.001 - 7 Driscoll CL, KELSO, NSW 2795 Storing 018.2021.00000115.001 - 34 Ironstone AVE, WHITE ROCK, NSW 2795 Storing 010.2021.00000163.001 - 1557 Rockley RD, FOSTERS VALLEY, NSW 2795 Storing 010.2021.00000165.001 - 41 Hampden Park RD, KELSO, NSW 2795 Collecting ATDIS feed data for berrigan... berrigan: ERROR: 500 Internal Server Error E, [2021-05-15T08:08:29.041030 #17] ERROR -- : 500 Internal Server Error /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' D, [2021-05-15T08:08:29.041176 #17] DEBUG -- : /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' scraper.rb:51:in `block in <main>' scraper.rb:45:in `each' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `<main>' scraper.rb:51:in `block in <main>' scraper.rb:45:in `each' scraper.rb:45:in `<main>' Collecting ATDIS feed data for dubbo... D, [2021-05-15T08:08:41.005066 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=3 D, [2021-05-15T08:08:45.813860 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=4 D, [2021-05-15T08:08:51.343820 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=5 D, [2021-05-15T08:08:55.023574 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=6 D, [2021-05-15T08:08:58.910109 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=7 D, [2021-05-15T08:09:02.502848 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=8 D, [2021-05-15T08:09:06.503872 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=9 D, [2021-05-15T08:09:09.632980 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=10 D, [2021-05-15T08:09:12.867751 #17] DEBUG -- : Fetching Storing 016.2021.00000066.001 - 38 Violet CRCT, DUBBO, NSW 2830 Storing 010.2021.00000301.001 - 2 Monash ST, DUBBO, NSW 2830 Storing 010.2021.00000296.001 - 82-84 Wellington ST, GEURIE, NSW 2818 Storing 010.2021.00000307.001 - 26 Evans DR, MOUNT ARTHUR, NSW 2820 Storing 010.2021.00000304.001 - 9R Mendooran RD, DUBBO, NSW 2830 Storing 010.2021.00000303.001 - 55 Castlereagh AVE, DUBBO, NSW 2830 Storing 016.2021.00000056.001 - 26L North Burrabadine RD, DUBBO, NSW 2830 Storing 010.2021.00000305.001 - 8 Gasnier PL, DUBBO, NSW 2830 Storing 010.2021.00000279.001 - 3R Barbigal ST, WONGARBON, NSW 2831 Storing 010.2021.00000298.001 - 141 Boundary RD, DUBBO, NSW 2830 Storing 010.2021.00000276.001 - 10B Bundemar ST, WONGARBON, NSW 2831 Storing 010.2021.00000300.001 - 15 Asset WY, DUBBO, NSW 2830 Storing 016.2021.00000063.001 - 17 Scope CRCT, DUBBO, NSW 2830 Storing 016.2021.00000062.001 - 22 Delta RD, DUBBO, NSW 2830 Storing 010.2021.00000254.001 - 4 Endeavour CL, DUBBO, NSW 2830 Storing 010.2021.00000302.001 - 48 Azure AVE, DUBBO, NSW 2830 Storing 010.2021.00000223.001 - 17 Gisborne LA, WELLINGTON, NSW 2820 Storing 010.2021.00000297.001 - 11 Scope CRCT, DUBBO, NSW 2830 Storing 010.2021.00000294.001 - 29 Allison ST, DUBBO, NSW 2830 Storing 010.2021.00000269.001 - 705 Ponto Falls RD, MARYVALE, NSW 2820 Storing 010.2021.00000295.001 - 4 Cudgegong PL, DUBBO, NSW 2830 Storing 016.2021.00000053.001 - 37 Peel PL, DUBBO, NSW 2830 Storing 016.2021.00000065.001 - 177-191 Macquarie ST, DUBBO, NSW 2830 Storing 010.2021.00000292.001 - 5 Tallwoods GR, DUBBO, NSW 2830 Storing 010.2021.00000293.001 - 1 Targa AVE, DUBBO, NSW 2830 Storing 010.2021.00000289.001 - Boundary RD, DUBBO, NSW 2830 Storing 010.2021.00000291.001 - 46 Sterling ST, DUBBO, NSW 2830 Storing 010.2021.00000288.001 - 32 East ST, DUBBO, NSW 2830 Storing 010.2021.00000231.001 - 5 Macintyre CRCT, DUBBO, NSW 2830 Storing 010.2021.00000286.001 - 12 Old Homestead DR, DUBBO, NSW 2830 Storing 010.2021.00000290.001 - 2 St Andrews DR, DUBBO, NSW 2830 Storing 010.2021.00000287.001 - 10 Belmore ST, DUBBO, NSW 2830 Storing 010.2021.00000247.001 - 33 Scope CRCT, DUBBO, NSW 2830 Storing 010.2021.00000278.001 - 29 Taylor ST, DUBBO, NSW 2830 Storing 016.2021.00000064.001 - 10 Yarra PL, DUBBO, NSW 2830 Storing 010.2021.00000263.001 - 10 Waterway CRES, DUBBO, NSW 2830 Storing 010.2021.00000273.001 - 381 Macquarie ST, DUBBO, NSW 2830 Storing 010.2021.00000275.001 - Boundary RD, DUBBO, NSW 2830 Storing 010.2021.00000272.001 - 51 Waterway CRES, DUBBO, NSW 2830 Storing 010.2021.00000270.001 - 233 Darling ST, DUBBO, NSW 2830 Storing 010.2021.00000284.001 - 33 Gisborne ST, WELLINGTON, NSW 2820 Storing 010.2021.00000285.001 - 4 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000282.001 - 7 Narrabeen PL, DUBBO, NSW 2830 Storing 016.2021.00000055.001 - 32-34 Church ST, DUBBO, NSW 2830 Storing 016.2021.00000057.001 - 31 Nicholii ST, MONTEFIORES, NSW 2820 Storing 016.2021.00000059.001 - 25 Hindmarsh ESP, DUBBO, NSW 2830 Storing 010.2021.00000277.001 - 12 Rhyana CT, DUBBO, NSW 2830 Storing 010.2021.00000266.001 - 13 Falconer WY, DUBBO, NSW 2830 Storing 010.2021.00000281.001 - 8 Fuzzy Box RD, GEURIE, NSW 2818 Storing 016.2020.00000014.003 - 7 Freshwater DR, DUBBO, NSW 2830 Storing 016.2021.00000061.001 - 177-191 Macquarie ST, DUBBO, NSW 2830 Storing 010.2021.00000253.001 - 92 Cobra ST, DUBBO, NSW 2830 Storing 010.2021.00000259.001 - 27R Whitewood RD, DUBBO, NSW 2830 Storing 010.2021.00000283.001 - 39 Montefiores ST, MONTEFIORES, NSW 2820 Storing 010.2021.00000271.001 - 58 Durum CRCT, DUBBO, NSW 2830 Storing 016.2021.00000060.001 - 16 Delta RD, DUBBO, NSW 2830 Storing 016.2021.00000058.001 - 8 Nicholii ST, MONTEFIORES, NSW 2820 Storing 010.2021.00000268.001 - 28 Regand Park BVD, DUBBO, NSW 2830 Storing 010.2021.00000248.001 - 95 Montefiores ST, MONTEFIORES, NSW 2820 Storing 010.2019.00000027.002 - 125 Cobra ST, DUBBO, NSW 2830 Storing 010.2021.00000264.001 - 10 Asset WY, DUBBO, NSW 2830 Storing 010.2021.00000280.001 - 20 Homestead DR, DUBBO, NSW 2830 Storing 010.2021.00000267.001 - 60 Durum CRCT, DUBBO, NSW 2830 Storing 010.2021.00000274.001 - 12 Springfield WY, DUBBO, NSW 2830 Storing 010.2021.00000234.001 - 36 Braun AVE, DUBBO, NSW 2830 Storing 010.2021.00000261.001 - 4 Wirraway CL, DUBBO, NSW 2830 Storing 010.2021.00000262.001 - 17 Delta RD, DUBBO, NSW 2830 Storing 010.2008.00000349.006 - Minore RD, DUBBO, NSW 2830 Storing 010.2021.00000265.001 - 66 Durum CRCT, DUBBO, NSW 2830 Storing 010.2021.00000260.001 - 94-98 Wellington ST, GEURIE, NSW 2818 Storing 010.2021.00000255.001 - 301 Darling ST, DUBBO, NSW 2830 Storing 010.2021.00000256.001 - 51L Minore RD, DUBBO, NSW 2830 Storing 010.2021.00000238.001 - 36 Sapphire ST, DUBBO, NSW 2830 Storing 010.2021.00000257.001 - 12 Melaleuca DR, MONTEFIORES, NSW 2820 Storing 010.2021.00000252.001 - 9 Waterway CRES, DUBBO, NSW 2830 Storing 010.2021.00000258.001 - 2661 Renshaw McGirr WY, WALMER, NSW 2820 Storing 010.2021.00000246.001 - 38 Howard AVE, DUBBO, NSW 2830 Storing 016.2021.00000054.001 - 12 Waterway CRES, DUBBO, NSW 2830 Storing 016.2021.00000046.001 - 4L Euston LA, DUBBO, NSW 2830 Storing 010.2021.00000245.001 - 84 Sheraton RD, DUBBO, NSW 2830 Storing 010.2021.00000251.001 - 101 Caves RD, APSLEY, NSW 2820 Storing 010.2021.00000250.001 - Barbigal RD, WONGARBON, NSW 2831 Storing 016.2021.00000052.001 - 8 Waterway CRES, DUBBO, NSW 2830 Storing 010.2021.00000249.001 - 4 Mary ST, DUBBO, NSW 2830 Storing 016.2021.00000051.001 - 20 Scope CRCT, DUBBO, NSW 2830 Storing 016.2021.00000050.001 - 64 Durum CRCT, DUBBO, NSW 2830 Storing 016.2021.00000049.001 - 33 Scope CRCT, DUBBO, NSW 2830 Storing 016.2021.00000048.001 - 6 Targa AVE, DUBBO, NSW 2830 Storing 016.2021.00000047.001 - 16 Scope CRCT, DUBBO, NSW 2830 Storing 010.2021.00000239.001 - 51 Bultje ST, DUBBO, NSW 2830 Storing 010.2021.00000229.001 - 67-69 Percy ST, WELLINGTON, NSW 2820 Storing 010.2021.00000244.001 - 26 Philipsons LA, MARYVALE, NSW 2820 Storing 010.2021.00000243.001 - 77 River ST, DUBBO, NSW 2830 Storing 010.2021.00000242.001 - 148L Coolbaggie Forest RD, EUMUNGERIE, NSW 2822 Collecting ATDIS feed data for kiama... D, [2021-05-15T08:09:31.330782 #17] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=3 D, [2021-05-15T08:09:39.990220 #17] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=4 D, [2021-05-15T08:09:48.031835 #17] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=5 D, [2021-05-15T08:09:51.351054 #17] DEBUG -- : Fetching Storing 010.2018.00000262.002 - 97 Attunga AV, KIAMA HEIGHTS, NSW 2533 Storing 010.2021.00000108.001 - Churchill ST, JAMBEROO, NSW 2533 Storing 010.2021.00000094.001 - 4/154 Charles AV, MINNAMURRA, NSW 2533 Storing 010.2021.00000084.001 - 7 Toorak PL, GERRINGONG, NSW 2534 Storing 010.2021.00000105.001 - 44 Headland DR, GERROA, NSW 2534 Storing 010.2021.00000103.001 - 22 O'Mara PL, JAMBEROO, NSW 2533 Storing 010.2021.00000099.001 - 4 Osborne ST, GERRINGONG, NSW 2534 Storing 010.2019.00000094.003 - 55 Surfleet PL, KIAMA, NSW 2533 Storing 010.2021.00000031.001 - 45 Allowrie ST, JAMBEROO, NSW 2533 Storing 010.2020.00000034.002 - 29 Anembo CR, KIAMA HEIGHTS, NSW 2533 Storing 010.2020.00000171.002 - 189 Jerrara RD, JERRARA, NSW 2533 Storing 010.2021.00000104.001 - 44 Stafford ST, GERROA, NSW 2534 Storing 010.2021.00000100.001 - 34 Halls RD, JERRARA, NSW 2533 Storing 010.2021.00000101.001 - 38 Hughes CR, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000096.001 - 39 Gibson CR, JAMBEROO, NSW 2533 Storing 010.2021.00000102.001 - 103 Charles AV, MINNAMURRA, NSW 2533 Storing 010.2021.00000093.001 - Shoalhaven ST, KIAMA, NSW 2533 Storing 010.2021.00000097.001 - 46 Surfleet PL, KIAMA, NSW 2533 Storing 010.2021.00000095.001 - 25 Oxley AV, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000071.001 - 129 Riverside DR, KIAMA DOWNS, NSW 2533 Storing 010.2018.00000235.002 - 56 Lilly Pilly WY, KIAMA, NSW 2533 Storing 010.2021.00000085.001 - 59 Stafford ST, GERROA, NSW 2534 Storing 010.2016.00000158.003 - 9 Hothersal ST, KIAMA, NSW 2533 Storing 010.2021.00000091.001 - 2/24 Collins ST, KIAMA, NSW 2533 Storing 010.2011.00000075.003 - 124 Factory LANE, JAMBEROO, NSW 2533 Storing 010.2021.00000082.001 - 11 Caliope ST, KIAMA, NSW 2533 Storing 010.2021.00000078.001 - 140A Charles AV, MINNAMURRA, NSW 2533 Storing 010.2021.00000076.001 - 19 Oxley AV, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000070.001 - 68 Charles AV, MINNAMURRA, NSW 2533 Storing 010.2021.00000073.001 - 184 North Kiama DR, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000090.001 - 15 Northpoint PL, BOMBO, NSW 2533 Storing 010.2021.00000089.001 - 125 Willowvale RD, WILLOW VALE, NSW 2534 Storing 010.2021.00000088.001 - 5A Noorinan ST, KIAMA, NSW 2533 Storing 010.2021.00000081.001 - 5 Links ST, MINNAMURRA, NSW 2533 Storing 010.2021.00000086.001 - 144 Curramore RD, CURRAMORE, NSW 2533 Storing 010.2021.00000068.001 - Irvine ST, KIAMA, NSW 2533 Storing 010.2021.00000083.001 - 131 Riverside DR, KIAMA DOWNS, NSW 2533 Storing 010.2017.00000363.002 - 141 Attunga AV, KIAMA HEIGHTS, NSW 2533 Storing 010.2021.00000075.001 - 59 Stafford ST, GERROA, NSW 2534 Storing 010.2021.00000087.001 - 2 Sandy Wha RD, GERRINGONG, NSW 2534 Storing 010.2021.00000077.001 - 24 Lilly Pilly WY, KIAMA, NSW 2533 Storing 010.2020.00000022.002 - 39 Werri ST, WERRI BEACH, NSW 2534 Storing 010.2019.00000153.002 - 10 Devonshire ST, KIAMA, NSW 2533 Collecting ATDIS feed data for upper_hunter... W, [2021-05-15T08:09:54.324427 #17] WARN -- : No/invalid pagination, assuming no records/aborting Collecting ATDIS feed data for armidale... Storing 287509 - 11136 New England Highway, ARMIDALE, NSW 2350 Collecting ATDIS feed data for leeton... leeton: ERROR: 500 Internal Server Error E, [2021-05-15T08:10:05.975233 #17] ERROR -- : 500 Internal Server Error /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' D, [2021-05-15T08:10:05.975344 #17] DEBUG -- : /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' scraper.rb:51:in `block in <main>' scraper.rb:45:in `each' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `<main>' scraper.rb:51:in `block in <main>' scraper.rb:45:in `each' scraper.rb:45:in `<main>' Collecting ATDIS feed data for muswellbrook... D, [2021-05-15T08:10:08.894515 #17] DEBUG -- : Fetching https://datracker.muswellbrook.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-05-15&lodgement_date_start=2021-04-15&page=3 D, [2021-05-15T08:10:10.003156 #17] DEBUG -- : Fetching Storing 005.2021.00000048.001 - Bridge ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000049.001 - 25 Babbler CR, MUSCLE CREEK, NSW 2333 Storing 005.2021.00000050.001 - 106B Maitland ST, MUSWELLBROOK, NSW 2333 Storing 005.2019.00000094.002 - 133 Hill ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000047.001 - 48 Stockyard PDE, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000045.001 - 20 Dorset RD, KAYUGA, NSW 2333 Storing 005.2021.00000044.001 - 299 Merriwa RD, DENMAN, NSW 2328 Storing 005.2021.00000043.001 - Peberdys RD, SANDY HOLLOW, NSW 2333 Storing 005.2021.00000042.001 - 35 Kingfisher CT, MUSCLE CREEK, NSW 2333 Storing 005.2021.00000040.001 - 54 Cook ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000041.001 - 12 Brook ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000037.001 - 130 Bureen RD, DENMAN, NSW 2328 Storing 017.2021.00000012.001 - 45 Scott ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000035.001 - 5 Jersey PL, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000036.001 - 55 Ford ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000033.001 - Sandy Creek RD, MCCULLYS GAP, NSW 2333 Storing 005.2021.00000034.001 - 84 Palace ST, DENMAN, NSW 2328 Storing 005.2021.00000031.001 - 1 Jeans ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000032.001 - 41 Maitland ST, MUSWELLBROOK, NSW 2333 Collecting ATDIS feed data for walgett... walgett: ERROR: 500 Internal Server Error E, [2021-05-15T08:10:13.360783 #17] ERROR -- : 500 Internal Server Error /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' D, [2021-05-15T08:10:13.360888 #17] DEBUG -- : /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' scraper.rb:51:in `block in <main>' scraper.rb:45:in `each' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `<main>' scraper.rb:51:in `block in <main>' scraper.rb:45:in `each' scraper.rb:45:in `<main>' scraper.rb:63:in `<main>': There were earlier errors. See output for details (RuntimeError)

Data

Downloaded 9231 times by openaustralia DevAppProject dcalde eduardomontero mattykn muzzie86 vietbaracon

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (4.18 MB) Use the API

rows 10 / 9030

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to authority_label
010.2019.00000164.001
119 Lochview Farm RD, LOCHIEL, NSW 2549
Installation of above ground swimming pool
2019-06-06
2019-05-07
bega_valley
010.2012.00000402.002
5 Bells PL, DOCTOR GEORGE MOUNTAIN, NSW 2550
Modification Erection of a dwelling & detached garage/studio & carport
2019-06-06
2019-05-07
2019-05-23
2019-06-06
bega_valley
018.2019.00005032.001
10 Willie Wagtail PL, ALSTONVILLE, NSW 2477
PCA CDC - New Dwelling - Single Residence
2019-06-06
2019-05-07
ballina
010.2019.00000283.001
2/188-202 Southern Cross DR, BALLINA, NSW 2478
Change of Use and Alterations to Commercial Building Change of use from a high technology industry to general industry involving the assembly, manufacture and fitting of prosthetic limbs and alterations to the building
2019-06-06
2019-05-07
ballina
010.2019.00000284.001
84 Meridian DR, COOLGARDIE, NSW 2478
New Dwelling - Single Residence Construction of a two storey dwelling house and associated studio as depicted in plans by Dean Poile Designs for 64 Meridian Drive Coolgardie dated 2 May 2019
2019-06-06
2019-05-07
ballina
010.2019.00000139.001
141 Keppel ST, BATHURST, NSW 2795
Stage 1 - Five lot torens title subdivision (four residential lots and one development lot) Stage 2 - Seven lot strata title subdivision
2019-06-06
2019-05-07
bathurst
018.2019.00000051.001
10 Newlands CR, KELSO, NSW 2795
Construction of a Single storey dwelling with attached garage
2019-06-06
2019-05-07
bathurst
010.2019.00000138.001
82 Graham DR, KELSO, NSW 2795
Dual Occupancy and two lot residential subdivision
2019-06-06
2019-05-07
bathurst
010.2019.00000078.001
21 Boyd ST, MINNAMURRA, NSW 2533
dwelling & air conditioning unit
2019-06-06
2019-05-07
kiama
010.2019.00000080.001
133 Headland DR, GERROA, NSW 2534
detached storage shed
2019-06-06
2019-05-07
2019-05-23
2019-06-06
kiama

Statistics

Average successful run time: 6 minutes

Total run time: 5 days

Total cpu time used: 30 minutes

Total disk space used: 6.23 MB

History

  • Auto ran revision b60b2d43 and failed .
    374 records added, 363 records removed in the database
  • Auto ran revision b60b2d43 and failed .
    378 records added, 369 records removed in the database
  • Auto ran revision b60b2d43 and failed .
    377 records added, 360 records removed in the database
  • Auto ran revision b60b2d43 and failed .
    373 records added, 364 records removed in the database
  • Manually ran revision b60b2d43 and failed .
    357 records added, 349 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

multiple_atdis / scraper.rb