planningalerts-scrapers / multiple_atdis

Scrapes multiple authorities that have ATDIS feed


This is a scraper that runs on Morph. To get started see the documentation

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

Contributors mlandauer dependabot[bot]

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching rake 12.3.3  Fetching concurrent-ruby 1.1.6  Fetching minitest 5.14.1  Installing minitest 5.14.1  Installing rake 12.3.3  Installing concurrent-ruby 1.1.6  Fetching thread_safe 0.3.6  Installing thread_safe 0.3.6  Fetching zeitwerk 2.4.0  Fetching public_suffix 4.0.5  Installing zeitwerk 2.4.0  Installing public_suffix 4.0.5  Fetching ast 2.4.1  Installing ast 2.4.1  Fetching multi_json 1.15.0  Installing multi_json 1.15.0  Fetching http-accept 1.7.0  Fetching unf_ext 0.0.7.7  Installing http-accept 1.7.0  Fetching mime-types-data 3.2020.0512  Installing unf_ext 0.0.7.7 with native extensions  Installing mime-types-data 3.2020.0512  Fetching netrc 0.11.0  Installing netrc 0.11.0  Fetching rgeo 2.1.1  Installing rgeo 2.1.1 with native extensions  Using bundler 1.15.2  Fetching safe_yaml 1.0.5  Installing safe_yaml 1.0.5  Fetching diff-lcs 1.4.4  Installing diff-lcs 1.4.4  Fetching hashdiff 1.0.1  Installing hashdiff 1.0.1  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching parallel 1.19.2  Installing parallel 1.19.2  Fetching rainbow 3.0.0  Installing rainbow 3.0.0  Fetching regexp_parser 1.7.1  Installing regexp_parser 1.7.1  Fetching rexml 3.2.5  Installing rexml 3.2.5  Fetching rspec-support 3.9.3  Installing rspec-support 3.9.3  Fetching ruby-progressbar 1.10.1  Installing ruby-progressbar 1.10.1  Fetching unicode-display_width 1.7.0  Installing unicode-display_width 1.7.0  Fetching sqlite3 1.4.2  Installing sqlite3 1.4.2 with native extensions  Fetching vcr 6.0.0  Installing vcr 6.0.0  Fetching tzinfo 1.2.7  Installing tzinfo 1.2.7  Fetching parser 2.7.1.4  Installing parser 2.7.1.4  Fetching addressable 2.7.0  Installing addressable 2.7.0  Fetching i18n 1.8.5  Installing i18n 1.8.5  Fetching mime-types 3.3.1  Installing mime-types 3.3.1  Fetching crack 0.4.3  Fetching rspec-core 3.9.2  Installing crack 0.4.3  Installing rspec-core 3.9.2  Fetching rspec-expectations 3.9.2  Installing rspec-expectations 3.9.2  Fetching rspec-mocks 3.9.1  Installing rspec-mocks 3.9.1  Fetching unf 0.1.4  Fetching rgeo-geojson 2.1.1  Installing rgeo-geojson 2.1.1  Installing unf 0.1.4  Fetching activesupport 6.0.3.2  Installing activesupport 6.0.3.2  Fetching rubocop-ast 0.2.0  Installing rubocop-ast 0.2.0  Fetching webmock 3.8.3  Installing webmock 3.8.3  Fetching rspec 3.9.0  Installing rspec 3.9.0  Fetching domain_name 0.5.20190701  Fetching rubocop 0.88.0  Installing domain_name 0.5.20190701  Installing rubocop 0.88.0  Fetching activemodel 6.0.3.2  Installing activemodel 6.0.3.2  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching rest-client 2.1.0  Installing rest-client 2.1.0  Fetching atdis 0.5.0  Installing atdis 0.5.0  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching scraperwiki 3.0.2  Installing scraperwiki 3.0.2  Fetching scraperwiki-morph 0.1.1  Installing scraperwiki-morph 0.1.1  Bundle complete! 7 Gemfile dependencies, 51 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  If you are upgrading your Rails application from an older version of Rails:  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  If you are starting a NEW Rails application, you can ignore this notice.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Bundle completed (14.54s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting ATDIS feed data for bega_valley... D, [2021-09-24T10:05:22.332104 #18] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=3 D, [2021-09-24T10:05:37.654632 #18] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=4 D, [2021-09-24T10:05:49.001567 #18] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=5 D, [2021-09-24T10:06:01.360155 #18] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=6 D, [2021-09-24T10:06:14.779913 #18] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=7 D, [2021-09-24T10:06:27.037917 #18] DEBUG -- : Fetching http://datracker.begavalley.nsw.gov.au/ATDIS/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=8 W, [2021-09-24T10:06:27.483255 #18] WARN -- : No/invalid pagination, assuming no records/aborting Storing 010.2020.00000394.002 - 164 Mirador DR, MIRADOR, NSW 2548 Storing 010.2021.00001177.001 - 126 Golf CCT, TURA BEACH, NSW 2548 Storing 010.2021.00001151.001 - 1187 Tathra Bermagui RD, TANJA, NSW 2550 Storing 010.2020.00000169.003 - 13877 Princes HWY, ANGLEDALE, NSW 2550 Storing 010.2021.00001147.001 - 53 Lomandra AVE, KALARU, NSW 2550 Storing 010.2021.00001140.001 - 32 The Crest, MIRADOR, NSW 2548 Storing 010.2021.00001173.001 - She Oak CL, BEGA, NSW 2550 Storing 018.2021.00002114.001 - 145 Headland DR, TURA BEACH, NSW 2548 Storing 010.2021.00001174.001 - 17B Tillabudgery CT, BERMAGUI, NSW 2546 Storing 010.2021.00000041.002 - 21 John CL, MERIMBULA, NSW 2548 Storing 010.2020.00000048.002 - 18 Monaro ST, MERIMBULA, NSW 2548 Storing 010.2021.00001167.001 - 107 Bournda Park WY, WALLAGOOT, NSW 2550 Storing 010.2019.00000111.002 - 14 Lamont ST, BERMAGUI, NSW 2546 Storing 010.2021.00001175.001 - 102 Barrabooka North RD, TANJA, NSW 2550 Storing 010.2021.00001122.001 - 68 Link RD, WANDELLA, NSW 2550 Storing 010.2021.00001164.001 - 75 KB Timms DR, EDEN, NSW 2551 Storing 010.2018.00000506.002 - 131 Strathmore CR, KALARU, NSW 2550 Storing 010.2021.00001166.001 - 3 Golf CCT, TURA BEACH, NSW 2548 Storing 010.2020.00000387.002 - Racecourse RD, BERMAGUI, NSW 2546 Storing 018.2021.00002113.001 - 1/6C Surf CIR, TURA BEACH, NSW 2548 Storing 010.2021.00001146.001 - 200 Mirador DR, MIRADOR, NSW 2548 Storing 010.2021.00001152.001 - 3 Lisa CT, MERIMBULA, NSW 2548 Storing 010.2019.00000429.002 - 32-42 Bullara ST, PAMBULA, NSW 2549 Storing 010.2019.00000359.002 - 371 Arthur Kaine DR, MERIMBULA, NSW 2548 Storing 010.2019.00000359.003 - 371 Arthur Kaine DR, MERIMBULA, NSW 2548 Storing 010.2021.00001145.001 - 102 O'Connells Point RD, WALLAGA LAKE, NSW 2546 Storing 018.2021.00002112.001 - 14C Elizabeth PDE, TURA BEACH, NSW 2548 Storing 018.2021.00002111.001 - 26 Woodlands LA, BALD HILLS, NSW 2549 Storing 010.2021.00000194.002 - 148 Shady LA, WALLAGOOT, NSW 2550 Storing 010.2021.00001160.001 - Cobargo ST, QUAAMA, NSW 2550 Storing 018.2021.00002110.001 - Evergreen LA, GREIGS FLAT, NSW 2549 Storing 010.2021.00001163.001 - 35 Beach ST, MERIMBULA, NSW 2548 Storing 010.2020.00000174.002 - 182 Mirador DR, MIRADOR, NSW 2548 Storing 010.2021.00001139.001 - 10 Lomandra AVE, KALARU, NSW 2550 Storing 010.2021.00001136.001 - 18 Tickalara CL, WALLAGOOT, NSW 2550 Storing 010.2021.00001159.001 - Barretts RD, BEMBOKA, NSW 2550 Storing 010.2021.00001161.001 - 13 Jacaranda PL, MIRADOR, NSW 2548 Storing 010.2021.00001138.001 - 9 Bournda CCT, TURA BEACH, NSW 2548 Storing 010.2021.00001153.001 - 19 Seaview AVE, MERIMBULA, NSW 2548 Storing 010.2021.00001154.001 - 25 Lake View DR, WALLAGA LAKE, NSW 2546 Storing 010.2021.00001137.001 - 239 Pacific WY, TURA BEACH, NSW 2548 Storing 010.2021.00001141.001 - 10 Jersey PL, COBARGO, NSW 2550 Storing 010.2021.00001155.001 - 32 The Crest, MIRADOR, NSW 2548 Storing 010.2003.00001051.011 - Candelo Bega RD, CANDELO, NSW 2550 Storing 010.2021.00001144.001 - 25 Bega ST, PAMBULA, NSW 2549 Storing 010.2021.00001134.001 - She Oak CL, BEGA, NSW 2550 Storing 010.2021.00001149.001 - 83 Bournda Park WY, WALLAGOOT, NSW 2550 Storing 010.2021.00001111.001 - 999 Back Creek RD, LOCHIEL, NSW 2549 Storing 010.2021.00001118.001 - 6 Thompson DR, TATHRA, NSW 2550 Storing 010.2021.00001143.001 - 153 Turingal Head RD, WALLAGOOT, NSW 2550 Storing 010.2021.00001133.001 - 26 Tillabudgery CT, BERMAGUI, NSW 2546 Storing 010.2021.00001109.001 - 24 Young ST, BERMAGUI, NSW 2546 Storing 010.2021.00001127.001 - 34 Parbery AVE, BERMAGUI, NSW 2546 Storing 010.2020.00000226.002 - 345 Wandella RD, COBARGO, NSW 2550 Storing 010.2021.00001142.001 - 27 Kulbardi CL, BOURNDA, NSW 2548 Storing 010.2021.00001110.001 - 7 Murrah ST, BERMAGUI, NSW 2546 Storing 010.2021.00001129.001 - 28 Buckajo RD, BEGA, NSW 2550 Storing 010.2021.00001130.001 - 151 Auckland ST, BEGA, NSW 2550 Storing 010.2021.00001120.001 - 39 Racecourse RD, BERMAGUI, NSW 2546 Collecting ATDIS feed data for ballina... W, [2021-09-24T10:06:52.125819 #18] WARN -- : No/invalid pagination, assuming no records/aborting Storing 010.2021.00000608.001 - 18 Dent CR, CUMBALUM, NSW 2478 Storing 010.2021.00000618.001 - Ellis RD, ALSTONVILLE, NSW 2477 Storing 010.2021.00000611.001 - 28 Silkwood RD, LENNOX HEAD, NSW 2478 Storing 010.2021.00000617.001 - 3 Montego PDE, ALSTONVILLE, NSW 2477 Storing 010.2021.00000606.001 - 17 Koellner RD, CUMBALUM, NSW 2478 Storing 010.2021.00000565.001 - 44 River DR, EAST WARDELL, NSW 2477 Storing 018.2021.00005159.001 - 12 Surf AVE, SKENNARS HEAD, NSW 2478 Collecting ATDIS feed data for bathurst... D, [2021-09-24T10:07:00.711391 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=3 D, [2021-09-24T10:07:05.374090 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=4 D, [2021-09-24T10:07:08.768057 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=5 D, [2021-09-24T10:07:11.062763 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=6 D, [2021-09-24T10:07:13.818288 #18] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=7 D, [2021-09-24T10:07:15.232996 #18] DEBUG -- : Fetching Storing 010.2021.00000481.001 - 4 Dunoon PL, ABERCROMBIE, NSW 2795 Storing 010.2021.00000488.001 - 233 Stewart ST, BATHURST, NSW 2795 Storing 010.2021.00000414.001 - 23 Bolton ST, KELSO, NSW 2795 Storing 010.2019.00000388.002 - 1611 Mid Western HWY, EVANS PLAINS, NSW 2795 Storing 010.2021.00000482.001 - 9 Darvall DR, KELSO, NSW 2795 Storing 010.2021.00000480.001 - 5229 Trunkey RD, TRUNKEY CREEK, NSW 2795 Storing 010.2021.00000477.001 - 1 Napier ST, WINDRADYNE, NSW 2795 Storing 010.2021.00000479.001 - 308 Laffing Waters LA, LAFFING WATERS, NSW 2795 Storing 018.2021.00000227.001 - 153 Hughes ST, KELSO, NSW 2795 Storing 010.2021.00000473.001 - 48 Lambert ST, BATHURST, NSW 2795 Storing 010.2021.00000404.001 - 47 Gilmour ST, KELSO, NSW 2795 Storing 018.2021.00000228.001 - 6 Copeman CT, ABERCROMBIE, NSW 2795 Storing 010.2021.00000466.001 - 119 Duramana RD, EGLINTON, NSW 2795 Storing 018.2021.00000218.001 - 32 Pine Ridge RD, ROCK FOREST, NSW 2795 Storing 010.2021.00000442.001 - 157 Russell ST, BATHURST, NSW 2795 Storing 010.2021.00000465.001 - 4 Locke ST, RAGLAN, NSW 2795 Storing 010.2021.00000403.001 - 11 Wattle CL, KELSO, NSW 2795 Storing 010.2021.00000474.001 - 16 Edgell ST, WEST BATHURST, NSW 2795 Storing 018.2021.00000224.001 - 8 Bolton ST, KELSO, NSW 2795 Storing 010.2021.00000435.001 - 5 Michigan RD, KELSO, NSW 2795 Storing 010.2021.00000415.001 - 67 George Thomas CL, THE LAGOON, NSW 2795 Storing 010.2021.00000452.001 - 189 Blue Ridge DR, KELSO, NSW 2795 Storing 010.2021.00000469.001 - 15 Arabella PL, ROBIN HILL, NSW 2795 Storing 010.2021.00000467.001 - 105 Sydney RD, KELSO, NSW 2795 Storing 010.2021.00000468.001 - 14 Watt DR, ROBIN HILL, NSW 2795 Storing 010.2021.00000459.001 - 9 Quigley CL, KELSO, NSW 2795 Storing 010.2021.00000455.001 - 32 Meagher ST, LLANARTH, NSW 2795 Storing 010.2021.00000409.001 - 3 Wylchris LA, MOUNT RANKIN, NSW 2795 Storing 010.2021.00000456.001 - Kendall AVE, BATHURST, NSW 2795 Storing 010.2021.00000103.002 - 107 Hen & Chicken LA, PERTHVILLE, NSW 2795 Storing 010.2020.00000487.002 - 32 Logan ST, EGLINTON, NSW 2795 Storing 010.2021.00000445.001 - 107 Mountain STGT, MOUNT PANORAMA, NSW 2795 Storing 018.2021.00000221.001 - 8 Lyon CL, KELSO, NSW 2795 Storing 010.2021.00000444.001 - 19 Rothery ST, EGLINTON, NSW 2795 Storing 010.2021.00000447.001 - 829 Redbank RD, TRIANGLE FLAT, NSW 2795 Storing 010.2021.00000431.001 - 63 George Thomas CL, THE LAGOON, NSW 2795 Storing 018.2021.00000219.001 - 199 White Rock RD, WHITE ROCK, NSW 2795 Storing 018.2021.00000202.001 - 77 Morrow PL, ROBIN HILL, NSW 2795 Storing 010.2021.00000438.001 - 455 Wambool RD, WAMBOOL, NSW 2795 Storing 018.2021.00000217.001 - 10 Driscoll CL, KELSO, NSW 2795 Storing 010.2021.00000454.001 - 25 Landseer ST, RAGLAN, NSW 2795 Storing 010.2021.00000437.001 - 8 Dovey DR, KELSO, NSW 2795 Storing 018.2021.00000210.001 - 30 Sunbright RD, KELSO, NSW 2795 Storing 018.2021.00000214.001 - 195 Burges RD, CALOOLA, NSW 2795 Storing 010.2021.00000448.001 - 11 Ignatius PL, KELSO, NSW 2795 Storing 010.2021.00000451.001 - 7 Maple CL, KELSO, NSW 2795 Storing 018.2021.00000215.001 - 14 Watt DR, ROBIN HILL, NSW 2795 Storing 018.2021.00000213.001 - 36 Darwin DR, LLANARTH, NSW 2795 Storing 010.2021.00000426.001 - 210 Howick ST, BATHURST, NSW 2795 Storing 018.2021.00000212.001 - 13 Connolly DR, KELSO, NSW 2795 Storing 018.2021.00000174.001 - 23 Bolton ST, KELSO, NSW 2795 Storing 018.2021.00000209.001 - 17 Darvall DR, KELSO, NSW 2795 Storing 010.2021.00000440.001 - 36 Stanley ST, BATHURST, NSW 2795 Storing 010.2021.00000433.001 - 8 Kirkcaldy ST, SOUTH BATHURST, NSW 2795 Storing 010.2021.00000106.002 - 12 Wolseley ST, SOUTH BATHURST, NSW 2795 Storing 010.2021.00000434.001 - 428 Conrod STGT, MOUNT PANORAMA, NSW 2795 Storing 010.2021.00000439.001 - 75 Thomas DR, EGLINTON, NSW 2795 Storing 010.2021.00000424.001 - Freemantle RD, EGLINTON, NSW 2795 Storing 018.2021.00000190.001 - 30 Meagher ST, LLANARTH, NSW 2795 Storing 018.2021.00000211.001 - 22 Darling ST, EGLINTON, NSW 2795 Storing 010.2021.00000436.001 - 715 Eusdale RD, MEADOW FLAT, NSW 2795 Collecting ATDIS feed data for dubbo... D, [2021-09-24T10:07:25.266813 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=3 D, [2021-09-24T10:07:30.368503 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=4 D, [2021-09-24T10:07:34.560986 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=5 D, [2021-09-24T10:07:38.985617 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=6 D, [2021-09-24T10:07:43.699947 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=7 D, [2021-09-24T10:07:48.636791 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=8 D, [2021-09-24T10:07:53.757880 #18] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=9 D, [2021-09-24T10:07:56.685928 #18] DEBUG -- : Fetching Storing 010.2021.00000545.001 - 3 Yarra PL, DUBBO, NSW 2830 Storing 010.2021.00000604.001 - 65 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000591.001 - 10 Commercial AVE, DUBBO, NSW 2830 Storing 010.2021.00000606.001 - 1 Montezuma TCE, DUBBO, NSW 2830 Storing 010.2021.00000543.001 - 99 Bultje ST, DUBBO, NSW 2830 Storing 010.2021.00000536.001 - 53 Peel PL, DUBBO, NSW 2830 Storing 010.2020.00000612.002 - 1 Sea CRES, DUBBO, NSW 2830 Storing 010.2021.00000587.001 - 15 O'Connor PL, DUBBO, NSW 2830 Storing 016.2021.00000124.001 - 10 Paterson CRCT, DUBBO, NSW 2830 Storing 010.2021.00000583.001 - 1779 Uungula RD, WUULUMAN, NSW 2820 Storing 010.2021.00000508.001 - 99 Champagne DR, DUBBO, NSW 2830 Storing 010.2021.00000588.001 - 28 Old Homestead DR, DUBBO, NSW 2830 Storing 010.2021.00000602.001 - Bligh ST, DUBBO, NSW 2830 Storing 010.2021.00000603.001 - 26 Southlakes PDE, DUBBO, NSW 2830 Storing 016.2021.00000123.001 - 14 Pine Knoll DR, DUBBO, NSW 2830 Storing 016.2021.00000119.001 - 35 Nicholii ST, MONTEFIORES, NSW 2820 Storing 010.2021.00000601.001 - 3R Numbardie DR, DUBBO, NSW 2830 Storing 010.2021.00000586.001 - 46 Oxley AVE, MOUNT ARTHUR, NSW 2820 Storing 010.2021.00000598.001 - 41 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000553.001 - 150 Whiteley ST, WELLINGTON, NSW 2820 Storing 010.2020.00000606.002 - 2 Waterway CRES, DUBBO, NSW 2830 Storing 010.2021.00000597.001 - 67 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000592.001 - 12 King ST, MONTEFIORES, NSW 2820 Storing 010.2021.00000529.001 - 6R Numbardie DR, DUBBO, NSW 2830 Storing 010.2021.00000596.001 - 13 Briny CT, DUBBO, NSW 2830 Storing 010.2021.00000558.001 - 44 Azure AVE, DUBBO, NSW 2830 Storing 010.2021.00000582.001 - 87 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000537.001 - 7 Allen RD, DUBBO, NSW 2830 Storing 010.2021.00000471.001 - 18 Hawkesbury PL, DUBBO, NSW 2830 Storing 010.2021.00000595.001 - 53 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000593.001 - 47 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000579.001 - 101-103 Talbragar ST, DUBBO, NSW 2830 Storing 010.2021.00000578.001 - 63 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000581.001 - 2 Delta RD, DUBBO, NSW 2830 Storing 010.2021.00000555.001 - 8 Tony McGrane PL, DUBBO, NSW 2830 Storing 010.2021.00000589.001 - 14 Drysdale CL, DUBBO, NSW 2830 Storing 010.2021.00000580.001 - 43 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000576.001 - Barbigal RD, WONGARBON, NSW 2831 Storing 010.2021.00000577.001 - 61 Argyle AVE, DUBBO, NSW 2830 Storing 010.2021.00000551.001 - 18 Ferrier DR, DUBBO, NSW 2830 Storing 010.2021.00000574.001 - 11L Rosedale RD, DUBBO, NSW 2830 Storing 010.2021.00000567.001 - 7 Durum CRCT, DUBBO, NSW 2830 Storing 010.2021.00000573.001 - 136 Bunglegumbie RD, DUBBO, NSW 2830 Storing 010.2021.00000550.001 - 4R Boundary Creek RD, DUBBO, NSW 2830 Storing 010.2021.00000563.001 - Pinedale RD, DUBBO, NSW 2830 Storing 010.2021.00000168.002 - 141 Sheraton RD, DUBBO, NSW 2830 Storing 010.2021.00000572.001 - 17L Kurrajong DR, DUBBO, NSW 2830 Storing 010.2021.00000554.001 - 11 Champagne DR, DUBBO, NSW 2830 Storing 010.2021.00000564.001 - 13 Rhyana CT, DUBBO, NSW 2830 Storing 010.2021.00000410.002 - 10L Pinedale RD, DUBBO, NSW 2830 Storing 010.2021.00000562.001 - 29 Durum CRCT, DUBBO, NSW 2830 Storing 010.2021.00000560.001 - 63 Fitzroy ST, DUBBO, NSW 2830 Storing 010.2020.00000610.002 - 11 Delta RD, DUBBO, NSW 2830 Storing 010.2021.00000565.001 - 188 Gipps ST, DUBBO, NSW 2830 Storing 010.2021.00000220.002 - 137 Boundary RD, DUBBO, NSW 2830 Storing 010.2021.00000556.001 - 7 Durum CRCT, DUBBO, NSW 2830 Storing 010.2020.00000648.002 - 10 Delta RD, DUBBO, NSW 2830 Storing 010.2020.00000203.002 - 34L Yarrandale RD, DUBBO, NSW 2830 Storing 010.2021.00000547.001 - 9 Briny CT, DUBBO, NSW 2830 Storing 010.2021.00000557.001 - 10 Chondelle CT, DUBBO, NSW 2830 Storing 010.2021.00000528.001 - 26 Quinn ST, DUBBO, NSW 2830 Storing 010.2021.00000530.001 - 5403 Golden HWY, BALLIMORE, NSW 2830 Storing 010.2021.00000548.001 - 54 Lincoln PKY, DUBBO, NSW 2830 Storing 010.2021.00000552.001 - 47 Macleay ST, DUBBO, NSW 2830 Storing 016.2021.00000117.001 - 11 Scope CRCT, DUBBO, NSW 2830 Storing 016.2021.00000115.001 - 4660 Goolma RD, SPICERS CREEK, NSW 2820 Storing 010.2021.00000494.001 - 314 Myall ST, DUBBO, NSW 2830 Storing 016.2021.00000118.001 - 8 Gulf CT, DUBBO, NSW 2830 Storing 016.2021.00000116.001 - 151-153 Macquarie ST, DUBBO, NSW 2830 Storing 010.2021.00000540.001 - 6 Dulhunty AVE, DUBBO, NSW 2830 Storing 010.2021.00000522.001 - 15L Angle Park RD, DUBBO, NSW 2830 Storing 010.2012.00000019.005 - Hennessy DR, DUBBO, NSW 2830 Storing 010.2021.00000485.001 - 59 Bournewood RD, NORTH YEOVAL, NSW 2868 Storing 010.2021.00000542.001 - 199 Wingewarra ST, DUBBO, NSW 2830 Storing 010.2021.00000488.001 - 3 Capstan DR, DUBBO, NSW 2830 Storing 010.2021.00000546.001 - 15R Boothenba RD, DUBBO, NSW 2830 Storing 010.2021.00000532.001 - 1 Gulf CT, DUBBO, NSW 2830 Storing 010.2021.00000533.001 - 14 Tamworth ST, DUBBO, NSW 2830 Storing 010.2021.00000544.001 - 16 Volta AVE, DUBBO, NSW 2830 Storing 010.2021.00000509.001 - 4 Briny CT, DUBBO, NSW 2830 Storing 010.2021.00000526.001 - 10 Jannali RD, DUBBO, NSW 2830 Collecting ATDIS feed data for kiama... D, [2021-09-24T10:08:14.446425 #18] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=3 D, [2021-09-24T10:08:22.989608 #18] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=4 D, [2021-09-24T10:08:28.261646 #18] DEBUG -- : Fetching Storing 010.2021.00000227.001 - 13 Sharwood PL, GERRINGONG, NSW 2534 Storing 010.2021.00000225.001 - 74 Fern ST, GERRINGONG, NSW 2534 Storing 010.2021.00000223.001 - 40 Anembo CR, KIAMA HEIGHTS, NSW 2533 Storing 010.2021.00000213.001 - 19 Farmer ST, KIAMA, NSW 2533 Storing 010.2021.00000217.001 - 23 Elanora RD, KIAMA HEIGHTS, NSW 2533 Storing 010.2021.00000228.001 - 4 Farmer ST, KIAMA, NSW 2533 Storing 010.2021.00000221.001 - 33 Beattie ST, JAMBEROO, NSW 2533 Storing 010.2011.00000086.026 - 27 Arnold CR, KIAMA, NSW 2533 Storing 010.2018.00000197.002 - 6 Pacific ST, KIAMA, NSW 2533 Storing 010.2021.00000200.001 - 69 Marsden ST, KIAMA, NSW 2533 Storing 010.2021.00000214.001 - 32 Jerrara RD, JERRARA, NSW 2533 Storing 010.2021.00000218.001 - 10 Tomlins RD, BROUGHTON VILLAGE, NSW 2534 Storing 010.2021.00000195.001 - 19 Lindsay AV, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000206.001 - 66 Minnamurra ST, KIAMA, NSW 2533 Storing 010.2021.00000185.001 - 106 Fern ST, GERRINGONG, NSW 2534 Storing 010.2021.00000212.001 - 1 Downes PL, JAMBEROO, NSW 2533 Storing 010.2021.00000222.001 - 5 Northpoint PL, BOMBO, NSW 2533 Storing 010.2021.00000205.001 - 58 Terralong ST, KIAMA, NSW 2533 Storing 010.2021.00000208.001 - 7 Marks ST, KIAMA, NSW 2533 Storing 010.2021.00000182.001 - 73 Jupiter ST, GERRINGONG, NSW 2534 Storing 010.2021.00000203.001 - 40 Tombonda DR, KIAMA, NSW 2533 Storing 010.2021.00000210.001 - 54A Boanyo AV, KIAMA, NSW 2533 Storing 010.2021.00000201.001 - 69 Hillview CRCT, KIAMA, NSW 2533 Storing 010.2021.00000179.001 - 37 Osborne ST, GERRINGONG, NSW 2534 Storing 010.2021.00000204.001 - 2 Pheasant Point DR, KIAMA, NSW 2533 Storing 010.2021.00000196.001 - 37 Pheasant Point DR, KIAMA, NSW 2533 Storing 010.2021.00000178.001 - 114 Terralong ST, KIAMA, NSW 2533 Storing 010.2021.00000202.001 - 144 Curramore RD, CURRAMORE, NSW 2533 Storing 010.1999.00000436.003 - 104 Fern ST, GERRINGONG, NSW 2534 Storing 010.2021.00000188.001 - 187 Fern ST, GERRINGONG, NSW 2534 Storing 010.2021.00000180.001 - 85 Merrick CCT, KIAMA, NSW 2533 Storing 010.2021.00000198.001 - 129 Charles AV, MINNAMURRA, NSW 2533 Storing 010.2020.00000239.002 - 77 Merrick CCT, KIAMA, NSW 2533 Collecting ATDIS feed data for upper_hunter... D, [2021-09-24T10:08:32.115078 #18] DEBUG -- : Fetching http://onlineservices.upperhunter.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=3 W, [2021-09-24T10:08:32.805126 #18] WARN -- : No/invalid pagination, assuming no records/aborting Collecting ATDIS feed data for armidale... W, [2021-09-24T10:08:37.101986 #18] WARN -- : No/invalid pagination, assuming no records/aborting Collecting ATDIS feed data for muswellbrook... D, [2021-09-24T10:08:40.943963 #18] DEBUG -- : Fetching https://datracker.muswellbrook.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2021-09-24&lodgement_date_start=2021-08-25&page=3 D, [2021-09-24T10:08:41.999954 #18] DEBUG -- : Fetching Storing 005.2021.00000117.001 - 121 Ironbark RD, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000116.001 - 29 Aberdeen ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000115.001 - 84 Ironbark RD, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000112.001 - 5 Palace ST, DENMAN, NSW 2328 Storing 005.2021.00000111.001 - 59 Sowerby ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000110.001 - 3 Osborn AVE, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000109.001 - 45 Scott ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000114.001 - 22 Common RD, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000113.001 - 9 Sepoy CR, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000107.001 - 99 Ironbark RD, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000106.001 - 2335 Martindale PH, COUNTY HUNTER, NSW 2333 Storing 005.2021.00000105.001 - 14 Chivers CCT, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000104.001 - 84-86 Maitland ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000103.001 - 7 Foley ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000102.001 - 37 Stockyard PDE, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000101.001 - 153 Queen ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000100.001 - 20 Lou Fisher PL, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000053.002 - 14 Pendula WAY, DENMAN, NSW 2328 Storing 005.2021.00000098.001 - Maitland ST, MUSWELLBROOK, NSW 2333 Storing 005.2021.00000099.001 - Rutherford RD, MUSWELLBROOK, NSW 2333 Collecting ATDIS feed data for walgett...

Data

Downloaded 10273 times by openaustralia DevAppProject dcalde eduardomontero muzzie86 mattykn vietbaracon

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (4.84 MB) Use the API

rows 10 / 10545

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to authority_label
010.2019.00000164.001
119 Lochview Farm RD, LOCHIEL, NSW 2549
Installation of above ground swimming pool
2019-06-06
2019-05-07
bega_valley
010.2012.00000402.002
5 Bells PL, DOCTOR GEORGE MOUNTAIN, NSW 2550
Modification Erection of a dwelling & detached garage/studio & carport
2019-06-06
2019-05-07
2019-05-23
2019-06-06
bega_valley
018.2019.00005032.001
10 Willie Wagtail PL, ALSTONVILLE, NSW 2477
PCA CDC - New Dwelling - Single Residence
2019-06-06
2019-05-07
ballina
010.2019.00000283.001
2/188-202 Southern Cross DR, BALLINA, NSW 2478
Change of Use and Alterations to Commercial Building Change of use from a high technology industry to general industry involving the assembly, manufacture and fitting of prosthetic limbs and alterations to the building
2019-06-06
2019-05-07
ballina
010.2019.00000284.001
84 Meridian DR, COOLGARDIE, NSW 2478
New Dwelling - Single Residence Construction of a two storey dwelling house and associated studio as depicted in plans by Dean Poile Designs for 64 Meridian Drive Coolgardie dated 2 May 2019
2019-06-06
2019-05-07
ballina
010.2019.00000139.001
141 Keppel ST, BATHURST, NSW 2795
Stage 1 - Five lot torens title subdivision (four residential lots and one development lot) Stage 2 - Seven lot strata title subdivision
2019-06-06
2019-05-07
bathurst
018.2019.00000051.001
10 Newlands CR, KELSO, NSW 2795
Construction of a Single storey dwelling with attached garage
2019-06-06
2019-05-07
bathurst
010.2019.00000138.001
82 Graham DR, KELSO, NSW 2795
Dual Occupancy and two lot residential subdivision
2019-06-06
2019-05-07
bathurst
010.2019.00000078.001
21 Boyd ST, MINNAMURRA, NSW 2533
dwelling & air conditioning unit
2019-06-06
2019-05-07
kiama
010.2019.00000080.001
133 Headland DR, GERROA, NSW 2534
detached storage shed
2019-06-06
2019-05-07
2019-05-23
2019-06-06
kiama

Statistics

Average successful run time: 5 minutes

Total run time: 5 days

Total cpu time used: 35 minutes

Total disk space used: 6.89 MB

History

  • Auto ran revision fdd72316 and completed successfully .
    261 records added, 245 records removed in the database
  • Auto ran revision fdd72316 and completed successfully .
    250 records added, 240 records removed in the database
  • Auto ran revision fdd72316 and completed successfully .
    268 records added, 243 records removed in the database
  • Auto ran revision fdd72316 and completed successfully .
    225 records added, 204 records removed in the database
  • Auto ran revision fdd72316 and completed successfully .
    209 records added, 205 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

multiple_atdis / scraper.rb