planningalerts-scrapers / multiple_atdis

Scrapes multiple authorities that have ATDIS feed


This is a scraper that runs on Morph. To get started see the documentation

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

Contributors mlandauer dependabot[bot]

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching rake 12.3.3  Fetching concurrent-ruby 1.1.6  Fetching minitest 5.14.1  Installing minitest 5.14.1  Installing rake 12.3.3  Installing concurrent-ruby 1.1.6  Fetching thread_safe 0.3.6  Fetching zeitwerk 2.4.0  Installing zeitwerk 2.4.0  Fetching public_suffix 4.0.5  Installing public_suffix 4.0.5  Fetching ast 2.4.1  Installing ast 2.4.1  Installing thread_safe 0.3.6  Fetching multi_json 1.15.0  Installing multi_json 1.15.0  Fetching http-accept 1.7.0  Installing http-accept 1.7.0  Fetching unf_ext 0.0.7.7  Installing unf_ext 0.0.7.7 with native extensions  Fetching mime-types-data 3.2020.0512  Installing mime-types-data 3.2020.0512  Fetching netrc 0.11.0  Installing netrc 0.11.0  Fetching rgeo 2.1.1  Using bundler 1.15.2  Fetching safe_yaml 1.0.5  Installing rgeo 2.1.1 with native extensions  Installing safe_yaml 1.0.5  Fetching diff-lcs 1.4.4  Installing diff-lcs 1.4.4  Fetching hashdiff 1.0.1  Installing hashdiff 1.0.1  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching parallel 1.19.2  Installing parallel 1.19.2  Fetching rainbow 3.0.0  Installing rainbow 3.0.0  Fetching regexp_parser 1.7.1  Installing regexp_parser 1.7.1  Fetching rexml 3.2.5  Installing rexml 3.2.5  Fetching rspec-support 3.9.3  Installing rspec-support 3.9.3  Fetching ruby-progressbar 1.10.1  Installing ruby-progressbar 1.10.1  Fetching unicode-display_width 1.7.0  Installing unicode-display_width 1.7.0  Fetching sqlite3 1.4.2  Installing sqlite3 1.4.2 with native extensions  Fetching vcr 6.0.0  Installing vcr 6.0.0  Fetching addressable 2.7.0  Installing addressable 2.7.0  Fetching parser 2.7.1.4  Installing parser 2.7.1.4  Fetching i18n 1.8.5  Installing i18n 1.8.5  Fetching tzinfo 1.2.7  Installing tzinfo 1.2.7  Fetching mime-types 3.3.1  Installing mime-types 3.3.1  Fetching crack 0.4.3  Installing crack 0.4.3  Fetching rspec-core 3.9.2  Installing rspec-core 3.9.2  Fetching rspec-expectations 3.9.2  Installing rspec-expectations 3.9.2  Fetching rspec-mocks 3.9.1  Fetching unf 0.1.4  Installing rspec-mocks 3.9.1  Installing unf 0.1.4  Fetching rgeo-geojson 2.1.1  Installing rgeo-geojson 2.1.1  Fetching rubocop-ast 0.2.0  Fetching activesupport 6.0.3.2  Installing rubocop-ast 0.2.0  Installing activesupport 6.0.3.2  Fetching webmock 3.8.3  Installing webmock 3.8.3  Fetching domain_name 0.5.20190701  Installing domain_name 0.5.20190701  Fetching rspec 3.9.0  Fetching rubocop 0.88.0  Installing rspec 3.9.0  Fetching activemodel 6.0.3.2  Installing activemodel 6.0.3.2  Installing rubocop 0.88.0  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching rest-client 2.1.0  Installing rest-client 2.1.0  Fetching atdis 0.5.0  Installing atdis 0.5.0  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching scraperwiki 3.0.2  Installing scraperwiki 3.0.2  Fetching scraperwiki-morph 0.1.1  Installing scraperwiki-morph 0.1.1  Bundle complete! 7 Gemfile dependencies, 51 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  If you are upgrading your Rails application from an older version of Rails:  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  If you are starting a NEW Rails application, you can ignore this notice.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Bundle completed (16.42s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting ATDIS feed data for bega_valley... bega_valley: ERROR: 403 Forbidden Collecting ATDIS feed data for ballina... /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `block in <main>' scraper.rb:39:in `each' scraper.rb:39:in `<main>' ballina: ERROR: 403 Forbidden Collecting ATDIS feed data for bathurst... /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:910:in `start' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `block in <main>' scraper.rb:39:in `each' scraper.rb:39:in `<main>' Collecting ATDIS feed data for dubbo... bathurst: ERROR: Timed out connecting to server /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:749:in `rescue in transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:651:in `transmit' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute' /app/vendor/bundle/ruby/2.5.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:79:in `read_url_raw' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/model.rb:83:in `read_url' /app/vendor/bundle/ruby/2.5.0/gems/atdis-0.5.0/lib/atdis/feed.rb:72:in `applications' /app/lib/atdisplanningalertsfeed.rb:45:in `fetch_all_pages' /app/lib/atdisplanningalertsfeed.rb:19:in `fetch' scraper.rb:45:in `block in <main>' scraper.rb:39:in `each' scraper.rb:39:in `<main>' D, [2022-01-22T11:48:54.315556 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-01-22&lodgement_date_start=2021-12-23&page=3 D, [2022-01-22T11:48:59.219757 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-01-22&lodgement_date_start=2021-12-23&page=4 D, [2022-01-22T11:49:01.632595 #17] DEBUG -- : Fetching https://planning.dubbo.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-01-22&lodgement_date_start=2021-12-23&page=5 W, [2022-01-22T11:49:02.396435 #17] WARN -- : No/invalid pagination, assuming no records/aborting Storing 010.2016.00000427.002 - 65L Mogriguy RD, BROCKLEHURST, NSW 2830 Storing 010.2021.00000726.001 - 28 Lakeside CRCT, DUBBO, NSW 2830 Storing 010.2021.00000221.002 - 91 Boundary RD, DUBBO, NSW 2830 Storing 010.2021.00000848.001 - 66 Fitzroy ST, DUBBO, NSW 2830 Storing 010.2022.00000002.001 - 25A Wheelers LA, DUBBO, NSW 2830 Storing 010.2021.00000840.001 - 16 Yuille CT, DUBBO, NSW 2830 Storing 010.2021.00000749.001 - 115 Champagne DR, DUBBO, NSW 2830 Storing 010.2021.00000846.001 - 2 Nolan ESP, DUBBO, NSW 2830 Storing 010.2022.00000005.001 - 37 Boundary RD, DUBBO, NSW 2830 Storing 016.2022.00000001.001 - 49 Ronald ST, DUBBO, NSW 2830 Storing 016.2022.00000004.001 - 45 Ronald ST, DUBBO, NSW 2830 Storing 010.2021.00000845.001 - 7L Oakdene RD, DUBBO, NSW 2830 Storing 010.2021.00000836.001 - Purvis LA, DUBBO, NSW 2830 Storing 010.2021.00000585.001 - 178 Tarrawingee RD, MUMBIL, NSW 2820 Storing 016.2022.00000005.001 - 2/15R Harefield RD, DUBBO, NSW 2830 Storing 010.2021.00000716.001 - 6 Short ST, WELLINGTON, NSW 2820 Storing 010.2022.00000004.001 - 11 Spence ST, DUBBO, NSW 2830 Storing 010.2022.00000003.001 - 500 Wheelers LA, DUBBO, NSW 2830 Storing 010.2022.00000006.001 - 12L Toorale RD, DUBBO, NSW 2830 Storing 016.2022.00000002.001 - 40 Southlakes PDE, DUBBO, NSW 2830 Storing 010.2021.00000847.001 - 146 Bunglegumbie RD, DUBBO, NSW 2830 Storing 010.2022.00000001.001 - 59 Sterling ST, DUBBO, NSW 2830 Storing 010.2021.00000772.001 - 3 Tulloch PL, DUBBO, NSW 2830 Storing 010.2021.00000777.001 - 15 Ferrier DR, DUBBO, NSW 2830 Storing 010.2021.00000843.001 - 62 Azure AVE, DUBBO, NSW 2830 Storing 010.2021.00000837.001 - 63 Azure AVE, DUBBO, NSW 2830 Storing 010.2021.00000835.001 - 8 Sunningdale CL, DUBBO, NSW 2830 Storing 010.2021.00000769.001 - 114 Wingewarra ST, DUBBO, NSW 2830 Storing 016.2021.00000160.001 - 25A Wheelers LA, DUBBO, NSW 2830 Storing 010.2021.00000767.001 - 8 Denison ST, DUBBO, NSW 2830 Storing 010.2021.00000844.001 - 27 Bishop ST, DUBBO, NSW 2830 Storing 016.2021.00000163.001 - 49 Argyle AVE, DUBBO, NSW 2830 Collecting ATDIS feed data for kiama... D, [2022-01-22T11:49:23.847753 #17] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2022-01-22&lodgement_date_start=2021-12-23&page=3 D, [2022-01-22T11:49:27.775518 #17] DEBUG -- : Fetching Storing 010.2021.00000336.001 - 43 Thomson ST, KIAMA, NSW 2533 Storing 010.2021.00000037.002 - 31 Noorinan ST, KIAMA, NSW 2533 Storing 010.2021.00000330.001 - 81 Oxley AV, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000349.001 - 52 Bong Bong ST, KIAMA, NSW 2533 Storing 010.2021.00000339.001 - 12 Cliff DR, KIAMA DOWNS, NSW 2533 Storing 010.2019.00000235.003 - 5 The Circuit, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000345.001 - 69 Foxground RD, FOXGROUND, NSW 2534 Storing 010.2021.00000357.001 - 231 Curramore RD, CURRAMORE, NSW 2533 Storing 010.2021.00000337.001 - 15 Surfleet PL, KIAMA, NSW 2533 Storing 010.2021.00000333.001 - 49 Talinga AV, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000334.001 - 537 Fountaindale RD, SADDLEBACK MOUNTAIN, NSW 2533 Storing 010.2021.00000341.001 - 75 Meehan DR, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000335.001 - 2 Elanora RD, KIAMA HEIGHTS, NSW 2533 Storing 010.2021.00000109.002 - 15 Taylor ST, KIAMA, NSW 2533 Storing 010.2021.00000343.001 - 42 Charles AV, MINNAMURRA, NSW 2533 Storing 010.2021.00000338.001 - 4 Brown ST, KIAMA, NSW 2533 Storing 010.2017.00000345.002 - 141 Charles AV, MINNAMURRA, NSW 2533 Storing 010.2019.00000187.002 - 7 Federal ST, MINNAMURRA, NSW 2533 Storing 010.2021.00000331.001 - 42 North Kiama DR, KIAMA DOWNS, NSW 2533 Storing 010.2021.00000296.001 - 20 Elanora RD, KIAMA HEIGHTS, NSW 2533 Storing 010.2021.00000329.001 - 90 North Curramore RD, CURRAMORE, NSW 2533 Collecting ATDIS feed data for upper_hunter... W, [2022-01-22T11:49:28.521441 #17] WARN -- : No/invalid pagination, assuming no records/aborting Collecting ATDIS feed data for armidale... Storing 297801 - 68 Judith Street, ARMIDALE, NSW 2350 Collecting ATDIS feed data for muswellbrook... D, [2022-01-22T11:49:36.053612 #17] DEBUG -- : Fetching https://datracker.muswellbrook.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2022-01-22&lodgement_date_start=2021-12-23&page=3 W, [2022-01-22T11:49:37.009064 #17] WARN -- : No/invalid pagination, assuming no records/aborting Storing 005.2022.00000007.001 - 10 Ted Clay ST, MUSWELLBROOK, NSW 2333 Storing 005.2022.00000006.001 - 5A St Heliers ST, MUSWELLBROOK, NSW 2333 Storing 017.2022.00000004.001 - 11 Burkill CL, DENMAN, NSW 2328 Storing 017.2022.00000005.001 - 11 Burkill CL, DENMAN, NSW 2328 Storing 005.2021.00000033.002 - Sandy Creek RD, MCCULLYS GAP, NSW 2333 Storing 005.2022.00000004.001 - 36-38 Finnegan CR, MUSWELLBROOK, NSW 2333 Storing 005.2022.00000005.001 - Victoria ST, MUSWELLBROOK, NSW 2333 Storing 005.2022.00000001.001 - 1490 Denman Rd, DENMAN, NSW 2328 Storing 005.2022.00000002.001 - 11 Sepoy CR, MUSWELLBROOK, NSW 2333 Collecting ATDIS feed data for walgett... scraper.rb:57:in `<main>': There were earlier errors. See output for details (RuntimeError)

Data

Downloaded 11427 times by openaustralia DevAppProject dcalde eduardomontero mattykn muzzie86 vietbaracon WeaponizedBytes

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (5.33 MB) Use the API

rows 10 / 11641

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to authority_label
010.2019.00000164.001
119 Lochview Farm RD, LOCHIEL, NSW 2549
Installation of above ground swimming pool
2019-06-06
2019-05-07
bega_valley
010.2012.00000402.002
5 Bells PL, DOCTOR GEORGE MOUNTAIN, NSW 2550
Modification Erection of a dwelling & detached garage/studio & carport
2019-06-06
2019-05-07
2019-05-23
2019-06-06
bega_valley
018.2019.00005032.001
10 Willie Wagtail PL, ALSTONVILLE, NSW 2477
PCA CDC - New Dwelling - Single Residence
2019-06-06
2019-05-07
ballina
010.2019.00000283.001
2/188-202 Southern Cross DR, BALLINA, NSW 2478
Change of Use and Alterations to Commercial Building Change of use from a high technology industry to general industry involving the assembly, manufacture and fitting of prosthetic limbs and alterations to the building
2019-06-06
2019-05-07
ballina
010.2019.00000284.001
84 Meridian DR, COOLGARDIE, NSW 2478
New Dwelling - Single Residence Construction of a two storey dwelling house and associated studio as depicted in plans by Dean Poile Designs for 64 Meridian Drive Coolgardie dated 2 May 2019
2019-06-06
2019-05-07
ballina
010.2019.00000139.001
141 Keppel ST, BATHURST, NSW 2795
Stage 1 - Five lot torens title subdivision (four residential lots and one development lot) Stage 2 - Seven lot strata title subdivision
2019-06-06
2019-05-07
bathurst
018.2019.00000051.001
10 Newlands CR, KELSO, NSW 2795
Construction of a Single storey dwelling with attached garage
2019-06-06
2019-05-07
bathurst
010.2019.00000138.001
82 Graham DR, KELSO, NSW 2795
Dual Occupancy and two lot residential subdivision
2019-06-06
2019-05-07
bathurst
010.2019.00000078.001
21 Boyd ST, MINNAMURRA, NSW 2533
dwelling & air conditioning unit
2019-06-06
2019-05-07
kiama
010.2019.00000080.001
133 Headland DR, GERROA, NSW 2534
detached storage shed
2019-06-06
2019-05-07
2019-05-23
2019-06-06
kiama

Statistics

Average successful run time: 5 minutes

Total run time: 6 days

Total cpu time used: 40 minutes

Total disk space used: 7.38 MB

History

  • Auto ran revision fdd72316 and failed .
    63 records added, 63 records removed in the database
  • Auto ran revision fdd72316 and failed .
    69 records added, 66 records removed in the database
  • Auto ran revision fdd72316 and failed .
    71 records added, 66 records removed in the database
  • Auto ran revision fdd72316 and failed .
    75 records added, 72 records removed in the database
  • Auto ran revision fdd72316 and failed .
    104 records added, 99 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

multiple_atdis / scraper.rb