planningalerts-scrapers / multiple_atdis

Scrapes multiple authorities that have ATDIS feed


This is a scraper that runs on Morph. To get started see the documentation

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

Contributors mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching rake 12.3.3  Fetching concurrent-ruby 1.1.6  Fetching minitest 5.14.1  Installing rake 12.3.3  Installing minitest 5.14.1  Installing concurrent-ruby 1.1.6  Fetching thread_safe 0.3.6  Installing thread_safe 0.3.6  Fetching zeitwerk 2.4.0  Installing zeitwerk 2.4.0  Fetching public_suffix 4.0.5  Fetching ast 2.4.1  Installing ast 2.4.1  Fetching multi_json 1.15.0  Installing multi_json 1.15.0  Fetching http-accept 1.7.0  Installing http-accept 1.7.0  Installing public_suffix 4.0.5  Fetching unf_ext 0.0.7.7  Fetching mime-types-data 3.2020.0512  Installing mime-types-data 3.2020.0512  Installing unf_ext 0.0.7.7 with native extensions  Fetching netrc 0.11.0  Installing netrc 0.11.0  Fetching rgeo 2.1.1  Installing rgeo 2.1.1 with native extensions  Using bundler 1.15.2  Fetching safe_yaml 1.0.5  Installing safe_yaml 1.0.5  Fetching diff-lcs 1.4.4  Installing diff-lcs 1.4.4  Fetching hashdiff 1.0.1  Installing hashdiff 1.0.1  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching parallel 1.19.2  Installing parallel 1.19.2  Fetching rainbow 3.0.0  Installing rainbow 3.0.0  Fetching regexp_parser 1.7.1  Installing regexp_parser 1.7.1  Fetching rexml 3.2.5  Installing rexml 3.2.5  Fetching rspec-support 3.9.3  Installing rspec-support 3.9.3  Fetching ruby-progressbar 1.10.1  Installing ruby-progressbar 1.10.1  Fetching unicode-display_width 1.7.0  Installing unicode-display_width 1.7.0  Fetching sqlite3 1.4.2  Installing sqlite3 1.4.2 with native extensions  Fetching vcr 6.0.0  Installing vcr 6.0.0  Fetching tzinfo 1.2.7  Installing tzinfo 1.2.7  Fetching parser 2.7.1.4  Installing parser 2.7.1.4  Fetching i18n 1.8.5  Installing i18n 1.8.5  Fetching addressable 2.7.0  Installing addressable 2.7.0  Fetching mime-types 3.3.1  Installing mime-types 3.3.1  Fetching crack 0.4.3  Fetching rspec-core 3.9.2  Installing crack 0.4.3  Installing rspec-core 3.9.2  Fetching rspec-expectations 3.9.2  Installing rspec-expectations 3.9.2  Fetching rspec-mocks 3.9.1  Installing rspec-mocks 3.9.1  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching rgeo-geojson 2.1.1  Fetching activesupport 6.0.3.2  Installing rgeo-geojson 2.1.1  Fetching rubocop-ast 0.2.0  Installing rubocop-ast 0.2.0  Installing activesupport 6.0.3.2  Fetching webmock 3.8.3  Installing webmock 3.8.3  Fetching domain_name 0.5.20190701  Installing domain_name 0.5.20190701  Fetching rspec 3.9.0  Fetching rubocop 0.88.0  Installing rspec 3.9.0  Fetching activemodel 6.0.3.2  Installing rubocop 0.88.0  Installing activemodel 6.0.3.2  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching rest-client 2.1.0  Installing rest-client 2.1.0  Fetching atdis 0.5.0  Installing atdis 0.5.0  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching scraperwiki 3.0.2  Installing scraperwiki 3.0.2  Fetching scraperwiki-morph 0.1.1  Installing scraperwiki-morph 0.1.1  Bundle complete! 7 Gemfile dependencies, 51 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  If you are upgrading your Rails application from an older version of Rails:  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  If you are starting a NEW Rails application, you can ignore this notice.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Bundle completed (14.63s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting ATDIS feed data for armidale... Storing 310129 - 34 White Avenue, ARMIDALE, NSW 2350 Storing 310128 - 32 White Avenue, ARMIDALE, NSW 2350 Storing 309980 - 107B Kirkwood Street, ARMIDALE, NSW 2350 Collecting ATDIS feed data for bathurst... D, [2022-10-04T10:44:56.077916 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-10-04&lodgement_date_start=2022-09-04&page=3 D, [2022-10-04T10:44:58.790456 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-10-04&lodgement_date_start=2022-09-04&page=4 D, [2022-10-04T10:45:02.125100 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-10-04&lodgement_date_start=2022-09-04&page=5 D, [2022-10-04T10:45:05.402735 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-10-04&lodgement_date_start=2022-09-04&page=6 D, [2022-10-04T10:45:08.518039 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-10-04&lodgement_date_start=2022-09-04&page=7 D, [2022-10-04T10:45:10.400204 #17] DEBUG -- : Fetching http://masterview.bathurst.nsw.gov.au/atdis/1.0//applications.json?lodgement_date_end=2022-10-04&lodgement_date_start=2022-09-04&page=8 W, [2022-10-04T10:45:10.831198 #17] WARN -- : No/invalid pagination, assuming no records/aborting Storing 010.2022.00000399.001 - 1120 Rockley RD, FOSTERS VALLEY, NSW 2795 Storing 010.2022.00000408.001 - 55 Basalt WAY, KELSO, NSW 2795 Storing 010.2022.00000362.001 - 54 Wylchris LA, MOUNT RANKIN, NSW 2795 Storing 018.2022.00000165.001 - 96 Bentinck ST, BATHURST, NSW 2795 Storing 010.2022.00000435.001 - 492 Hill End RD, CRUDINE, NSW 2795 Storing 018.2022.00000156.001 - 49 Newlands CR, KELSO, NSW 2795 Storing 018.2022.00000159.001 - 7 Samuel WAY, THE LAGOON, NSW 2795 Storing 010.2022.00000426.001 - 38 Brennan DR, KELSO, NSW 2795 Storing 010.2022.00000409.001 - 9 Jones LA, WATTLE FLAT, NSW 2795 Storing 018.2022.00000155.001 - 23 Munro ST, WINDRADYNE, NSW 2795 Storing 010.2022.00000428.001 - 8 Lyon CL, KELSO, NSW 2795 Storing 010.2022.00000424.001 - 2455 Mid Western HWY, FITZGERALDS MOUNT, NSW 2799 Storing 010.2022.00000423.001 - Evans Plains RD, EVANS PLAINS, NSW 2795 Storing 010.2022.00000381.001 - 11 Gell PL, ABERCROMBIE, NSW 2795 Storing 018.2022.00000160.001 - 46 Brennan DR, KELSO, NSW 2795 Storing 010.2022.00000418.001 - 102 Esrom ST, WEST BATHURST, NSW 2795 Storing 010.2022.00000411.001 - Mountain STGT, MOUNT PANORAMA, NSW 2795 Storing 018.2022.00000158.001 - 79 Basalt WAY, KELSO, NSW 2795 Storing 010.2022.00000427.001 - 87 Lorimer ST, LLANARTH, NSW 2795 Storing 010.2022.00000385.001 - 11 Vittoria ST, WEST BATHURST, NSW 2795 Storing 010.2022.00000414.001 - 108 Marys LA, DUNKELD, NSW 2795 Storing 018.2022.00000161.002 - 88 George Thomas CL, THE LAGOON, NSW 2795 Storing 018.2022.00000161.001 - 88 George Thomas CL, THE LAGOON, NSW 2795 Storing 018.2022.00000162.001 - 235 Russell ST, BATHURST, NSW 2795 Storing 010.2022.00000380.001 - 102 William ST, BATHURST, NSW 2795 Storing 010.2022.00000397.001 - 65 Bant ST, SOUTH BATHURST, NSW 2795 Storing 010.2022.00000404.001 - 24 Crawford CR, WINDRADYNE, NSW 2795 Storing 010.2022.00000394.001 - 12 Eugenie ST, RAGLAN, NSW 2795 Storing 010.2022.00000422.001 - 46 Crescent Orchard RD, LOCKSLEY, NSW 2795 Storing 010.2022.00000371.001 - 2 Appledore DR, KELSO, NSW 2795 Storing 010.2022.00000417.001 - 14 Northcott DR, WEST BATHURST, NSW 2795 Storing 010.2022.00000416.001 - 2361 Tarana RD, TARANA, NSW 2787 Storing 010.2022.00000288.001 - 110 William ST, BATHURST, NSW 2795 Storing 010.2022.00000402.001 - 12 Cain DR, KELSO, NSW 2795 Storing 018.2022.00000120.001 - 5 Lyon CL, KELSO, NSW 2795 Storing 010.2022.00000396.001 - 34 College RD, SOUTH BATHURST, NSW 2795 Storing 010.2022.00000389.001 - 20 Fairleigh PL, KELSO, NSW 2795 Storing 010.2022.00000410.001 - 3733 O'Connell RD, KELSO, NSW 2795 Storing 010.2022.00000407.001 - 67 Corporation AVE, ROBIN HILL, NSW 2795 Storing 010.2022.00000421.001 - 171 Howick ST, BATHURST, NSW 2795 Storing 010.2022.00000413.001 - 26 Henry ST, BATHURST, NSW 2795 Storing 010.2021.00000036.002 - 235 Bentinck ST, BATHURST, NSW 2795 Storing 010.2022.00000391.001 - 107 William ST, BATHURST, NSW 2795 Storing 010.2022.00000403.001 - 27 Brennan DR, KELSO, NSW 2795 Storing 010.2022.00000395.001 - 64 Tandora ST, KELSO, NSW 2795 Storing 010.2022.00000401.001 - 6 Topaz CT, KELSO, NSW 2795 Storing 010.2022.00000368.001 - 32 Bassett DR, WEST BATHURST, NSW 2795 Storing 010.2022.00000373.001 - Marsden LA, KELSO, NSW 2795 Storing 010.2022.00000378.001 - 11 Kemp ST, EGLINTON, NSW 2795 Storing 010.2022.00000406.001 - 78 Stanley ST, BATHURST, NSW 2795 Storing 010.2022.00000398.001 - 814 Hen & Chicken LA, EVANS PLAINS, NSW 2795 Storing 010.2022.00000388.001 - 3 Wigmore DR, ROBIN HILL, NSW 2795 Storing 010.2022.00000405.001 - 53 Durham ST, BATHURST, NSW 2795 Storing 010.2022.00000370.001 - 465 Mount Rankin RD, MOUNT RANKIN, NSW 2795 Storing 010.2022.00000387.001 - 2212 Turondale RD, TURONDALE, NSW 2795 Storing 018.2022.00000151.001 - 217 Eusdale RD, YETHOLME, NSW 2795 Storing 018.2022.00000150.001 - 7/14 Stockland DR, KELSO, NSW 2795 Storing 018.2022.00000150.002 - 7/14 Stockland DR, KELSO, NSW 2795 Storing 010.2022.00000293.001 - 1552 Freemantle RD, WATTON, NSW 2795 Collecting ATDIS feed data for kiama... D, [2022-10-04T10:45:34.254751 #17] DEBUG -- : Fetching https://da.kiama.nsw.gov.au/atdis/1.0/applications.json?lodgement_date_end=2022-10-04&lodgement_date_start=2022-09-04&page=3 D, [2022-10-04T10:45:41.914082 #17] DEBUG -- : Fetching Storing 010.2022.00000140.001 - 75 Merrick CCT, KIAMA, NSW 2533 Storing 010.2022.00000170.001 - Panama ST, BOMBO, NSW 2533 Storing 010.2022.00000180.001 - 7B Sommerville CL, KIAMA, NSW 2533 Storing 010.2020.00000161.003 - 159 Princes HWY, KIAMA HEIGHTS, NSW 2533 Storing 010.2021.00000017.002 - 59 Headland DR, GERROA, NSW 2534 Storing 010.2022.00000166.001 - 9 Tasman DR, GERRINGONG, NSW 2534 Storing 010.2022.00000156.001 - 59 Kalang RD, KIAMA HEIGHTS, NSW 2533 Storing 010.2022.00000161.001 - 137 Headland DR, GERROA, NSW 2534 Storing 010.2022.00000176.001 - 18 Irvine ST, KIAMA, NSW 2533 Storing 010.2022.00000158.001 - 541 Jamberoo RD, JAMBEROO, NSW 2533 Storing 010.2022.00000172.001 - 81 Jerrara RD, JERRARA, NSW 2533 Storing 010.2022.00000173.001 - 39 Union WY, GERRINGONG, NSW 2534 Storing 010.2022.00000169.001 - 108 Stafford ST, GERROA, NSW 2534 Storing 010.2022.00000144.001 - 10 Keppel ST, KIAMA, NSW 2533 Storing 010.2022.00000171.001 - 12 Archibald RD, GERRINGONG, NSW 2534 Storing 010.2022.00000168.001 - 4 Boona AV, GERRINGONG, NSW 2534 Storing 010.2022.00000159.001 - 37 Eureka AV, KIAMA DOWNS, NSW 2533 Storing 010.2020.00000221.002 - 24 Barney ST, KIAMA, NSW 2533 Storing 010.2022.00000167.001 - 22 Arnold CR, KIAMA, NSW 2533 Storing 010.2020.00000013.002 - 5 Surfleet PL, KIAMA, NSW 2533 Storing 010.2022.00000153.001 - 42 Jupiter ST, GERRINGONG, NSW 2534 Storing 010.2022.00000163.001 - 1 Evans PL, KIAMA, NSW 2533 Storing 010.2022.00000162.001 - 59 Churchill ST, JAMBEROO, NSW 2533 Storing 010.2022.00000157.001 - 18 Henry Parkes DR, KIAMA DOWNS, NSW 2533 Storing 010.2022.00000160.001 - 23 Riverleigh AV, GERROA, NSW 2534 Storing 010.2022.00000151.001 - 28 Headland DR, GERROA, NSW 2534

Data

Downloaded 13315 times by openaustralia DevAppProject dcalde eduardomontero muzzie86 mattykn vietbaracon WeaponizedBytes

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (5.82 MB) Use the API

rows 10 / 12732

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to authority_label
010.2019.00000164.001
119 Lochview Farm RD, LOCHIEL, NSW 2549
Installation of above ground swimming pool
2019-06-06
2019-05-07
bega_valley
010.2012.00000402.002
5 Bells PL, DOCTOR GEORGE MOUNTAIN, NSW 2550
Modification Erection of a dwelling & detached garage/studio & carport
2019-06-06
2019-05-07
2019-05-23
2019-06-06
bega_valley
018.2019.00005032.001
10 Willie Wagtail PL, ALSTONVILLE, NSW 2477
PCA CDC - New Dwelling - Single Residence
2019-06-06
2019-05-07
ballina
010.2019.00000283.001
2/188-202 Southern Cross DR, BALLINA, NSW 2478
Change of Use and Alterations to Commercial Building Change of use from a high technology industry to general industry involving the assembly, manufacture and fitting of prosthetic limbs and alterations to the building
2019-06-06
2019-05-07
ballina
010.2019.00000284.001
84 Meridian DR, COOLGARDIE, NSW 2478
New Dwelling - Single Residence Construction of a two storey dwelling house and associated studio as depicted in plans by Dean Poile Designs for 64 Meridian Drive Coolgardie dated 2 May 2019
2019-06-06
2019-05-07
ballina
010.2019.00000139.001
141 Keppel ST, BATHURST, NSW 2795
Stage 1 - Five lot torens title subdivision (four residential lots and one development lot) Stage 2 - Seven lot strata title subdivision
2019-06-06
2019-05-07
bathurst
018.2019.00000051.001
10 Newlands CR, KELSO, NSW 2795
Construction of a Single storey dwelling with attached garage
2019-06-06
2019-05-07
bathurst
010.2019.00000138.001
82 Graham DR, KELSO, NSW 2795
Dual Occupancy and two lot residential subdivision
2019-06-06
2019-05-07
bathurst
010.2019.00000078.001
21 Boyd ST, MINNAMURRA, NSW 2533
dwelling & air conditioning unit
2019-06-06
2019-05-07
kiama
010.2019.00000080.001
133 Headland DR, GERROA, NSW 2534
detached storage shed
2019-06-06
2019-05-07
2019-05-23
2019-06-06
kiama

Statistics

Average successful run time: 4 minutes

Total run time: 6 days

Total cpu time used: about 1 hour

Total disk space used: 6.77 MB

History

  • Auto ran revision 5afa7a3a and completed successfully .
    88 records added, 81 records removed in the database
  • Auto ran revision 5afa7a3a and completed successfully .
    65 records added, 65 records removed in the database
  • Auto ran revision 5afa7a3a and completed successfully .
    69 records added, 69 records removed in the database
  • Auto ran revision 5afa7a3a and completed successfully .
    70 records added, 70 records removed in the database
  • Auto ran revision 5afa7a3a and completed successfully .
    71 records added, 69 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

multiple_atdis / scraper.rb