ianheggie-oaf / multiple_nsw_planning_lgas

NSW planning portal LGA DA Applications

README.md

NSW Planning LGAs

This scraper runs on Morph to scrape DA Applications for LGA's that use it.

As Date range filtering is currently broken, it initially searches for All types, but if it finds more than 5 pages it then filters by "DA Status = On Ehxhibition".

When LGA is not set, it currently shows 11,770 results with 1308 pages of 9 cards per page, so its early days so far.

To get started see the documentation

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

Contributors ianheggie-oaf

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Scraping authorities: gunnedah Collecting feed data for gunnedah, attempt: 1... INITIAL URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value= Scraping page 1 for GUNNEDAH - All... Saving record gunnedah - 9 Eighth Division Memorial Avenue, Gunnedah NSW 2380 Saving record gunnedah - 8521 Oxley Highway, Gunnedah NSW 2380 Saving record gunnedah - 1693 Blue Vale Rd, Blue Vale NSW Saving record gunnedah - View Street, Gunnedah NSW 2380 Saving record gunnedah - 29-31 Breeza St, Carroll NSW 2340 Saving record gunnedah - 107 Barber Street, Gunnedah NSW 2380 Saving record gunnedah - 107 George Street, Gunnedah NSW 2380 Saving record gunnedah - 42 Bindea Place, Gunnedah NSW 2380 Saving record gunnedah - 73 Maitland Street, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=1 Scraping page 2 for GUNNEDAH - All... Saving record gunnedah - 23 Palmer Crescent, Gunnedah NSW 2380 Saving record gunnedah - 21 George Street, Gunnedah NSW 2380 Saving record gunnedah - 2-42 Kamilaroi Highway, Gunnedah NSW 2380 Saving record gunnedah - 12 Parkview Drive, Gunnedah NSW 2380 Saving record gunnedah - 160 Wandobah Road, Gunnedah NSW 2380 Saving record gunnedah - 100B Beulah Street, Gunnedah NSW 2380 Saving record gunnedah - 1258 The Dip Rd, Curlewis NSW Saving record gunnedah - 131 Quia Road, Gunnedah NSW 2380 Saving record gunnedah - 277 Conadilly Street, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=2 Scraping page 3 for GUNNEDAH - All... Saving record gunnedah - 25 Booloocooroo Road, Gunnedah NSW 2380 Saving record gunnedah - 170 Bushs Lane, Gunnedah NSW 2380 Saving record gunnedah - 309-329 Conadilly Street, Gunnedah NSW 2380 Saving record gunnedah - 49-75 Finch Street, Gunnedah NSW 2380 Saving record gunnedah - 44 Booloocooroo Road, Gunnedah NSW 2380 Saving record gunnedah - 34-36 Silversmith Place, Gunnedah NSW 2380 Saving record gunnedah - 64 Kerry Elizabeth Drive, Gunnedah NSW 2380 Saving record gunnedah - 8 Goran St, Curlewis NSW Saving record gunnedah - 76A Beulah Street, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=3 Scraping page 4 for GUNNEDAH - All... Saving record gunnedah - 18-22 Kamilaroi Road, Gunnedah NSW 2380 Saving record gunnedah - 150 Little Barber Street, Gunnedah NSW 2380 Saving record gunnedah - 418 Quia Road, Gunnedah NSW 2380 Saving record gunnedah - 6B Parkview Drive, Gunnedah NSW 2380 Saving record gunnedah - 27 Booloocooroo Road, Gunnedah NSW 2380 Saving record gunnedah - 37 Wattle Street, Gunnedah NSW 2380 Saving record gunnedah - 35 Wentworth Street, Gunnedah NSW 2380 Saving record gunnedah - 30 Galway Avenue, Gunnedah NSW 2380 Saving record gunnedah - 617 Beeson Road, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=4 Scraping page 5 for GUNNEDAH - All... Saving record gunnedah - 341-347 Conadilly Street, Gunnedah NSW 2380 Saving record gunnedah - 24-42 Goodwin Road, Gunnedah NSW 2380 Saving record gunnedah - 334 Pownall Rd, Mullaley NSW 2379 Saving record gunnedah - 2 Vera Close, Gunnedah NSW 2380 Saving record gunnedah - 54 Hunter Street, Gunnedah NSW 2380 Saving record gunnedah - 90 Marquis Street, Gunnedah NSW 2380 Saving record gunnedah - 809 Oakey Creek Rd, Piallaway NSW 2342 Saving record gunnedah - Modification of Consent NSW NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=5 Scraping page 6 for GUNNEDAH - All... Saving record gunnedah - 12 Allgayer Drive, Gunnedah NSW 2380 MORPH_EXPECT_BAD= Scraping Summary: Authority OK Bad Exception -------------------- ------ ------ -------------------------------------------------- gunnedah 45 0 - Exiting with OK status!

Data

Downloaded 478 times by ianheggie-oaf

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (56 KB) Use the API

rows 10 / 45

council_reference address description info_url date_scraped authority_label
2025.028
9 Eighth Division Memorial Avenue, Gunnedah NSW 2380
Shed
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025028-9-eighth-division-memorial-ave-gunnedah-shed
2025-05-11
gunnedah
2025.025
8521 Oxley Highway, Gunnedah NSW 2380
Dwelling
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025025-8521-oxley-hwy-gunnedah-dwelling
2025-05-11
gunnedah
2025.019
1693 Blue Vale Rd, Blue Vale NSW
Temporary Event Infrastructure
https://www.planningportal.nsw.gov.au/daex/determined/2025019-1693-blue-vale-rd-blue-vale-temporary-event-infrastructure
2025-05-11
gunnedah
2025.011
View Street, Gunnedah NSW 2380
Circus
https://www.planningportal.nsw.gov.au/daex/determined/2025011-view-street-gunnedah-circus
2025-05-11
gunnedah
2024/098
29-31 Breeza St, Carroll NSW 2340
Shed
https://www.planningportal.nsw.gov.au/daex/determined/2024098-29-31-breeza-st-carroll-shed
2025-05-11
gunnedah
202/2017.002
107 Barber Street, Gunnedah NSW 2380
Mod to Stage Development
https://www.planningportal.nsw.gov.au/daex/under-consideration/2022017002-107-barber-st-gunnedah-mod-stage-development
2025-05-11
gunnedah
2024.095
107 George Street, Gunnedah NSW 2380
Dual Occupancy & Subdivision
https://www.planningportal.nsw.gov.au/daex/determined/2024095-107-george-st-gunnedah-dual-occupancy-subdivision
2025-05-11
gunnedah
2024.097
42 Bindea Place, Gunnedah NSW 2380
Dwelling, Garage & Pool
https://www.planningportal.nsw.gov.au/daex/under-consideration/2024097-42-bindea-pl-gunnedah-dwelling-garage-pool
2025-05-11
gunnedah
2025.012
73 Maitland Street, Gunnedah NSW 2380
Dwelling
https://www.planningportal.nsw.gov.au/daex/determined/2025012-73-maitland-st-gunnedah-dwelling
2025-05-11
gunnedah
2024.093
23 Palmer Crescent, Gunnedah NSW 2380
Home Business
https://www.planningportal.nsw.gov.au/daex/determined/2024093-23-palmer-cres-gunnedah-home-business
2025-05-11
gunnedah

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (56 KB) Use the API

rows 6 / 6

run_at attempt authority_label records_saved unprocessable_records status error_message error_class error_backtrace
2025-05-06T21:03:06+00:00
1
gunnedah
45
0
successful
2025-05-07T23:26:12+00:00
1
gunnedah
45
0
successful
2025-05-08T20:29:10+00:00
1
gunnedah
45
0
successful
2025-05-09T03:30:10+00:00
1
gunnedah
45
0
successful
2025-05-10T11:24:15+00:00
1
gunnedah
45
0
successful
2025-05-11T02:33:11+00:00
1
gunnedah
45
0
successful

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (56 KB) Use the API

rows 6 / 6

run_at attempt duration successful failed interrupted successful_count interrupted_count failed_count public_ip
2025-05-06T21:03:06+00:00
1
8.9
gunnedah
1
0
0
2025-05-07T23:26:12+00:00
1
8.6
gunnedah
1
0
0
2025-05-08T20:29:10+00:00
1
7.9
gunnedah
1
0
0
2025-05-09T03:30:10+00:00
1
7.7
gunnedah
1
0
0
2025-05-10T11:24:15+00:00
1
8.7
gunnedah
1
0
0
2025-05-11T02:33:11+00:00
1
8.3
gunnedah
1
0
0

Statistics

Average successful run time: less than 20 seconds

Total run time: 1 minute

Total cpu time used: less than 20 seconds

Total disk space used: 495 KB

History

  • Auto ran revision 24915c2b and completed successfully .
    47 records added, 45 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    47 records added, 45 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    47 records added, 45 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    47 records added, 45 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    47 records added, 45 records removed in the database
  • ...
  • Created on morph.io

Show complete history