ianheggie-oaf / multiple_nsw_planning_lgas

NSW planning portal LGA DA Applications

README.md

NSW Planning LGAs

This scraper runs on Morph to scrape DA Applications for LGA's that use it.

As Date range filtering is currently broken, it initially searches for All types, but if it finds more than 5 pages it then filters by "DA Status = On Ehxhibition".

When LGA is not set, it currently shows 11,770 results with 1308 pages of 9 cards per page, so its early days so far.

To get started see the documentation

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

Contributors ianheggie-oaf

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Scraping authorities: gunnedah Collecting feed data for gunnedah, attempt: 1... INITIAL URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value= Scraping page 1 for GUNNEDAH - All... Saving record gunnedah - 46-48 Gunnedah St, Carroll NSW 2340 Saving record gunnedah - 6 Vera Close, Gunnedah NSW 2380 Saving record gunnedah - 207 Conadilly Street, Gunnedah NSW 2380 Saving record gunnedah - 16-24 Wentworth Street, Gunnedah NSW 2380 Saving record gunnedah - 179 Black Jack Road, Gunnedah NSW 2380 Saving record gunnedah - 109-139 Kamilaroi Road, Gunnedah NSW 2380 Saving record gunnedah - 5 Anderson Place, Gunnedah NSW 2380 Saving record gunnedah - 13 Henry Street, Gunnedah NSW 2380 Saving record gunnedah - 7 Ashford Street, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=1 Scraping page 2 for GUNNEDAH - All... Saving record gunnedah - 1-15 Winder Place, Gunnedah NSW 2380 Saving record gunnedah - 91Beulah Street, Gunnedah NSW 2380 Saving record gunnedah - 100B Beulah Street, Gunnedah NSW 2380 Saving record gunnedah - 93-95 Marquis Street, Gunnedah NSW 2380 Saving record gunnedah - 396 Conadilly Street, Gunnedah NSW 2380 Saving record gunnedah - 91 Reading Road, Gunnedah NSW 2380 Saving record gunnedah - 124 Stock Road, Gunnedah NSW 2380 Saving record gunnedah - 9 Eighth Division Memorial Avenue, Gunnedah NSW 2380 Saving record gunnedah - 8521 Oxley Highway, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=2 Scraping page 3 for GUNNEDAH - All... Saving record gunnedah - 1693 Blue Vale Rd, Blue Vale NSW Saving record gunnedah - View Street, Gunnedah NSW 2380 Saving record gunnedah - 29-31 Breeza St, Carroll NSW 2340 Saving record gunnedah - 107 Barber Street, Gunnedah NSW 2380 Saving record gunnedah - 107 George Street, Gunnedah NSW 2380 Saving record gunnedah - 42 Bindea Place, Gunnedah NSW 2380 Saving record gunnedah - 73 Maitland Street, Gunnedah NSW 2380 Saving record gunnedah - 23 Palmer Crescent, Gunnedah NSW 2380 Saving record gunnedah - 21 George Street, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=3 Scraping page 4 for GUNNEDAH - All... Saving record gunnedah - 2-42 Kamilaroi Highway, Gunnedah NSW 2380 Saving record gunnedah - 12 Parkview Drive, Gunnedah NSW 2380 Saving record gunnedah - 160 Wandobah Road, Gunnedah NSW 2380 Saving record gunnedah - 100B Beulah Street, Gunnedah NSW 2380 Saving record gunnedah - 1258 The Dip Rd, Curlewis NSW Saving record gunnedah - 131 Quia Road, Gunnedah NSW 2380 Saving record gunnedah - 277 Conadilly Street, Gunnedah NSW 2380 Saving record gunnedah - 25 Booloocooroo Road, Gunnedah NSW 2380 Saving record gunnedah - 170 Bushs Lane, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=4 Scraping page 5 for GUNNEDAH - All... Saving record gunnedah - 309-329 Conadilly Street, Gunnedah NSW 2380 Saving record gunnedah - 49-75 Finch Street, Gunnedah NSW 2380 Saving record gunnedah - 44 Booloocooroo Road, Gunnedah NSW 2380 Saving record gunnedah - 34-36 Silversmith Place, Gunnedah NSW 2380 Saving record gunnedah - 64 Kerry Elizabeth Drive, Gunnedah NSW 2380 Saving record gunnedah - 8 Goran St, Curlewis NSW Saving record gunnedah - 76A Beulah Street, Gunnedah NSW 2380 Saving record gunnedah - 18-22 Kamilaroi Road, Gunnedah NSW 2380 Saving record gunnedah - 150 Little Barber Street, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=5 Scraping page 6 for GUNNEDAH - All... Saving record gunnedah - 418 Quia Road, Gunnedah NSW 2380 Saving record gunnedah - 6B Parkview Drive, Gunnedah NSW 2380 Saving record gunnedah - 27 Booloocooroo Road, Gunnedah NSW 2380 Saving record gunnedah - 37 Wattle Street, Gunnedah NSW 2380 Saving record gunnedah - 35 Wentworth Street, Gunnedah NSW 2380 Saving record gunnedah - 30 Galway Avenue, Gunnedah NSW 2380 Saving record gunnedah - 617 Beeson Road, Gunnedah NSW 2380 Saving record gunnedah - 341-347 Conadilly Street, Gunnedah NSW 2380 Saving record gunnedah - 24-42 Goodwin Road, Gunnedah NSW 2380 NEXT URL: https://www.planningportal.nsw.gov.au/daexhibitions?field_daex_status_value=All&field_local_government_area_value=GUNNEDAH&field_application_type_value=All&field_panel_reference_number_value=&field_council_unique_number_value=&title=&field_exhibition_start_date_value=&field_exhibition_end_date_value=&page=6 Scraping page 7 for GUNNEDAH - All... Saving record gunnedah - 334 Pownall Rd, Mullaley NSW 2379 Saving record gunnedah - 2 Vera Close, Gunnedah NSW 2380 Saving record gunnedah - 54 Hunter Street, Gunnedah NSW 2380 Saving record gunnedah - 90 Marquis Street, Gunnedah NSW 2380 Saving record gunnedah - 809 Oakey Creek Rd, Piallaway NSW 2342 Saving record gunnedah - Modification of Consent NSW Saving record gunnedah - 12 Allgayer Drive, Gunnedah NSW 2380 MORPH_EXPECT_BAD= Scraping Summary: Authority OK Bad Exception -------------------- ------ ------ -------------------------------------------------- gunnedah 61 0 - Exiting with OK status!

Data

Downloaded 11559 times by ianheggie-oaf sufiyanirfan3

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (60 KB) Use the API

rows 10 / 61

council_reference address description info_url date_scraped authority_label
2025.086
46-48 Gunnedah St, Carroll NSW 2340
Outbuilding
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025086-46-48-gunnedah-st-carroll-outbuilding
2025-09-06
gunnedah
2025.085
6 Vera Close, Gunnedah NSW 2380
Dwelling
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025085-6-vera-cl-gunnedah-dwelling
2025-09-06
gunnedah
2025.083
207 Conadilly Street, Gunnedah NSW 2380
Mural
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025083-207-conadilly-st-gunnedah-mural
2025-09-06
gunnedah
2025.082
16-24 Wentworth Street, Gunnedah NSW 2380
Subdivision
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025082-16-24-wentworth-st-gunnedah-subdivision
2025-09-06
gunnedah
2025.078
179 Black Jack Road, Gunnedah NSW 2380
Waste & Resource Transfer Station
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025078-179-black-jack-rd-gunnedah-waste-resource-transfer-station
2025-09-06
gunnedah
2025.077
109-139 Kamilaroi Road, Gunnedah NSW 2380
Subdivision
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025077-109-139-kamilaroi-rd-gunnedah-subdivision
2025-09-06
gunnedah
2025.062
5 Anderson Place, Gunnedah NSW 2380
Garage
https://www.planningportal.nsw.gov.au/daex/determined/2025062-5-anderson-pl-gunnedah-garage
2025-09-06
gunnedah
2025.061
13 Henry Street, Gunnedah NSW 2380
Demolition, Dual Occupancy & Subdivision
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025061-13-henry-street-gunnedah-demolition-dual-occupancy-subdivision
2025-09-06
gunnedah
2025.057
7 Ashford Street, Gunnedah NSW 2380
Shed & Carport
https://www.planningportal.nsw.gov.au/daex/under-consideration/2025057-7-ashford-st-gunnedah-shed-carport
2025-09-06
gunnedah
2025.054
1-15 Winder Place, Gunnedah NSW 2380
Bus Shelter
https://www.planningportal.nsw.gov.au/daex/determined/2025054-1-15-winder-pl-gunnedah-bus-shelter
2025-09-06
gunnedah

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (60 KB) Use the API

rows 10 / 31

run_at attempt authority_label records_saved unprocessable_records status error_message error_class error_backtrace
2025-08-07T22:19:28+00:00
1
gunnedah
58
0
successful
2025-08-08T15:59:59+00:00
1
gunnedah
58
0
successful
2025-08-09T12:39:17+00:00
1
gunnedah
58
0
successful
2025-08-10T19:39:19+00:00
1
gunnedah
58
0
successful
2025-08-11T10:01:03+00:00
1
gunnedah
58
0
successful
2025-08-12T01:02:09+00:00
1
gunnedah
58
0
successful
2025-08-13T01:22:52+00:00
1
gunnedah
58
0
successful
2025-08-14T20:08:43+00:00
1
gunnedah
61
0
successful
2025-08-15T13:27:38+00:00
1
gunnedah
61
0
successful
2025-08-16T00:05:14+00:00
1
gunnedah
61
0
successful

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (60 KB) Use the API

rows 10 / 31

run_at attempt duration successful failed interrupted successful_count interrupted_count failed_count public_ip
2025-08-07T22:19:28+00:00
1
9.7
gunnedah
1
0
0
2025-08-08T15:59:59+00:00
1
9.8
gunnedah
1
0
0
2025-08-09T12:39:17+00:00
1
9.2
gunnedah
1
0
0
2025-08-10T19:39:19+00:00
1
9.7
gunnedah
1
0
0
2025-08-11T10:01:03+00:00
1
12.4
gunnedah
1
0
0
2025-08-12T01:02:09+00:00
1
9.8
gunnedah
1
0
0
2025-08-13T01:22:52+00:00
1
9.7
gunnedah
1
0
0
2025-08-14T20:08:43+00:00
1
9.6
gunnedah
1
0
0
2025-08-15T13:27:38+00:00
1
9.3
gunnedah
1
0
0
2025-08-16T00:05:14+00:00
1
9.9
gunnedah
1
0
0

Statistics

Average successful run time: less than 20 seconds

Total run time: 31 minutes

Total cpu time used: 5 minutes

Total disk space used: 499 KB

History

  • Auto ran revision 24915c2b and completed successfully .
    63 records added, 63 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    63 records added, 63 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    63 records added, 63 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    63 records added, 63 records removed in the database
  • Auto ran revision 24915c2b and completed successfully .
    63 records added, 63 records removed in the database
  • ...
  • Created on morph.io

Show complete history