This is a scraper that runs on Morph. To get started see the documentation
Add any issues to https://github.com/planningalerts-scrapers/issues/issues
bash
bundle exec ruby scraper.rb
bash
bundle exec rake
OR
bash
bin/rake
Some of the authorities require an Australia proxy otherwise they either timeout or error out. They are tagged australian_proxy: true.
I have created the down_under project for my own australian proxy. You are free to use my project, but you will need to pay for your own linode VPS.
Set MORPH_AUSTRALIA_PROXY
to the proxy setting reported by bin/status
Testing is performed using a cached copy of the external sites. Clobber the cache to force testing against recent copies of external site results.
bash
bundle exec rake clobber_cache
OR
bash
bin/rake clobber_cache
bash
bin/rake morph_status:report
Produces a table report by authority of:
And then a Summary by Status detailing recommended Actions to Take
You will need to set an environment variable (eg via .envrc file and direnv command):
bash
bundle exec rubocop
Optionally set the following ENV variables before running scraper.rb or tests:
LIMIT=5
- to limit the number of records to 5 (for example) for quick test (different cache files used)AUTHORITIES=name,name
- to run/test with a subset of authoritiesDEBUG=1
- to show the GET and POST calls to the external siteFAIL_FAST=1
- to make spec stop on first failureUNKNOWN_IS_NOT_FATAL=1
- for when you get sick of adding all the new determinations one test run at a time(To update this screenshot run bundle exec rake screenshots
)
This scraper runs on heroku-18 which has older GLIBC versions. The Gemfile has been configured to use compatible gem versions like nokogiri 1.15.x and sqlite3 1.6.x.
If you encounter GLIBC version errors when deploying (e.g. GLIBC_2.29 not found
), try forcing gem compilation from
source by creating .bundle/config
:
BUNDLEFORCERUBY_PLATFORM: "true" ```
Then rebuild:
bash
rm Gemfile.lock
script/bundle install # Uses the same bundler version (2.3.10) as morph.io
This will compile gems from source rather than using pre-built binaries, which may help with compatibility at the cost of longer installation times.
To download data sign in with GitHub
rows 10 / 1279
council_reference | address | description | info_url | date_scraped | date_received | authority_label |
---|---|---|---|---|---|---|
39/2024 - 1
|
70 Oxford Rd, Scone NSW 2337
|
Dwelling extension
|
2025-01-26
|
2024-12-27
|
upper_hunter
|
|
15.2023.30910.1
|
504 Saunders Av, East Albury NSW 2640
|
Alterations and Additions to Existing Dwelling
|
2025-01-27
|
2025-01-13
|
albury
|
|
11.2024.41041.1
|
5 Enid La, Albury NSW 2640
|
Retaining Walls
|
2025-01-27
|
2025-01-13
|
albury
|
|
11.2024.41201.1
|
1072 Table Top Rd, Table Top NSW 2640
|
Open Bay Colorbond Shed
|
2025-01-27
|
2025-01-08
|
albury
|
|
15.2024.41000.1
|
27 Liverpool Av, Ettamogah NSW 2640
|
Construction of a colorbond shed
|
2025-01-27
|
2025-01-24
|
albury
|
|
11.2024.40960.1
|
595 Mudge St, Hamilton Valley NSW 2641
|
Two (2) Colorbond Sheds
|
2025-01-27
|
2025-01-23
|
albury
|
|
10.2024.41217.1
|
550 Ebden St, South Albury NSW 2640
|
Demolition of Dwelling, Colorbond Shed & Vegetation Removal
|
2025-01-27
|
2025-01-03
|
albury
|
|
11.2024.41152.1
|
28 Bennett Rd, Thurgoona NSW 2640
|
Colorbond Shed & Awning
|
2025-01-27
|
2025-01-23
|
albury
|
|
11.2024.40923.1
|
30 Strand Tce, Ettamogah NSW 2640
|
Colorbond Shed
|
2025-01-27
|
2025-01-10
|
albury
|
|
13.2025.31269.1
|
78 Huntington Gr, Thurgoona NSW 2640
|
Dwelling & Garage
|
2025-01-27
|
2025-01-10
|
albury
|
To download data sign in with GitHub
rows 10 / 36
run_at | authority | records_scraped | use_proxy | seconds | error_message |
---|---|---|---|---|---|
2025-02-09T21:19:16+00:00
|
albury
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
ballina
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
bega_valley
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
broken_hill
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
bundaberg
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
byron
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
cessnock
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
dubbo
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
fairfield
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
2025-02-09T21:19:16+00:00
|
gympie
|
0
|
1
|
0.0
|
undefined method `public_ip' for ScraperUtils::MechanizeUtils:Module
|
Average successful run time: about 1 hour
Total run time: about 18 hours
Total cpu time used: 2 minutes
Total disk space used: 93.6 MB