This is a scraper that runs on Morph. To get started see the documentation

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

To run the scraper

bash bundle exec ruby scraper.rb

To run the tests

bash bundle exec rake

OR

bash bin/rake

Running under morph.io

Some of the authorities require an Australia proxy otherwise they either timeout or error out. They are tagged australian_proxy: true.

I have created the down_under project for my own australian proxy. You are free to use my project, but you will need to pay for your own linode VPS.

Set MORPH_AUSTRALIA_PROXY to the proxy setting reported by bin/status

To force testing using fresh cache of external sites

Testing is performed using a cached copy of the external sites. Clobber the cache to force testing against recent copies of external site results.

bash bundle exec rake clobber_cache

OR

bash bin/rake clobber_cache

Report on morph.io dev/test and live status pages

bash bin/rake morph_status:report

Produces a table report by authority of:

• Week, Month, Population, Warning - from Morph.io Live Status Page
• Test - local spec/expected/AUTHORITY.yml record count
• Morph - records collected by test of this repo on Morph.io

And then a Summary by Status detailing recommended Actions to Take

You will need to set an environment variable (eg via .envrc file and direnv command):

• MORPHAPIKEY - so the report can query the database updated on morph.io by your local repository

To run style and coding checks

bash bundle exec rubocop

Available filters

Optionally set the following ENV variables before running scraper.rb or tests:

• LIMIT=5 - to limit the number of records to 5 (for example) for quick test (different cache files used)
• AUTHORITIES=name,name - to run/test with a subset of authorities
• DEBUG=1 - to show the GET and POST calls to the external site
• FAIL_FAST=1 - to make spec stop on first failure

Production variables

These variables allow for silencing errors until the site is fixed (in the case of EXPECT_BAD) or This scraper is updated to handle the new variations.

• MORPH_EXPECT_BAD - List authorities that currently block access
• MORPH_EXPECT_POOR list of sites with unprocessable records

If the authority stops having errors, then the program will issue an error telling you to remove them from the list.

Note - these do NOT stop the sites from being tried, just if scraping failures cause the program to exit with an error status.

What a Masterview website looks like

(To update this screenshot run bundle exec rake screenshots)

Bundle Configuration

GLIBC Compatibility

This scraper runs on heroku-18 which has older GLIBC versions. The Gemfile has been configured to use compatible gem versions like nokogiri 1.15.x and sqlite3 1.6.x.

If you encounter GLIBC version errors when deploying (e.g. GLIBC_2.29 not found), try forcing gem compilation from source by creating .bundle/config as follows

```yaml

BUNDLEFORCERUBY_PLATFORM: "true" ```

Then rebuild:

bash rm Gemfile.lock script/bundle install # Uses the same bundler version (2.3.10) as morph.io

This will compile gems from source rather than using pre-built binaries, which may help with compatibility at the cost of longer installation times.