Charters Towers Regional Council scraper

This is a scraper that runs on Morph. To get started see the documentation.

It scrapes development applications for Charters Towers Regional Council from the All Development Applications pages, one page per year, and feeds PlanningAlerts.

Issues with this authority are tracked in the shared issue tracker: planningalerts-scrapers/issues#79.

How the site is structured

• Pages from 2022 onwards list one accordion per application. The accordion heading is REF - ADDRESS and the first body paragraph is REF - DESCRIPTION - ADDRESS.
• Pages for 2017-2021 (and a leftover accordion on the 2022 page) are flat lists of decision notice PDFs. A record is extracted when the link text contains an address; most 2017-2019 links carry only a reference so they are skipped.
• The application received date is not published anywhere on these pages, so date_received is not set.
• The site sits behind Akamai, which rejects requests without ordinary browser headers (including the Sec-Fetch-* set).

Running

bundle install bundle exec ruby scraper.rb

Environment variables:

• MORPH_AUSTRALIAN_PROXY - optional; when set (e.g. http://morph:password@au.proxy.oaf.org.au:8888) all requests go through the Australian proxy.
• MORPH_DEBUG - optional; when set, prints each record as it is saved and the reason for every skipped accordion or notice link.

Expected output is a per-year-page count of records found followed by:

Finished - added N records

with N around 200, and a data.sqlite file in the working directory.