ianheggie-oaf / multiple_horizon-prs

Test All multiple_horizon PRs

README.md

This is a scraper that runs on Morph. To get started see the documentation

Add any issues to https://github.com/planningalerts-scrapers/issues/issues

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

What an Horizon/SolOrient website looks like

Screenshots

(To update this screenshot run bundle exec rake screenshots)

Contributors mlandauer tseyiokorodudu ianheggie-oaf jamezpolley simonzippy

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Scraping authorities: cowra, liverpool_plains, maitland, walcha, warrumbungle, weddin Collecting feed data for cowra... cowra: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) Collecting feed data for liverpool_plains... /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:30:in `<main>' Collecting feed data for maitland... liverpool_plains: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:30:in `<main>' Collecting feed data for walcha... Collecting feed data for warrumbungle... walcha: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:30:in `<main>' Collecting feed data for weddin... warrumbungle: ERROR: 403 "Forbidden" /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:258:in `error!' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:267:in `value' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1293:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:30:in `<main>' *************************************************** Now retrying authorities which earlier had failures *************************************************** Collecting feed data for cowra... weddin: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:30:in `<main>' Collecting feed data for liverpool_plains... cowra: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:37:in `<main>' Collecting feed data for walcha... liverpool_plains: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:37:in `<main>' walcha: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) Collecting feed data for warrumbungle... /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:37:in `<main>' Collecting feed data for weddin... warrumbungle: ERROR: 403 "Forbidden" /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:258:in `error!' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:267:in `value' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1293:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:37:in `<main>' weddin: ERROR: Failed to open TCP connection to mycouncil.solorient.com.au:443 (execution expired) /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/timeout.rb:196:in `timeout' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:712:in `start' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:641:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/horizon_xml.rb:243:in `scrape_url' /app/lib/horizon_xml.rb:142:in `scrape' scraper.rb:14:in `block in scrape' scraper.rb:10:in `each' scraper.rb:10:in `scrape' scraper.rb:37:in `<main>' scraper.rb:41:in `<main>': There were errors with the following authorities: [:cowra, :liverpool_plains, :walcha, :warrumbungle, :weddin]. See earlier output for details (RuntimeError)

Data

Downloaded 16644 times by ianheggie-oaf

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (432 KB) Use the API

rows 10 / 1394

council_reference address description info_url date_scraped date_received on_notice_from on_notice_to authority_label
DA 2023/9.1
5 Tokyo Terrace COWRA 2794 NSW
Dwelling and portico
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-02-06
cowra
DA 2023/8.1
49 Valley View Road COWRA 2794 NSW
Garage
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-02-06
cowra
DA 2022/56.2
64 Macquarie Street COWRA 2794 NSW
S4.55(1A) Modification to dwelling additions (reduce garage size)
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-02-02
cowra
DA 2020/121.2
14 Richards Drive COWRA 2794 NSW
S4.55(1A) Modification to aircraft hangar and workshop (modify hangar
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-02-02
cowra
DA 2020/109.2
2 Lachlan Street COWRA 2794 NSW
S4.55(1A) Modification to change of use of commercial premises to
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-02-02
cowra
DA 2023/7.1
377 Greenethorpe Road KOORAWATHA 2807 NSW
Dwelling (transportable) and garage
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-01-31
cowra
CDC 2023/2.1
17 Pinot Street COWRA 2794 NSW
New 4 bedroom brick veneer dwelling with attached garage
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-01-31
cowra
DA 2022/75.2
Melaleuca Circuit COWRA 2794 NSW
S4.55(1A) Modification to dwelling (relocate dwelling approximately
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-01-25
cowra
DA 2023/6.1
2006 Canowindra Road CANOWINDRA 2804 NSW
boundary adjustment
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-01-20
cowra
DA 2023/5.1
1 Dawson Drive COWRA 2794 NSW
boundary and internal fencing
https://mycouncil.solorient.com.au/Horizon/logonGuest.aw?domain=horizondap_cowra
2025-05-14
2023-01-19
cowra

Statistics

Total run time: about 2 hours

Total cpu time used: 7 minutes

Total disk space used: 3.05 MB

History

  • Auto ran revision e5450b95 and failed .
    nothing changed in the database
  • Auto ran revision e5450b95 and failed .
    nothing changed in the database
  • Auto ran revision e5450b95 and failed .
    nothing changed in the database
  • Auto ran revision e5450b95 and failed .
    nothing changed in the database
  • Auto ran revision e5450b95 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

multiple_horizon-prs / scraper.rb