maximlevey / masterview_Scraper

README.md

This is a scraper that runs on Morph. To get started see the documentation

To run the scraper

bundle exec ruby scraper.rb

To run the tests

bundle exec rake

To run style and coding checks

bundle exec rubocop

What a Masterview website looks like

Sign up

(To update this screenshot run bundle exec rake screenshots)

Contributors mlandauer ianheggie-oaf simonzippy njenkins maximlevey

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Default EXPECT_BAD: --- :bogan: https://github.com/planningalerts-scrapers/issues/issues/935 - Council website bad, does not publish Scraping authorities: bayside Collecting feed data for bayside, attempt: 1... bayside: ERROR: end of file reached /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:237:in `rbuf_fill' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:199:in `readuntil' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:209:in `readline' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:158:in `read_status_line' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:147:in `read_new' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1862:in `block in transport_request' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1853:in `catch' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1853:in `transport_request' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1826:in `request' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:948:in `block in request' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:652:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' *************************************************** Now retrying authorities which earlier had failures bayside *************************************************** /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/masterview_scraper.rb:150:in `scrape_url' Collecting feed data for bayside, attempt: 2... /app/lib/masterview_scraper.rb:55:in `scrape_period' /app/lib/masterview_scraper.rb:19:in `scrape' scraper.rb:21:in `block in scrape' scraper.rb:17:in `each' scraper.rb:17:in `scrape' scraper.rb:44:in `run' scraper.rb:86:in `<main>' bayside: ERROR: end of file reached /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:237:in `rbuf_fill' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:199:in `readuntil' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:209:in `readline' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:158:in `read_status_line' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:147:in `read_new' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1862:in `block in transport_request' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1853:in `catch' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1853:in `transport_request' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1826:in `request' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:948:in `block in request' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:652:in `connection_for' /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.6/lib/net/http/persistent.rb:942:in `request' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/masterview_scraper.rb:150:in `scrape_url' /app/lib/masterview_scraper.rb:55:in `scrape_period' /app/lib/masterview_scraper.rb:19:in `scrape' scraper.rb:21:in `block in scrape' scraper.rb:17:in `each' scraper.rb:17:in `scrape' scraper.rb:59:in `run' scraper.rb:86:in `<main>' MORPH_EXPECT_BAD=bogan Scraping Summary: Authority OK Bad Exception -------------------- ------ ------ -------------------------------------------------- /app/vendor/bundle/ruby/3.2.0/gems/scraper_utils-0.9.1/lib/scraper_utils/log_utils.rb:192:in `report_on_results': ERROR: Unexpected errors in: bayside (Add to MORPH_EXPECT_BAD?) (RuntimeError) bayside: EOFError - end of file reached bayside 0 0 EOFError - end of file reached [lib/masterview_scraper.rb:150:in `scrape_url'] See earlier output for details from scraper.rb:69:in `run' from scraper.rb:86:in `<main>'

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (28 KB) Use the API

rows 4 / 4

run_at attempt authority_label records_saved unprocessable_records status error_message error_class error_backtrace
2025-09-10T04:47:22+00:00
1
bayside
0
0
failed
403 => Net::HTTPForbidden for https://www.bayside.vic.gov.au/services/planning-and-building/planning-application-status/ -- unhandled response
Mechanize::ResponseCodeError
/app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:333:in `fetch' /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get' /app/lib/masterview_scraper.rb:85:in `scrape_api_period' /app/lib/masterview_scraper.rb:45:in `scrape_period' /app/lib/masterview_scraper.rb:19:in `scrape' scraper.rb:21:in `block in scrape'
2025-09-10T04:47:23+00:00
2
bayside
0
0
failed
end of file reached
EOFError
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:237:in `rbuf_fill' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:199:in `readuntil' /app/lib/masterview_scraper.rb:85:in `scrape_api_period' /app/lib/masterview_scraper.rb:45:in `scrape_period' /app/lib/masterview_scraper.rb:19:in `scrape' scraper.rb:21:in `block in scrape'
2025-09-10T04:53:00+00:00
1
bayside
0
0
failed
end of file reached
EOFError
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:237:in `rbuf_fill' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:199:in `readuntil' /app/lib/masterview_scraper.rb:150:in `scrape_url' /app/lib/masterview_scraper.rb:55:in `scrape_period' /app/lib/masterview_scraper.rb:19:in `scrape' scraper.rb:21:in `block in scrape'
2025-09-10T04:53:01+00:00
2
bayside
0
0
failed
end of file reached
EOFError
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:237:in `rbuf_fill' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:199:in `readuntil' /app/lib/masterview_scraper.rb:150:in `scrape_url' /app/lib/masterview_scraper.rb:55:in `scrape_period' /app/lib/masterview_scraper.rb:19:in `scrape' scraper.rb:21:in `block in scrape'

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (28 KB) Use the API

rows 4 / 4

run_at attempt duration successful failed interrupted successful_count interrupted_count failed_count public_ip
2025-09-10T04:47:22+00:00
1
0.5
bayside
0
0
1
2025-09-10T04:47:23+00:00
2
0.8
bayside
0
0
1
2025-09-10T04:53:00+00:00
1
0.8
bayside
0
0
1
2025-09-10T04:53:01+00:00
2
0.6
bayside
0
0
1

Statistics

Total run time: less than 20 seconds

Total cpu time used: less than 10 seconds

Total disk space used: 133 MB

History

  • Manually ran revision 9a141ad8 and failed .
    4 records added in the database
  • Manually ran revision 27327e97 and failed .
    4 records added in the database
  • Manually ran revision 3399ec3d and failed .
  • Created on morph.io

Scraper code

Ruby

masterview_Scraper / scraper.rb