ianheggie-oaf / city_of_whittlesea_aura_planning_register
City of Whittlesea Planning Register from aura backend
README
This is a scraper that runs on Morph. To get started see the documentation
- Previously a civica based site, discontinued 28 Oct 2023 as documented by City of Whittlesea - Issue #849
Installing
Select ruby using your favourite ruby version manager, for example:
shell
chruby 3.2.2
Then install gems using bundler. Currently morph.io has bundler v1.15.2 installed, so you need to force the installation and use of earlier version. Ignore the warnings to upgrade bundler till morph.io ruby runners are updated.
gem install bundler -v 1.15.2
bundle _1.15.2_ install
Running scraper
bundle exec ruby scraper.rb
Useful links
- Main Site
/ Services
/ Building, planning and development
/ Planning
/ Planning services and online forms
/ Online planning application portal
-> online planning application portal
which presents apply for permit link form
- Home (Home of the online portal)
- Planning Services & Register - search form for Planning Register
Contributors
Last run failed
with status code 1.
Console output of last run
Injecting configuration and compiling...
Injecting scraper and running...
WARN: data_count: Ignoring: no such table: data [returning 0]
INFO: DB has 0 records.
INFO: Visiting website using capybara, will retry with updated chromedriver if necessary ...
2024-10-07 19:17:15 INFO Selenium [:logger_info] Details on how to use and modify Selenium logger:
https://selenium.dev/documentation/webdriver/troubleshooting/logging
2024-10-07 19:17:15 WARN Selenium [:selenium_manager] The chromedriver version (103.0.5060.53) detected in PATH at /usr/local/bin/chromedriver might not be compatible with the detected chrome version (104.0.5112.79); currently, chromedriver 104.0.5112.79 is recommended for chrome 104.*, so it is advised to delete the driver in PATH and retry
ERROR: Error visiting URL: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
INFO: Attempting to update ChromeDriver with default version and retry...
ERROR: Chrome failed to start on retry: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
ERROR: Chrome logs:
ERROR:
cat: /tmp/chrome_debug.log: No such file or directory
ERROR: Error retrying to visit URL: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
INFO: Attempting to determine required chromedriver to update ChromeDriver and retry...
INFO: Found chrome version: 104
Chrome major version is: 104
Found Chromedriver major version: 104
INFO: Found compatible ChromeDriver version: 104.0.5112.79
ERROR: Chrome failed to start on retry: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
ERROR: Chrome logs:
cat: /tmp/chrome_debug.log: No such file or directory
ERROR:
INFO: Quitting capybara
/app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/response.rb:63:in `add_cause': unknown error: Chrome failed to start: exited abnormally. (Selenium::WebDriver::Error::UnknownError)
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/response.rb:41:in `error'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/response.rb:52:in `assert_ok'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/response.rb:34:in `initialize'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/http/common.rb:101:in `new'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/http/common.rb:101:in `create_response'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/http/default.rb:103:in `request'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/http/common.rb:67:in `call'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/bridge.rb:685:in `execute'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/remote/bridge.rb:76:in `create_session'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/common/driver.rb:323:in `block in create_bridge'
from <internal:kernel>:90:in `tap'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/common/driver.rb:322:in `create_bridge'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/common/driver.rb:73:in `initialize'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/chrome/driver.rb:35:in `initialize'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/common/driver.rb:47:in `new'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver/common/driver.rb:47:in `for'
from /app/vendor/bundle/ruby/3.2.0/gems/selenium-webdriver-4.25.0/lib/selenium/webdriver.rb:89:in `for'
from /app/vendor/bundle/ruby/3.2.0/gems/capybara-3.40.0/lib/capybara/selenium/driver.rb:75:in `browser'
from /app/vendor/bundle/ruby/3.2.0/gems/capybara-3.40.0/lib/capybara/selenium/driver.rb:95:in `visit'
from /app/vendor/bundle/ruby/3.2.0/gems/capybara-3.40.0/lib/capybara/session.rb:281:in `visit'
from /app/capybara_utilities.rb:63:in `visit_url'
from /app/capybara_utilities.rb:92:in `rescue in rescue in visit_with_retry'
from /app/capybara_utilities.rb:80:in `rescue in visit_with_retry'
from /app/capybara_utilities.rb:74:in `visit_with_retry'
from scraper.rb:174:in `main'
from scraper.rb:222:in `<main>'
#0 0x6205508aa403 <unknown>
#1 0x6205506b0778 <unknown>
#2 0x6205506d4fa9 <unknown>
#3 0x6205506d012b <unknown>
#4 0x62055070b83a <unknown>
#5 0x6205507058f3 <unknown>
#6 0x6205506db0d8 <unknown>
#7 0x6205506dc205 <unknown>
#8 0x6205508f1e3d <unknown>
#9 0x6205508f4db6 <unknown>
#10 0x6205508db13e <unknown>
#11 0x6205508f59b5 <unknown>
#12 0x6205508cf970 <unknown>
#13 0x620550912228 <unknown>
#14 0x6205509123bf <unknown>
#15 0x62055092cabe <unknown>
#16 0x7234a98076db <unknown>
: Selenium::WebDriver::Error::WebDriverError
Statistics
Total run time: 4 minutes
Total cpu time used: less than 20 seconds
Total disk space used: 117 KB
History
-
Manually
ran
revision
dd6939b1
and
failed
.
nothing changed in the database
-
Manually
ran
revision
dd6939b1
and
failed
.
nothing changed in the database
-
Manually
ran
revision
1b43000e
and
failed
.
nothing changed in the database
-
Manually
ran
revision
5c76c446
and
failed
.
nothing changed in the database
-
Manually
ran
revision
efba0a8e
and
failed
.
nothing changed in the database
- ...
- Created on morph.io