Logan City Council scraper
- Server - .NET - ApplicationMaster
- Accept Terms - Yes
- Cookie tracking - Yes
- Pagnation - Yes
- Javascript - Yes
- UserAgent - No
- Request actual DA page for data - No
- Clearly defined data within a row - Yes
Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)
Enjoy
Contributors
Last run failed
with status code 1.
Console output of last run
Injecting configuration and compiling...
[1G [1G-----> Ruby app detected
[1G-----> Compiling Ruby
[1G-----> Using Ruby version: ruby-2.0.0-p648
[1G-----> Installing dependencies using bundler 1.15.2
[1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
[1G Fetching gem metadata from https://rubygems.org/........
[1G Fetching version metadata from https://rubygems.org/.
[1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git
[1G Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.
[1G Using bundler 1.15.2
[1G Fetching unf_ext 0.0.7.3
[1G Installing unf_ext 0.0.7.3 with native extensions
[1G Fetching httpclient 2.8.3
[1G Installing httpclient 2.8.3
[1G Fetching mime-types-data 3.2016.0521
[1G Installing mime-types-data 3.2016.0521
[1G Fetching net-http-digest_auth 1.4.1
[1G Installing net-http-digest_auth 1.4.1
[1G Fetching net-http-persistent 2.9.4
[1G Installing net-http-persistent 2.9.4
[1G Fetching mini_portile2 2.1.0
[1G Installing mini_portile2 2.1.0
[1G Fetching ntlm-http 0.1.1
[1G Installing ntlm-http 0.1.1
[1G Fetching webrobots 0.1.2
[1G Installing webrobots 0.1.2
[1G Fetching sqlite3 1.3.13
[1G Installing sqlite3 1.3.13 with native extensions
[1G Fetching unf 0.1.4
[1G Installing unf 0.1.4
[1G Fetching mime-types 3.1
[1G Installing mime-types 3.1
[1G Fetching nokogiri 1.6.8.1
[1G Installing nokogiri 1.6.8.1 with native extensions
[1G Fetching sqlite_magic 0.0.6
[1G Installing sqlite_magic 0.0.6
[1G Fetching domain_name 0.5.20170404
[1G Installing domain_name 0.5.20170404
[1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)
[1G Fetching http-cookie 1.0.3
[1G Installing http-cookie 1.0.3
[1G Fetching mechanize 2.7.5
[1G Installing mechanize 2.7.5
[1G Bundle complete! 2 Gemfile dependencies, 18 gems now installed.
[1G Gems in the groups development and test were not installed.
[1G Bundled gems are installed into ./vendor/bundle.
[1G Bundle completed (28.00s)
[1G Cleaning up the bundler cache.
[1G-----> Detecting rake tasks
[1G
[1G [1G-----> Discovering process types
[1G Procfile declares types -> scraper
Injecting scraper and running...
Getting data in `thisweek`, changable via MORPH_PERIOD environment
Scraping page 1...
scraper.rb:48:in `click': undefined method `[]' for nil:NilClass (NoMethodError)
from scraper.rb:88:in `<main>'
Data
Downloaded
6174 times
by
To download data sign in with GitHub
rows 10 / 42491
| info_url | council_reference | date_received | description | address | date_scraped | comment_url |
|---|---|---|---|---|---|---|
|
MCUC-72/2014
|
2014-04-01
|
Light Industry
|
2/2-16 Beal Street MEADOWBROOK QLD 4131
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-74/2014
|
2014-04-03
|
Dual Occupancy
|
24 John Davison Place CRESTMEAD QLD 4132
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-76/2014
|
2014-04-07
|
Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)
|
1-11 Katherine Court LOGAN VILLAGE QLD 4207
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-78/2014
|
2014-04-07
|
Warehouse
|
9-11 Calcium Court CRESTMEAD QLD 4132
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
RL-17/2014
|
2014-04-04
|
Reconfiguring a Lot (1 into 98 lots)
|
311-331 Logan Reserve Road LOGAN RESERVE QLD 4133
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-73/2014
|
2014-04-02
|
Secondary Dwelling
|
80-90 Greensward Road TAMBORINE QLD 4270
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-75/2014
|
2014-04-03
|
Dual Occupancy
|
13-15 John Davison Place CRESTMEAD QLD 4132
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-77/2014
|
2014-04-02
|
Place of Worship
|
7 University Drive MEADOWBROOK QLD 4131
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
RL-16/2014
|
2014-04-02
|
Reconfiguring a Lot (1 into 2 lots and access easement)
|
11-21 Kingston Road UNDERWOOD QLD 4119
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
COM-1/2012
|
2014-04-02
|
Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road
|
1-13 Chase Close UNDERWOOD QLD 4119
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
Statistics
Average successful run time: 5 minutes
Total run time: 8 days
Total cpu time used: about 1 hour
Total disk space used: 12.8 MB
History
-
Auto
ran
revision
10364b3e
and
failed
.
nothing changed in the database
-
Auto
ran
revision
10364b3e
and
completed successfully
.
132 records added, 126 records removed in the database
-
Auto
ran
revision
10364b3e
and
completed successfully
.
126 records added, 81 records removed in the database
-
Auto
ran
revision
10364b3e
and
completed successfully
.
80 records added, 50 records removed in the database
-
Auto
ran
revision
10364b3e
and
completed successfully
.
48 records added, 23 records removed in the database
- ...
- Created on morph.io