Logan City Council scraper
- Server - .NET - ApplicationMaster
- Accept Terms - Yes
- Cookie tracking - Yes
- Pagnation - Yes
- Javascript - Yes
- UserAgent - No
- Request actual DA page for data - No
- Clearly defined data within a row - Yes
Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)
Enjoy
Contributors
Last run completed successfully
.
Console output of last run
Injecting configuration and compiling...
[1G [1G-----> Ruby app detected
[1G-----> Compiling Ruby
[1G-----> Using Ruby version: ruby-2.0.0-p648
[1G-----> Installing dependencies using bundler 1.15.2
[1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
[1G Fetching gem metadata from https://rubygems.org/........
[1G Fetching version metadata from https://rubygems.org/.
[1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git
[1G Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.
[1G Using bundler 1.15.2
[1G Fetching unf_ext 0.0.7.3
[1G Installing unf_ext 0.0.7.3 with native extensions
[1G Fetching httpclient 2.8.3
[1G Installing httpclient 2.8.3
[1G Fetching mime-types-data 3.2016.0521
[1G Installing mime-types-data 3.2016.0521
[1G Fetching net-http-digest_auth 1.4.1
[1G Installing net-http-digest_auth 1.4.1
[1G Fetching net-http-persistent 2.9.4
[1G Installing net-http-persistent 2.9.4
[1G Fetching mini_portile2 2.1.0
[1G Installing mini_portile2 2.1.0
[1G Fetching ntlm-http 0.1.1
[1G Installing ntlm-http 0.1.1
[1G Fetching webrobots 0.1.2
[1G Installing webrobots 0.1.2
[1G Fetching sqlite3 1.3.13
[1G Installing sqlite3 1.3.13 with native extensions
[1G Fetching unf 0.1.4
[1G Installing unf 0.1.4
[1G Fetching mime-types 3.1
[1G Installing mime-types 3.1
[1G Fetching nokogiri 1.6.8.1
[1G Installing nokogiri 1.6.8.1 with native extensions
[1G Fetching sqlite_magic 0.0.6
[1G Installing sqlite_magic 0.0.6
[1G Fetching domain_name 0.5.20170404
[1G Installing domain_name 0.5.20170404
[1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)
[1G Fetching http-cookie 1.0.3
[1G Installing http-cookie 1.0.3
[1G Fetching mechanize 2.7.5
[1G Installing mechanize 2.7.5
[1G Bundle complete! 2 Gemfile dependencies, 18 gems now installed.
[1G Gems in the groups development and test were not installed.
[1G Bundled gems are installed into ./vendor/bundle.
[1G Bundle completed (33.75s)
[1G Cleaning up the bundler cache.
[1G-----> Detecting rake tasks
[1G
[1G [1G-----> Discovering process types
[1G Procfile declares types -> scraper
Injecting scraper and running...
Getting data in `thisweek`, changable via MORPH_PERIOD environment
Scraping page 1...
Saving record MCUC-201/2018 - 2 Andrew Place MARSDEN QLD 4132
Skipping already saved record RL-37/2017
Skipping already saved record COM-77/2017
Saving record OW-170/2018 - 18 Melinda Street MARSDEN QLD 4132
Saving record PD-3540/2017 - 75-87 Kirk Road CHAMBERS FLAT QLD 4133
Skipping already saved record PD-3540/2017
Skipping already saved record PD-3540/2017
Saving record BW-3471/2018 - 5 Balladonia Street WOODRIDGE QLD 4114
Saving record BW-3473/2018 - 8 Lindley Street WOODRIDGE QLD 4114
Saving record BW-3474/2018 - 130 Herriman Court JIMBOOMBA QLD 4280
Scraping page 2...
Saving record BW-3476/2018 - 21 Holder Street LOGANHOLME QLD 4129
Saving record BW-3477/2018 - 3 Welbeck Street LOGAN CENTRAL QLD 4114
Saving record BW-3479/2018 - 44 Anakie Drive CORNUBIA QLD 4130
Saving record BW-3480/2018 - 17-19 Huntingdon Street FORESTDALE QLD 4118
Saving record BW-3483/2018 - 7 Kuprin Street LOGAN RESERVE QLD 4133
Saving record BW-3484/2018 - 82 Killara Boulevard LOGAN RESERVE QLD 4133
Saving record BW-3486/2018 - 6 Andrew Place MARSDEN QLD 4132
Saving record BW-3488/2018 - 140-144 Huntingdale Drive GREENBANK QLD 4124
Saving record BW-3490/2018 - 6 McCormack Court ROCHEDALE SOUTH QLD 4123
Saving record BW-3491/2018 - 2 Florence Street BORONIA HEIGHTS QLD 4124
Scraping page 3...
Saving record BW-3492/2018 - 43-49 Wongawallan Drive YARRABILBA QLD 4207
Saving record SP-92/2018 - 21 Bahrs Scrub Road BAHRS SCRUB QLD 4207
Saving record SP-93/2018 - 2-8 John Street JIMBOOMBA QLD 4280
Data
Downloaded
4269 times
by
To download data sign in with GitHub
rows 10 / 33224
| info_url | council_reference | date_received | description | address | date_scraped | comment_url |
|---|---|---|---|---|---|---|
|
MCUC-72/2014
|
2014-04-01
|
Light Industry
|
2/2-16 Beal Street MEADOWBROOK QLD 4131
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-74/2014
|
2014-04-03
|
Dual Occupancy
|
24 John Davison Place CRESTMEAD QLD 4132
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-76/2014
|
2014-04-07
|
Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)
|
1-11 Katherine Court LOGAN VILLAGE QLD 4207
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-78/2014
|
2014-04-07
|
Warehouse
|
9-11 Calcium Court CRESTMEAD QLD 4132
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
RL-17/2014
|
2014-04-04
|
Reconfiguring a Lot (1 into 98 lots)
|
311-331 Logan Reserve Road LOGAN RESERVE QLD 4133
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-73/2014
|
2014-04-02
|
Secondary Dwelling
|
80-90 Greensward Road TAMBORINE QLD 4270
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-75/2014
|
2014-04-03
|
Dual Occupancy
|
13-15 John Davison Place CRESTMEAD QLD 4132
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
MCUC-77/2014
|
2014-04-02
|
Place of Worship
|
7 University Drive MEADOWBROOK QLD 4131
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
RL-16/2014
|
2014-04-02
|
Reconfiguring a Lot (1 into 2 lots and access easement)
|
11-21 Kingston Road UNDERWOOD QLD 4119
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
|
|
COM-1/2012
|
2014-04-02
|
Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road
|
1-13 Chase Close UNDERWOOD QLD 4119
|
2014-04-08
|
mailto:council@logan.qld.gov.au
|
Statistics
Average successful run time: 6 minutes
Total run time: 8 days
Total cpu time used: about 1 hour
Total disk space used: 10 MB
History
-
Auto
ran
revision
c87b02c3
and
completed successfully
.
19 records added in the database5 pages scraped
-
Auto
ran
revision
c87b02c3
and
failed
.
nothing changed in the database
-
Auto
ran
revision
c87b02c3
and
completed successfully
.
7 records added in the database24 pages scraped
-
Auto
ran
revision
c87b02c3
and
completed successfully
.
70 records added in the database23 pages scraped
-
Auto
ran
revision
c87b02c3
and
completed successfully
.
14 records added in the database16 pages scraped
- ...
- Created on morph.io