planningalerts-scrapers / logan

Scrapes pdonline.logan.qld.gov.au

MasterView 2.0 Application Master


Logan City Council scraper

  • Server - .NET - ApplicationMaster
  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Pagnation - Yes
  • Javascript - Yes
  • UserAgent - No
  • Request actual DA page for data - No
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)

Enjoy

Contributors mlandauer LoveMyData henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0-p648 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Using bundler 1.15.2  Fetching unf_ext 0.0.7.3  Installing unf_ext 0.0.7.3 with native extensions  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (33.75s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1... Saving record MCUC-201/2018 - 2 Andrew Place MARSDEN QLD 4132 Skipping already saved record RL-37/2017 Skipping already saved record COM-77/2017 Saving record OW-170/2018 - 18 Melinda Street MARSDEN QLD 4132 Saving record PD-3540/2017 - 75-87 Kirk Road CHAMBERS FLAT QLD 4133 Skipping already saved record PD-3540/2017 Skipping already saved record PD-3540/2017 Saving record BW-3471/2018 - 5 Balladonia Street WOODRIDGE QLD 4114 Saving record BW-3473/2018 - 8 Lindley Street WOODRIDGE QLD 4114 Saving record BW-3474/2018 - 130 Herriman Court JIMBOOMBA QLD 4280 Scraping page 2... Saving record BW-3476/2018 - 21 Holder Street LOGANHOLME QLD 4129 Saving record BW-3477/2018 - 3 Welbeck Street LOGAN CENTRAL QLD 4114 Saving record BW-3479/2018 - 44 Anakie Drive CORNUBIA QLD 4130 Saving record BW-3480/2018 - 17-19 Huntingdon Street FORESTDALE QLD 4118 Saving record BW-3483/2018 - 7 Kuprin Street LOGAN RESERVE QLD 4133 Saving record BW-3484/2018 - 82 Killara Boulevard LOGAN RESERVE QLD 4133 Saving record BW-3486/2018 - 6 Andrew Place MARSDEN QLD 4132 Saving record BW-3488/2018 - 140-144 Huntingdale Drive GREENBANK QLD 4124 Saving record BW-3490/2018 - 6 McCormack Court ROCHEDALE SOUTH QLD 4123 Saving record BW-3491/2018 - 2 Florence Street BORONIA HEIGHTS QLD 4124 Scraping page 3... Saving record BW-3492/2018 - 43-49 Wongawallan Drive YARRABILBA QLD 4207 Saving record SP-92/2018 - 21 Bahrs Scrub Road BAHRS SCRUB QLD 4207 Saving record SP-93/2018 - 2-8 John Street JIMBOOMBA QLD 4280

Data

Downloaded 4269 times by slav123 openaustralia LoveMyData lewis-CWD TheZepto dsccramb spatialsteve rossrandell dcarterlandsburys peteclowes MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (9.97 MB) Use the API

rows 10 / 33224

info_url council_reference date_received description address date_scraped comment_url
MCUC-72/2014
2014-04-01
Light Industry
2/2-16 Beal Street MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-74/2014
2014-04-03
Dual Occupancy
24 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-76/2014
2014-04-07
Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)
1-11 Katherine Court LOGAN VILLAGE QLD 4207
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-78/2014
2014-04-07
Warehouse
9-11 Calcium Court CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
RL-17/2014
2014-04-04
Reconfiguring a Lot (1 into 98 lots)
311-331 Logan Reserve Road LOGAN RESERVE QLD 4133
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-73/2014
2014-04-02
Secondary Dwelling
80-90 Greensward Road TAMBORINE QLD 4270
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-75/2014
2014-04-03
Dual Occupancy
13-15 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-77/2014
2014-04-02
Place of Worship
7 University Drive MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
RL-16/2014
2014-04-02
Reconfiguring a Lot (1 into 2 lots and access easement)
11-21 Kingston Road UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au
COM-1/2012
2014-04-02
Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road
1-13 Chase Close UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au

Statistics

Average successful run time: 6 minutes

Total run time: 8 days

Total cpu time used: about 1 hour

Total disk space used: 10 MB

History

  • Auto ran revision c87b02c3 and completed successfully .
    19 records added in the database
    5 pages scraped
  • Auto ran revision c87b02c3 and failed .
    nothing changed in the database
  • Auto ran revision c87b02c3 and completed successfully .
    7 records added in the database
    24 pages scraped
  • Auto ran revision c87b02c3 and completed successfully .
    70 records added in the database
    23 pages scraped
  • Auto ran revision c87b02c3 and completed successfully .
    14 records added in the database
    16 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

logan / scraper.rb