planningalerts-scrapers / logan

README.md

Logan City Council scraper

  • Server - .NET - ApplicationMaster
  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Pagnation - Yes
  • Javascript - Yes
  • UserAgent - No
  • Request actual DA page for data - No
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)

Enjoy

Contributors mlandauer LoveMyData henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0-p648 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Using bundler 1.15.2  Fetching unf_ext 0.0.7.3  Installing unf_ext 0.0.7.3 with native extensions  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (27.14s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1... Skipping already saved record MCUC-471/2017 Skipping already saved record RL-84/2014 Skipping already saved record SBR-12/2018 Skipping already saved record BWAP-529/2018 Skipping already saved record BWAP-531/2018 Skipping already saved record BWAP-532/2018 Skipping already saved record BWAP-534/2018 Skipping already saved record BWAP-535/2018 Skipping already saved record BWAP-537/2018 Saving record BWAP-538/2018 - 76-88 Boulia Street GREENBANK QLD 4124 Scraping page 2... Saving record BWAP-539/2018 - 145-151 Swan Road JIMBOOMBA QLD 4280 Skipping already saved record PD-3319/2018 Skipping already saved record PD-3317/2018 Skipping already saved record PD-3318/2018 Skipping already saved record PD-3320/2018 Skipping already saved record PD-3321/2018 Skipping already saved record PD-3321/2018 Skipping already saved record PD-3321/2018 Skipping already saved record PD-3321/2018 Skipping already saved record PD-3322/2018 Scraping page 3... Skipping already saved record PD-3323/2018 Skipping already saved record PD-3324/2018 Skipping already saved record PD-3325/2018 Skipping already saved record PD-3326/2018 Skipping already saved record PD-3327/2018 Skipping already saved record PD-3328/2018 Skipping already saved record PD-3329/2018 Skipping already saved record PD-3330/2018 Skipping already saved record PD-3331/2018 Skipping already saved record PD-3332/2018 Scraping page 4... Skipping already saved record PD-3335/2018 Skipping already saved record PD-3336/2018 Skipping already saved record PD-3337/2018 Skipping already saved record PD-3338/2018 Skipping already saved record PD-3339/2018 Skipping already saved record PD-3341/2018 Skipping already saved record PD-3342/2018 Skipping already saved record PD-3343/2018 Skipping already saved record PD-3344/2018 Skipping already saved record PD-3345/2018 Scraping page 5... Saving record PD-3352/2018 - 9 Skyring Street GREENBANK QLD 4124 Saving record PD-3347/2018 - 134-136 Bidyan Boulevard NEW BEITH QLD 4124 Saving record PD-3348/2018 - 36 Belyando Street HOLMVIEW QLD 4207 Saving record PD-3349/2018 - 5 Willow Circuit YARRABILBA QLD 4207 Saving record PD-3350/2018 - 79 Killara Boulevard LOGAN RESERVE QLD 4133 Saving record PD-3351/2018 - 54 Southwood Circuit YARRABILBA QLD 4207 Saving record PD-3353/2018 - 11 Enclave Drive BAHRS SCRUB QLD 4207 Skipping already saved record BW-9143/2018 Skipping already saved record BW-9144/2018 Skipping already saved record BW-9145/2018 Scraping page 6... Skipping already saved record BW-9147/2018 Skipping already saved record BW-9148/2018 Skipping already saved record BW-9150/2018 Skipping already saved record BW-9152/2018 Skipping already saved record BW-9154/2018 Skipping already saved record BW-9155/2018 Skipping already saved record BW-9156/2018 Skipping already saved record BW-9160/2018 Skipping already saved record BW-9161/2018 Skipping already saved record BW-9164/2018 Scraping page 7... Skipping already saved record BW-9165/2018 Skipping already saved record BW-9167/2018 Skipping already saved record BW-9168/2018 Skipping already saved record BW-9169/2018 Skipping already saved record BW-9170/2018 Skipping already saved record BW-9171/2018 Skipping already saved record BW-9174/2018 Skipping already saved record BW-9175/2018 Skipping already saved record BW-9176/2018 Skipping already saved record BW-9177/2018 Scraping page 8... Skipping already saved record BW-9178/2018 Skipping already saved record BW-9180/2018 Skipping already saved record BW-9182/2018 Skipping already saved record BW-9183/2018 Skipping already saved record BW-9184/2018 Skipping already saved record BW-9185/2018 Skipping already saved record BW-9186/2018 Skipping already saved record BW-9188/2018 Skipping already saved record BW-9189/2018 Skipping already saved record BW-9190/2018 Scraping page 9... Skipping already saved record BW-9191/2018 Skipping already saved record BW-9192/2018 Skipping already saved record BW-9195/2018 Skipping already saved record BW-9196/2018 Skipping already saved record BW-9203/2018 Skipping already saved record BW-9206/2018 Skipping already saved record BW-9209/2018 Skipping already saved record BW-9212/2018 Skipping already saved record BW-9214/2018 Skipping already saved record BW-9216/2018 Scraping page 10... Skipping already saved record BW-9217/2018 Skipping already saved record BW-9218/2018 Skipping already saved record BW-9220/2018 Skipping already saved record BW-9221/2018 Skipping already saved record BW-9223/2018 Skipping already saved record BW-9224/2018 Skipping already saved record BW-9225/2018 Skipping already saved record BW-9227/2018 Skipping already saved record BW-9228/2018 Skipping already saved record BW-9229/2018 Scraping page 11... Skipping already saved record BW-9230/2018 Skipping already saved record BW-9231/2018 Skipping already saved record BW-9233/2018 Skipping already saved record BW-9235/2018 Skipping already saved record BW-9236/2018 Skipping already saved record BW-9239/2018 Skipping already saved record BW-9240/2018 Skipping already saved record BW-9241/2018 Skipping already saved record BW-9242/2018 Skipping already saved record BW-9243/2018 Scraping page 12... Skipping already saved record BW-9245/2018 Skipping already saved record BW-9246/2018 Skipping already saved record BW-9247/2018 Skipping already saved record BW-9249/2018 Skipping already saved record BW-9252/2018 Skipping already saved record BW-9253/2018 Skipping already saved record BW-9255/2018 Skipping already saved record BW-9256/2018 Skipping already saved record BW-9259/2018 Skipping already saved record BW-9260/2018 Scraping page 13... Skipping already saved record BW-9261/2018 Skipping already saved record BW-9263/2018 Skipping already saved record BW-9265/2018 Skipping already saved record BW-9266/2018 Skipping already saved record BW-9267/2018 Skipping already saved record BW-9268/2018 Saving record BW-9272/2018 - 6 Gloria Street CORNUBIA QLD 4130 Saving record BW-9273/2018 - 1 Prince Of Wales Court MUNDOOLUN QLD 4285 Saving record BW-9280/2018 - 13 Stephania Court CORNUBIA QLD 4130 Saving record BW-9281/2018 - 45 Turquoise Street BETHANIA QLD 4205 Scraping page 14... Saving record BW-9282/2018 - 2 Kadlunga Court BORONIA HEIGHTS QLD 4124 Saving record BW-9283/2018 - 40 Samford Drive HOLMVIEW QLD 4207 Saving record BW-9284/2018 - 312-314 Red Gum Road NEW BEITH QLD 4124 Saving record BW-9285/2018 - 37-39 Abbey Street FORESTDALE QLD 4118 Saving record BW-9286/2018 - 164-168 Panitz Drive JIMBOOMBA QLD 4280 Saving record BW-9289/2018 - 34 Stellaris Way ROCHEDALE SOUTH QLD 4123 Saving record BW-9290/2018 - 181-195 Bamsey Road STOCKLEIGH QLD 4280 Saving record BW-9291/2018 - 6 Zac Street MARSDEN QLD 4132 Saving record BW-9293/2018 - 171 Darlington Drive YARRABILBA QLD 4207 Saving record BW-9294/2018 - 20 Atkinson Street SLACKS CREEK QLD 4127 Scraping page 15... Saving record BW-9296/2018 - 36-40 Carbrook Road CORNUBIA QLD 4130 Saving record BW-9297/2018 - 19 Kumbari Street ROCHEDALE SOUTH QLD 4123 Saving record BW-9298/2018 - 9 Thora Street CRESTMEAD QLD 4132 Saving record BW-9299/2018 - 35 Smallwood Street UNDERWOOD QLD 4119 Saving record BW-9300/2018 - 97 Julie Street CRESTMEAD QLD 4132 Saving record BW-9301/2018 - 110 Grand Terrace WATERFORD QLD 4133 Saving record BW-9302/2018 - 29 Bushland Drive REGENTS PARK QLD 4118 Saving record BW-9304/2018 - 286A Dairy Creek Road WATERFORD QLD 4133 Saving record BW-9305/2018 - 117 Eildon Close MUNRUBEN QLD 4125 Saving record BW-9306/2018 - 54/11 Allora Street WATERFORD WEST QLD 4133 Scraping page 16... Skipping already saved record SP-243/2018 Skipping already saved record SP-244/2018 Skipping already saved record SP-245/2018 Skipping already saved record SP-246/2018 Skipping already saved record SP-21/2018 Skipping already saved record SP-119/2018 Saving record SP-247/2018 - 64-76 Pakenham Road GREENBANK QLD 4124 Saving record SP-248/2018 - 124 Logan Street EAGLEBY QLD 4207 Skipping already saved record OWVM-46/2018 Skipping already saved record LACA-37/2018 Scraping page 17... Skipping already saved record LACA-38/2018

Data

Downloaded 5276 times by slav123 openaustralia LoveMyData lewis-CWD jesse-a-reilly TheZepto rossrandell dsccramb spatialsteve dcarterlandsburys MikeRalphson peteclowes

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (11.6 MB) Use the API

rows 10 / 38477

info_url council_reference date_received description address date_scraped comment_url
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403056
MCUC-72/2014
2014-04-01
Light Industry
2/2-16 Beal Street MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404297
MCUC-74/2014
2014-04-03
Dual Occupancy
24 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405277
MCUC-76/2014
2014-04-07
Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)
1-11 Katherine Court LOGAN VILLAGE QLD 4207
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405578
MCUC-78/2014
2014-04-07
Warehouse
9-11 Calcium Court CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404871
RL-17/2014
2014-04-04
Reconfiguring a Lot (1 into 98 lots)
311-331 Logan Reserve Road LOGAN RESERVE QLD 4133
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403608
MCUC-73/2014
2014-04-02
Secondary Dwelling
80-90 Greensward Road TAMBORINE QLD 4270
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404307
MCUC-75/2014
2014-04-03
Dual Occupancy
13-15 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403117
MCUC-77/2014
2014-04-02
Place of Worship
7 University Drive MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403277
RL-16/2014
2014-04-02
Reconfiguring a Lot (1 into 2 lots and access easement)
11-21 Kingston Road UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403388
COM-1/2012
2014-04-02
Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road
1-13 Chase Close UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au

Statistics

Average successful run time: 6 minutes

Total run time: 8 days

Total cpu time used: about 1 hour

Total disk space used: 11.6 MB

History

  • Auto ran revision c87b02c3 and completed successfully .
    35 records added in the database
  • Auto ran revision c87b02c3 and completed successfully .
    35 records added in the database
  • Auto ran revision c87b02c3 and completed successfully .
    53 records added in the database
    11 pages scraped
  • Auto ran revision c87b02c3 and completed successfully .
    31 records added in the database
    6 pages scraped
  • Auto ran revision c87b02c3 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

logan / scraper.rb