planningalerts-scrapers / logan

Scrapes pdonline.logan.qld.gov.au

MasterView 2.0 Application Master

README.md

Logan City Council scraper

  • Server - .NET - ApplicationMaster
  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Pagnation - Yes
  • Javascript - Yes
  • UserAgent - No
  • Request actual DA page for data - No
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)

Enjoy

Contributors mlandauer LoveMyData henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1... Saving record MCUC-6/2019 - 140 Meakin Road KINGSTON QLD 4114 Saving record BWAP-14/2019 - 20-24 Whipbird Place GREENBANK QLD 4124 Saving record BWAP-15/2019 - 43 Forestglen Crescent BROWNS PLAINS QLD 4118 Saving record PD-115/2019 - 1 Deloraine Drive SPRINGWOOD QLD 4127 Saving record PD-118/2019 - 11 Turquoise Street BETHANIA QLD 4205 Saving record PD-119/2019 - 37-41 Blyth Court FORESTDALE QLD 4118 Saving record PD-121/2019 - 25 Irma Circuit PARK RIDGE QLD 4125 Saving record BW-338/2019 - 31-35 Barcoo Drive NEW BEITH QLD 4124 Saving record BW-339/2019 - 22 Windsor Street FLAGSTONE QLD 4280 Saving record BW-340/2019 - 49 Welford Circuit YARRABILBA QLD 4207 Scraping page 2... Saving record BW-341/2019 - 2 Welford Circuit YARRABILBA QLD 4207 Saving record BW-342/2019 - 36 Niels Crescent PARK RIDGE QLD 4125 Saving record BW-344/2019 - 14 Village Court LOGAN VILLAGE QLD 4207 Saving record BW-346/2019 - 11-19 Pakenham Road GREENBANK QLD 4124 Saving record BW-347/2019 - 3517-3521 Pacific Highway SLACKS CREEK QLD 4127 Saving record BW-348/2019 - 1 Glendale Street MARSDEN QLD 4132 Saving record BW-350/2019 - 1 Meranti Street CRESTMEAD QLD 4132 Saving record BW-352/2019 - 17 Brosnahan Court BELIVAH QLD 4207 Saving record BW-353/2019 - 2371-2379 Waterford Tamborine Road TAMBORINE QLD 4270 Saving record BW-354/2019 - 27A Evergreen Avenue LOGANLEA QLD 4131 Scraping page 3... Saving record BW-355/2019 - 7 Copper Drive BETHANIA QLD 4205 Saving record BW-356/2019 - EAGLEBY STATE SCHOOL 222-260 Fryar Road EAGLEBY QLD 4207 Saving record BW-358/2019 - 80-82 Melrose Place NEW BEITH QLD 4124 Saving record BW-359/2019 - 2 Arif Place HERITAGE PARK QLD 4118 Saving record BW-360/2019 - 25-31 McCubbin Court NEW BEITH QLD 4124 Saving record BW-361/2019 - 34 Anne Street PARK RIDGE QLD 4125 Saving record BW-362/2019 - 70 Falco Drive BAHRS SCRUB QLD 4207 Saving record BW-363/2019 - 68 Falco Drive BAHRS SCRUB QLD 4207 Saving record SP-8/2019 - 24 Ranchwood Avenue BROWNS PLAINS QLD 4118

Data

Downloaded 5564 times by slav123 openaustralia LoveMyData lewis-CWD jesse-a-reilly TheZepto rossrandell dsccramb spatialsteve dcarterlandsburys peteclowes MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (11.9 MB) Use the API

rows 10 / 39727

info_url council_reference date_received description address date_scraped comment_url
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403056
MCUC-72/2014
2014-04-01
Light Industry
2/2-16 Beal Street MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404297
MCUC-74/2014
2014-04-03
Dual Occupancy
24 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405277
MCUC-76/2014
2014-04-07
Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)
1-11 Katherine Court LOGAN VILLAGE QLD 4207
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405578
MCUC-78/2014
2014-04-07
Warehouse
9-11 Calcium Court CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404871
RL-17/2014
2014-04-04
Reconfiguring a Lot (1 into 98 lots)
311-331 Logan Reserve Road LOGAN RESERVE QLD 4133
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403608
MCUC-73/2014
2014-04-02
Secondary Dwelling
80-90 Greensward Road TAMBORINE QLD 4270
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404307
MCUC-75/2014
2014-04-03
Dual Occupancy
13-15 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403117
MCUC-77/2014
2014-04-02
Place of Worship
7 University Drive MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403277
RL-16/2014
2014-04-02
Reconfiguring a Lot (1 into 2 lots and access easement)
11-21 Kingston Road UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403388
COM-1/2012
2014-04-02
Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road
1-13 Chase Close UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au

Statistics

Average successful run time: 5 minutes

Total run time: 8 days

Total cpu time used: about 1 hour

Total disk space used: 12 MB

History

  • Auto ran revision c87b02c3 and completed successfully .
    29 records added in the database
    5 pages scraped
  • Auto ran revision c87b02c3 and failed .
    nothing changed in the database
    3 pages scraped
  • Auto ran revision c87b02c3 and failed .
    nothing changed in the database
  • Auto ran revision c87b02c3 and completed successfully .
    38 records added in the database
    24 pages scraped
  • Auto ran revision c87b02c3 and completed successfully .
    43 records added in the database
    20 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

logan / scraper.rb