planningalerts-scrapers / logan

Scrapes pdonline.logan.qld.gov.au

MasterView 2.0 Application Master


Logan City Council scraper

  • Server - .NET - ApplicationMaster
  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Pagnation - Yes
  • Javascript - Yes
  • UserAgent - No
  • Request actual DA page for data - No
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)

Enjoy

Contributors mlandauer LoveMyData henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0-p648 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Using bundler 1.15.2  Fetching unf_ext 0.0.7.3  Installing unf_ext 0.0.7.3 with native extensions  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.15s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1... Saving record RL-19/2019 - 28-32 Lawnton Street DAISY HILL QLD 4127 Skipping already saved record BWAP-103/2017 Skipping already saved record BWAP-105/2019 Skipping already saved record BWAP-107/2019 Skipping already saved record BWAP-108/2019 Skipping already saved record BWAP-109/2019 Saving record BWAP-110/2019 - 208-216 Peppertree Drive JIMBOOMBA QLD 4280 Skipping already saved record BWAP-583/2018 Skipping already saved record PD-625/2019 Skipping already saved record PD-626/2019 Scraping page 2... Skipping already saved record PD-624/2019 Skipping already saved record PD-627/2019 Skipping already saved record PD-628/2019 Skipping already saved record PD-629/2019 Skipping already saved record PD-630/2019 Skipping already saved record PD-632/2019 Skipping already saved record PD-634/2019 Skipping already saved record PD-641/2019 Skipping already saved record PD-631/2019 Skipping already saved record PD-633/2019 Scraping page 3... Skipping already saved record PD-635/2019 Skipping already saved record PD-636/2019 Skipping already saved record PD-637/2019 Skipping already saved record PD-638/2019 Skipping already saved record PD-639/2019 Skipping already saved record PD-640/2019 Skipping already saved record PD-643/2019 Saving record PD-644/2019 - 10 Acacia Loop GREENBANK QLD 4124 Saving record PD-645/2019 - 13 Linton Street LOGANLEA QLD 4131 Saving record PD-646/2019 - 45 Enclave Drive BAHRS SCRUB QLD 4207 Scraping page 4... Saving record PD-652/2019 - 10 Ralston Street LOGAN RESERVE QLD 4133 Saving record PD-647/2019 - 90 Welford Circuit YARRABILBA QLD 4207 Saving record PD-648/2019 - 14 Katrina Crescent WATERFORD WEST QLD 4133 Saving record PD-649/2019 - 10 Rochester Street HILLCREST QLD 4118 Saving record PD-650/2019 - 6 Jackson Place GREENBANK QLD 4124 Saving record PD-651/2019 - 12 Miamax Place LOGAN RESERVE QLD 4133 Saving record PD-653/2019 - 3 Barkala Avenue LOGANLEA QLD 4131 Saving record PD-654/2019 - 58-60 Alluring Court JIMBOOMBA QLD 4280 Saving record PD-655/2019 - 1 Hayter Street LOGAN RESERVE QLD 4133 Skipping already saved record BW-1899/2019 Scraping page 5... Skipping already saved record BW-1901/2019 Skipping already saved record BW-1902/2019 Skipping already saved record BW-1903/2019 Skipping already saved record BW-1904/2019 Skipping already saved record BW-1905/2019 Skipping already saved record BW-1906/2019 Skipping already saved record BW-1907/2019 Skipping already saved record BW-1908/2019 Skipping already saved record BW-1910/2019 Skipping already saved record BW-1911/2019 Scraping page 6... Skipping already saved record BW-1913/2019 Skipping already saved record BW-1915/2019 Skipping already saved record BW-1916/2019 Skipping already saved record BW-1917/2019 Skipping already saved record BW-1919/2019 Skipping already saved record BW-1920/2019 Skipping already saved record BW-1921/2019 Skipping already saved record BW-1923/2019 Skipping already saved record BW-1924/2019 Skipping already saved record BW-1925/2019 Scraping page 7... Skipping already saved record BW-1926/2019 Skipping already saved record BW-1927/2019 Skipping already saved record BW-1928/2019 Skipping already saved record BW-1930/2019 Skipping already saved record BW-1931/2019 Skipping already saved record BW-1932/2019 Skipping already saved record BW-1933/2019 Skipping already saved record BW-1935/2019 Skipping already saved record BW-1950/2019 Skipping already saved record BW-1951/2019 Scraping page 8... Skipping already saved record BW-1956/2019 Skipping already saved record BW-1963/2019 Skipping already saved record BW-1965/2019 Skipping already saved record BW-1968/2019 Skipping already saved record BW-1970/2019 Skipping already saved record BW-1971/2019 Skipping already saved record BW-1972/2019 Skipping already saved record BW-1973/2019 Skipping already saved record BW-1974/2019 Skipping already saved record BW-1975/2019 Scraping page 9... Skipping already saved record BW-1976/2019 Skipping already saved record BW-1977/2019 Skipping already saved record BW-1978/2019 Skipping already saved record BW-1979/2019 Skipping already saved record BW-1980/2019 Skipping already saved record BW-1981/2019 Skipping already saved record BW-1990/2019 Skipping already saved record BW-1992/2019 Skipping already saved record BW-1993/2019 Skipping already saved record BW-1994/2019 Scraping page 10... Skipping already saved record BW-1995/2019 Skipping already saved record BW-1996/2019 Skipping already saved record BW-1997/2019 Skipping already saved record BW-1998/2019 Skipping already saved record BW-1999/2019 Skipping already saved record BW-2000/2019 Skipping already saved record BW-2001/2019 Skipping already saved record BW-2004/2019 Skipping already saved record BW-2005/2019 Skipping already saved record BW-2006/2019 Scraping page 11... Skipping already saved record BW-2010/2019 Skipping already saved record BW-2011/2019 Skipping already saved record BW-2012/2019 Skipping already saved record BW-2013/2019 Skipping already saved record BW-2014/2019 Skipping already saved record BW-2015/2019 Skipping already saved record BW-2018/2019 Skipping already saved record BW-2019/2019 Saving record BW-2021/2019 - 8-18 Lagoon Road CARBROOK QLD 4130 Saving record BW-2022/2019 - 39 Jonas Circuit HOLMVIEW QLD 4207 Scraping page 12... Saving record BW-2023/2019 - 127-131 Smailes Road NORTH MACLEAN QLD 4280 Saving record BW-2024/2019 - 41A Norton Drive SHAILER PARK QLD 4128 Saving record BW-2025/2019 - 9 Samford Drive HOLMVIEW QLD 4207 Saving record BW-2026/2019 - 34 Almandin Street LOGAN RESERVE QLD 4133 Saving record BW-2028/2019 - 28 Samford Drive HOLMVIEW QLD 4207 Saving record BW-2030/2019 - 38 Almandin Street LOGAN RESERVE QLD 4133 Saving record BW-2033/2019 - 51 Sheridan Drive FLAGSTONE QLD 4280 Saving record BW-2045/2019 - 56 Rural Street PARK RIDGE QLD 4125 Saving record BW-2046/2019 - 58 Rural Street PARK RIDGE QLD 4125 Saving record BW-2047/2019 - 54 Almandin Street LOGAN RESERVE QLD 4133 Scraping page 13... Saving record BW-2053/2019 - 75-85 Andalusian Drive NORTH MACLEAN QLD 4280 Saving record BW-2055/2019 - 55-59 Crystal Brook Road NEW BEITH QLD 4124 Saving record BW-2057/2019 - 5 Aletta Street SHAILER PARK QLD 4128 Saving record BW-2059/2019 - 10-12 Barrine Court PARK RIDGE SOUTH QLD 4125 Saving record BW-2060/2019 - 9 Mortimer Drive FLAGSTONE QLD 4280 Saving record BW-2061/2019 - 11 Almandin Street LOGAN RESERVE QLD 4133 Saving record BW-2062/2019 - 13-15 Victor Street BORONIA HEIGHTS QLD 4124 Saving record BW-2063/2019 - 68 Colorado Street BAHRS SCRUB QLD 4207 Saving record BW-2064/2019 - 12 Winston Street ROCHEDALE SOUTH QLD 4123 Saving record BW-2065/2019 - 458-460 Teviot Road NORTH MACLEAN QLD 4280 Scraping page 14... Saving record BW-2066/2019 - TALLOWOOD PARK 35-55 Rholanda Crescent SPRINGWOOD QLD 4127 Saving record BW-2067/2019 - 121-125 Munroe Drive WOODHILL QLD 4285 Saving record BW-2069/2019 - 30 Simon Street UNDERWOOD QLD 4119 Saving record BW-2070/2019 - 10 Ralston Street LOGAN RESERVE QLD 4133 Saving record BW-2071/2019 - 33 Kerstin Court ROCHEDALE SOUTH QLD 4123 Skipping already saved record SP-44/2019 Skipping already saved record SP-45/2019 Saving record SP-46/2019 - 25-27 Tingi Avenue TANAH MERAH QLD 4128 Skipping already saved record MCUR-49/2019

Data

Downloaded 5855 times by slav123 openaustralia LoveMyData lewis-CWD jesse-a-reilly TheZepto rossrandell dsccramb spatialsteve dcarterlandsburys peteclowes MikeRalphson Jedwards157

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (12.4 MB) Use the API

rows 10 / 41233

info_url council_reference date_received description address date_scraped comment_url
MCUC-72/2014
2014-04-01
Light Industry
2/2-16 Beal Street MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-74/2014
2014-04-03
Dual Occupancy
24 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-76/2014
2014-04-07
Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)
1-11 Katherine Court LOGAN VILLAGE QLD 4207
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-78/2014
2014-04-07
Warehouse
9-11 Calcium Court CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
RL-17/2014
2014-04-04
Reconfiguring a Lot (1 into 98 lots)
311-331 Logan Reserve Road LOGAN RESERVE QLD 4133
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-73/2014
2014-04-02
Secondary Dwelling
80-90 Greensward Road TAMBORINE QLD 4270
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-75/2014
2014-04-03
Dual Occupancy
13-15 John Davison Place CRESTMEAD QLD 4132
2014-04-08
mailto:council@logan.qld.gov.au
MCUC-77/2014
2014-04-02
Place of Worship
7 University Drive MEADOWBROOK QLD 4131
2014-04-08
mailto:council@logan.qld.gov.au
RL-16/2014
2014-04-02
Reconfiguring a Lot (1 into 2 lots and access easement)
11-21 Kingston Road UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au
COM-1/2012
2014-04-02
Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road
1-13 Chase Close UNDERWOOD QLD 4119
2014-04-08
mailto:council@logan.qld.gov.au

Statistics

Average successful run time: 5 minutes

Total run time: 8 days

Total cpu time used: about 1 hour

Total disk space used: 12.4 MB

History

  • Auto ran revision c87b02c3 and completed successfully .
    42 records added in the database
    16 pages scraped
  • Auto ran revision c87b02c3 and completed successfully .
    26 records added in the database
    12 pages scraped
  • Auto ran revision c87b02c3 and completed successfully .
    39 records added in the database
    10 pages scraped
  • Auto ran revision c87b02c3 and completed successfully .
    30 records added in the database
    5 pages scraped
  • Auto ran revision c87b02c3 and failed .
    nothing changed in the database
    3 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

logan / scraper.rb