planningalerts-scrapers / logan

MasterView 2.0 Application Master

README.md

Logan City Council scraper

Server - .NET - ApplicationMaster
Accept Terms - Yes
Cookie tracking - Yes
Pagnation - Yes
Javascript - Yes
UserAgent - No
Request actual DA page for data - No
Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)

Enjoy

Contributors

Last run completed successfully 2019-03-21.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.0.0-p648 [1G-----> Installing dependencies using bundler 1.15.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Fetching gem metadata from https://rubygems.org/........ [1G Fetching version metadata from https://rubygems.org/. [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation. [1G Using bundler 1.15.2 [1G Fetching unf_ext 0.0.7.3 [1G Installing unf_ext 0.0.7.3 with native extensions [1G Fetching httpclient 2.8.3 [1G Installing httpclient 2.8.3 [1G Fetching mime-types-data 3.2016.0521 [1G Installing mime-types-data 3.2016.0521 [1G Fetching net-http-digest_auth 1.4.1 [1G Installing net-http-digest_auth 1.4.1 [1G Fetching net-http-persistent 2.9.4 [1G Installing net-http-persistent 2.9.4 [1G Fetching mini_portile2 2.1.0 [1G Installing mini_portile2 2.1.0 [1G Fetching ntlm-http 0.1.1 [1G Installing ntlm-http 0.1.1 [1G Fetching webrobots 0.1.2 [1G Installing webrobots 0.1.2 [1G Fetching sqlite3 1.3.13 [1G Installing sqlite3 1.3.13 with native extensions [1G Fetching unf 0.1.4 [1G Installing unf 0.1.4 [1G Fetching mime-types 3.1 [1G Installing mime-types 3.1 [1G Fetching nokogiri 1.6.8.1 [1G Installing nokogiri 1.6.8.1 with native extensions [1G Fetching sqlite_magic 0.0.6 [1G Installing sqlite_magic 0.0.6 [1G Fetching domain_name 0.5.20170404 [1G Installing domain_name 0.5.20170404 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching http-cookie 1.0.3 [1G Installing http-cookie 1.0.3 [1G Fetching mechanize 2.7.5 [1G Installing mechanize 2.7.5 [1G Bundle complete! 2 Gemfile dependencies, 18 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into ./vendor/bundle. [1G Bundle completed (25.15s) [1G Cleaning up the bundler cache. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1... Saving record RL-19/2019 - 28-32 Lawnton Street DAISY HILL QLD 4127 Skipping already saved record BWAP-103/2017 Skipping already saved record BWAP-105/2019 Skipping already saved record BWAP-107/2019 Skipping already saved record BWAP-108/2019 Skipping already saved record BWAP-109/2019 Saving record BWAP-110/2019 - 208-216 Peppertree Drive JIMBOOMBA QLD 4280 Skipping already saved record BWAP-583/2018 Skipping already saved record PD-625/2019 Skipping already saved record PD-626/2019 Scraping page 2... Skipping already saved record PD-624/2019 Skipping already saved record PD-627/2019 Skipping already saved record PD-628/2019 Skipping already saved record PD-629/2019 Skipping already saved record PD-630/2019 Skipping already saved record PD-632/2019 Skipping already saved record PD-634/2019 Skipping already saved record PD-641/2019 Skipping already saved record PD-631/2019 Skipping already saved record PD-633/2019 Scraping page 3... Skipping already saved record PD-635/2019 Skipping already saved record PD-636/2019 Skipping already saved record PD-637/2019 Skipping already saved record PD-638/2019 Skipping already saved record PD-639/2019 Skipping already saved record PD-640/2019 Skipping already saved record PD-643/2019 Saving record PD-644/2019 - 10 Acacia Loop GREENBANK QLD 4124 Saving record PD-645/2019 - 13 Linton Street LOGANLEA QLD 4131 Saving record PD-646/2019 - 45 Enclave Drive BAHRS SCRUB QLD 4207 Scraping page 4... Saving record PD-652/2019 - 10 Ralston Street LOGAN RESERVE QLD 4133 Saving record PD-647/2019 - 90 Welford Circuit YARRABILBA QLD 4207 Saving record PD-648/2019 - 14 Katrina Crescent WATERFORD WEST QLD 4133 Saving record PD-649/2019 - 10 Rochester Street HILLCREST QLD 4118 Saving record PD-650/2019 - 6 Jackson Place GREENBANK QLD 4124 Saving record PD-651/2019 - 12 Miamax Place LOGAN RESERVE QLD 4133 Saving record PD-653/2019 - 3 Barkala Avenue LOGANLEA QLD 4131 Saving record PD-654/2019 - 58-60 Alluring Court JIMBOOMBA QLD 4280 Saving record PD-655/2019 - 1 Hayter Street LOGAN RESERVE QLD 4133 Skipping already saved record BW-1899/2019 Scraping page 5... Skipping already saved record BW-1901/2019 Skipping already saved record BW-1902/2019 Skipping already saved record BW-1903/2019 Skipping already saved record BW-1904/2019 Skipping already saved record BW-1905/2019 Skipping already saved record BW-1906/2019 Skipping already saved record BW-1907/2019 Skipping already saved record BW-1908/2019 Skipping already saved record BW-1910/2019 Skipping already saved record BW-1911/2019 Scraping page 6... Skipping already saved record BW-1913/2019 Skipping already saved record BW-1915/2019 Skipping already saved record BW-1916/2019 Skipping already saved record BW-1917/2019 Skipping already saved record BW-1919/2019 Skipping already saved record BW-1920/2019 Skipping already saved record BW-1921/2019 Skipping already saved record BW-1923/2019 Skipping already saved record BW-1924/2019 Skipping already saved record BW-1925/2019 Scraping page 7... Skipping already saved record BW-1926/2019 Skipping already saved record BW-1927/2019 Skipping already saved record BW-1928/2019 Skipping already saved record BW-1930/2019 Skipping already saved record BW-1931/2019 Skipping already saved record BW-1932/2019 Skipping already saved record BW-1933/2019 Skipping already saved record BW-1935/2019 Skipping already saved record BW-1950/2019 Skipping already saved record BW-1951/2019 Scraping page 8... Skipping already saved record BW-1956/2019 Skipping already saved record BW-1963/2019 Skipping already saved record BW-1965/2019 Skipping already saved record BW-1968/2019 Skipping already saved record BW-1970/2019 Skipping already saved record BW-1971/2019 Skipping already saved record BW-1972/2019 Skipping already saved record BW-1973/2019 Skipping already saved record BW-1974/2019 Skipping already saved record BW-1975/2019 Scraping page 9... Skipping already saved record BW-1976/2019 Skipping already saved record BW-1977/2019 Skipping already saved record BW-1978/2019 Skipping already saved record BW-1979/2019 Skipping already saved record BW-1980/2019 Skipping already saved record BW-1981/2019 Skipping already saved record BW-1990/2019 Skipping already saved record BW-1992/2019 Skipping already saved record BW-1993/2019 Skipping already saved record BW-1994/2019 Scraping page 10... Skipping already saved record BW-1995/2019 Skipping already saved record BW-1996/2019 Skipping already saved record BW-1997/2019 Skipping already saved record BW-1998/2019 Skipping already saved record BW-1999/2019 Skipping already saved record BW-2000/2019 Skipping already saved record BW-2001/2019 Skipping already saved record BW-2004/2019 Skipping already saved record BW-2005/2019 Skipping already saved record BW-2006/2019 Scraping page 11... Skipping already saved record BW-2010/2019 Skipping already saved record BW-2011/2019 Skipping already saved record BW-2012/2019 Skipping already saved record BW-2013/2019 Skipping already saved record BW-2014/2019 Skipping already saved record BW-2015/2019 Skipping already saved record BW-2018/2019 Skipping already saved record BW-2019/2019 Saving record BW-2021/2019 - 8-18 Lagoon Road CARBROOK QLD 4130 Saving record BW-2022/2019 - 39 Jonas Circuit HOLMVIEW QLD 4207 Scraping page 12... Saving record BW-2023/2019 - 127-131 Smailes Road NORTH MACLEAN QLD 4280 Saving record BW-2024/2019 - 41A Norton Drive SHAILER PARK QLD 4128 Saving record BW-2025/2019 - 9 Samford Drive HOLMVIEW QLD 4207 Saving record BW-2026/2019 - 34 Almandin Street LOGAN RESERVE QLD 4133 Saving record BW-2028/2019 - 28 Samford Drive HOLMVIEW QLD 4207 Saving record BW-2030/2019 - 38 Almandin Street LOGAN RESERVE QLD 4133 Saving record BW-2033/2019 - 51 Sheridan Drive FLAGSTONE QLD 4280 Saving record BW-2045/2019 - 56 Rural Street PARK RIDGE QLD 4125 Saving record BW-2046/2019 - 58 Rural Street PARK RIDGE QLD 4125 Saving record BW-2047/2019 - 54 Almandin Street LOGAN RESERVE QLD 4133 Scraping page 13... Saving record BW-2053/2019 - 75-85 Andalusian Drive NORTH MACLEAN QLD 4280 Saving record BW-2055/2019 - 55-59 Crystal Brook Road NEW BEITH QLD 4124 Saving record BW-2057/2019 - 5 Aletta Street SHAILER PARK QLD 4128 Saving record BW-2059/2019 - 10-12 Barrine Court PARK RIDGE SOUTH QLD 4125 Saving record BW-2060/2019 - 9 Mortimer Drive FLAGSTONE QLD 4280 Saving record BW-2061/2019 - 11 Almandin Street LOGAN RESERVE QLD 4133 Saving record BW-2062/2019 - 13-15 Victor Street BORONIA HEIGHTS QLD 4124 Saving record BW-2063/2019 - 68 Colorado Street BAHRS SCRUB QLD 4207 Saving record BW-2064/2019 - 12 Winston Street ROCHEDALE SOUTH QLD 4123 Saving record BW-2065/2019 - 458-460 Teviot Road NORTH MACLEAN QLD 4280 Scraping page 14... Saving record BW-2066/2019 - TALLOWOOD PARK 35-55 Rholanda Crescent SPRINGWOOD QLD 4127 Saving record BW-2067/2019 - 121-125 Munroe Drive WOODHILL QLD 4285 Saving record BW-2069/2019 - 30 Simon Street UNDERWOOD QLD 4119 Saving record BW-2070/2019 - 10 Ralston Street LOGAN RESERVE QLD 4133 Saving record BW-2071/2019 - 33 Kerstin Court ROCHEDALE SOUTH QLD 4123 Skipping already saved record SP-44/2019 Skipping already saved record SP-45/2019 Saving record SP-46/2019 - 25-27 Tingi Avenue TANAH MERAH QLD 4128 Skipping already saved record MCUR-49/2019

Data

Downloaded 5855 times by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (12.4 MB) Use the API

rows 10 / 41233

info_url	council_reference	date_received	description	address	date_scraped	comment_url
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403056	MCUC-72/2014	2014-04-01	Light Industry	2/2-16 Beal Street MEADOWBROOK QLD 4131	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404297	MCUC-74/2014	2014-04-03	Dual Occupancy	24 John Davison Place CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405277	MCUC-76/2014	2014-04-07	Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)	1-11 Katherine Court LOGAN VILLAGE QLD 4207	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405578	MCUC-78/2014	2014-04-07	Warehouse	9-11 Calcium Court CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404871	RL-17/2014	2014-04-04	Reconfiguring a Lot (1 into 98 lots)	311-331 Logan Reserve Road LOGAN RESERVE QLD 4133	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403608	MCUC-73/2014	2014-04-02	Secondary Dwelling	80-90 Greensward Road TAMBORINE QLD 4270	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404307	MCUC-75/2014	2014-04-03	Dual Occupancy	13-15 John Davison Place CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403117	MCUC-77/2014	2014-04-02	Place of Worship	7 University Drive MEADOWBROOK QLD 4131	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403277	RL-16/2014	2014-04-02	Reconfiguring a Lot (1 into 2 lots and access easement)	11-21 Kingston Road UNDERWOOD QLD 4119	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403388	COM-1/2012	2014-04-02	Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road	1-13 Chase Close UNDERWOOD QLD 4119	2014-04-08	mailto:council@logan.qld.gov.au

Statistics

Average successful run time: 5 minutes

Total run time: 8 days

Total cpu time used: about 1 hour

Total disk space used: 12.4 MB

History

Auto ran revision c87b02c3 and completed successfully 2019-03-21.

run time 1 minute

42 records added in the database

16 pages scraped
Auto ran revision c87b02c3 and completed successfully 2019-03-20.

run time 1 minute

26 records added in the database

12 pages scraped
Auto ran revision c87b02c3 and completed successfully 2019-03-19.

run time 19 s

39 records added in the database

10 pages scraped
Auto ran revision c87b02c3 and completed successfully 2019-03-18.

run time 1 minute

30 records added in the database

5 pages scraped
Auto ran revision c87b02c3 and failed 2019-03-17.

run time 1 minute

nothing changed in the database

3 pages scraped
...
Created on morph.io 2014-04-08

Show complete history

Scraper code

Ruby

logan / scraper.rb

git clone URL