planningalerts-scrapers / logan

MasterView 2.0 Application Master

README.md

Logan City Council scraper

Server - .NET - ApplicationMaster
Accept Terms - Yes
Cookie tracking - Yes
Pagnation - Yes
Javascript - Yes
UserAgent - No
Request actual DA page for data - No
Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)

Enjoy

Contributors

Last run completed successfully 2018-07-19.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.0.0-p648 [1G-----> Installing dependencies using bundler 1.15.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Fetching gem metadata from https://rubygems.org/........ [1G Fetching version metadata from https://rubygems.org/. [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation. [1G Using bundler 1.15.2 [1G Fetching unf_ext 0.0.7.3 [1G Installing unf_ext 0.0.7.3 with native extensions [1G Fetching httpclient 2.8.3 [1G Installing httpclient 2.8.3 [1G Fetching mime-types-data 3.2016.0521 [1G Installing mime-types-data 3.2016.0521 [1G Fetching net-http-digest_auth 1.4.1 [1G Installing net-http-digest_auth 1.4.1 [1G Fetching net-http-persistent 2.9.4 [1G Installing net-http-persistent 2.9.4 [1G Fetching mini_portile2 2.1.0 [1G Installing mini_portile2 2.1.0 [1G Fetching ntlm-http 0.1.1 [1G Installing ntlm-http 0.1.1 [1G Fetching webrobots 0.1.2 [1G Installing webrobots 0.1.2 [1G Fetching sqlite3 1.3.13 [1G Installing sqlite3 1.3.13 with native extensions [1G Fetching unf 0.1.4 [1G Installing unf 0.1.4 [1G Fetching mime-types 3.1 [1G Installing mime-types 3.1 [1G Fetching nokogiri 1.6.8.1 [1G Installing nokogiri 1.6.8.1 with native extensions [1G Fetching sqlite_magic 0.0.6 [1G Installing sqlite_magic 0.0.6 [1G Fetching domain_name 0.5.20170404 [1G Installing domain_name 0.5.20170404 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching http-cookie 1.0.3 [1G Installing http-cookie 1.0.3 [1G Fetching mechanize 2.7.5 [1G Installing mechanize 2.7.5 [1G Bundle complete! 2 Gemfile dependencies, 18 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into ./vendor/bundle. [1G Bundle completed (29.81s) [1G Cleaning up the bundler cache. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1... Skipping already saved record RL-59/2018 Skipping already saved record COM-31/2017 Skipping already saved record OW-241/2018 Skipping already saved record OW-242/2018 Skipping already saved record SBR-8/2018 Skipping already saved record BWAP-313/2018 Skipping already saved record BWAP-314/2018 Skipping already saved record BWAP-315/2018 Skipping already saved record BWAP-316/2018 Skipping already saved record BWAP-317/2018 Scraping page 2... Skipping already saved record BWAP-319/2018 Skipping already saved record BWAP-320/2018 Skipping already saved record BWAP-321/2018 Skipping already saved record PD-2111/2018 Skipping already saved record PD-2111/2018 Skipping already saved record PD-2131/2018 Skipping already saved record PD-2139/2018 Skipping already saved record PD-2140/2018 Skipping already saved record PD-2154/2018 Skipping already saved record PD-2160/2018 Scraping page 3... Skipping already saved record PD-2162/2018 Skipping already saved record PD-2163/2018 Skipping already saved record PD-2164/2018 Skipping already saved record PD-2166/2018 Skipping already saved record PD-2167/2018 Skipping already saved record PD-2169/2018 Skipping already saved record PD-2175/2018 Skipping already saved record PD-2180/2018 Skipping already saved record PD-2185/2018 Saving record PD-2186/2018 - 76-80 Weatherly Drive JIMBOOMBA QLD 4280 Scraping page 4... Saving record PD-2187/2018 - 12 Potocki Street LOGAN RESERVE QLD 4133 Saving record PD-2188/2018 - 633-637 Ford Road PRIESTDALE QLD 4127 Saving record PD-2189/2018 - 16 Rosella Drive BAHRS SCRUB QLD 4207 Saving record PD-2190/2018 - 341 Waller Road PARK RIDGE QLD 4125 Saving record PD-2191/2018 - 17 Kourounis Street LOGAN RESERVE QLD 4133 Saving record PD-2192/2018 - 20 Salter Close FLAGSTONE QLD 4280 Saving record PD-2193/2018 - 86 Mount Huntley Street PARK RIDGE QLD 4125 Saving record PD-2195/2018 - 43 Gateway Drive FLAGSTONE QLD 4280 Skipping already saved record BW-5745/2018 Skipping already saved record BW-5746/2018 Scraping page 5... Skipping already saved record BW-5748/2018 Skipping already saved record BW-5751/2018 Skipping already saved record BW-5752/2018 Skipping already saved record BW-5753/2018 Skipping already saved record BW-5759/2018 Skipping already saved record BW-5762/2018 Skipping already saved record BW-5763/2018 Skipping already saved record BW-5767/2018 Skipping already saved record BW-5768/2018 Skipping already saved record BW-5772/2018 Scraping page 6... Skipping already saved record BW-5773/2018 Skipping already saved record BW-5774/2018 Skipping already saved record BW-5776/2018 Skipping already saved record BW-5778/2018 Skipping already saved record BW-5779/2018 Skipping already saved record BW-5780/2018 Skipping already saved record BW-5781/2018 Skipping already saved record BW-5782/2018 Skipping already saved record BW-5783/2018 Skipping already saved record BW-5785/2018 Scraping page 7... Skipping already saved record BW-5786/2018 Skipping already saved record BW-5787/2018 Skipping already saved record BW-5788/2018 Skipping already saved record BW-5789/2018 Skipping already saved record BW-5795/2018 Skipping already saved record BW-5796/2018 Skipping already saved record BW-5799/2018 Skipping already saved record BW-5800/2018 Skipping already saved record BW-5801/2018 Skipping already saved record BW-5802/2018 Scraping page 8... Skipping already saved record BW-5803/2018 Skipping already saved record BW-5811/2018 Skipping already saved record BW-5815/2018 Skipping already saved record BW-5816/2018 Skipping already saved record BW-5817/2018 Skipping already saved record BW-5821/2018 Skipping already saved record BW-5822/2018 Skipping already saved record BW-5823/2018 Skipping already saved record BW-5824/2018 Skipping already saved record BW-5826/2018 Scraping page 9... Skipping already saved record BW-5827/2018 Skipping already saved record BW-5828/2018 Skipping already saved record BW-5829/2018 Skipping already saved record BW-5830/2018 Skipping already saved record BW-5831/2018 Skipping already saved record BW-5832/2018 Skipping already saved record BW-5835/2018 Skipping already saved record BW-5839/2018 Skipping already saved record BW-5840/2018 Skipping already saved record BW-5842/2018 Scraping page 10... Skipping already saved record BW-5844/2018 Skipping already saved record BW-5845/2018 Skipping already saved record BW-5847/2018 Skipping already saved record BW-5849/2018 Skipping already saved record BW-5851/2018 Skipping already saved record BW-5853/2018 Skipping already saved record BW-5855/2018 Skipping already saved record BW-5856/2018 Skipping already saved record BW-5857/2018 Skipping already saved record BW-5859/2018 Scraping page 11... Skipping already saved record BW-5860/2018 Skipping already saved record BW-5861/2018 Skipping already saved record BW-5862/2018 Skipping already saved record BW-5863/2018 Skipping already saved record BW-5864/2018 Saving record BW-5867/2018 - 6 Fleming Street LOGAN RESERVE QLD 4133 Saving record BW-5868/2018 - 58 Slate Street YARRABILBA QLD 4207 Saving record BW-5869/2018 - 106-126 Stapylton Road FORESTDALE QLD 4118 Saving record BW-5870/2018 - 8 Welford Circuit YARRABILBA QLD 4207 Saving record BW-5872/2018 - 10 Welford Circuit YARRABILBA QLD 4207 Scraping page 12... Saving record BW-5873/2018 - 7 Maynor Court YARRABILBA QLD 4207 Saving record BW-5875/2018 - 3 Andrew Place MARSDEN QLD 4132 Saving record BW-5878/2018 - 6 Katelyn Street UNDERWOOD QLD 4119 Saving record BW-5881/2018 - 10 Kao Street MARSDEN QLD 4132 Saving record BW-5882/2018 - 105 Redgum Drive REGENTS PARK QLD 4118 Skipping already saved record SP-149/2018 Skipping already saved record SP-150/2018

Data

Downloaded 4666 times by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (10.6 MB) Use the API

rows 10 / 35370

info_url	council_reference	date_received	description	address	date_scraped	comment_url
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403056	MCUC-72/2014	2014-04-01	Light Industry	2/2-16 Beal Street MEADOWBROOK QLD 4131	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404297	MCUC-74/2014	2014-04-03	Dual Occupancy	24 John Davison Place CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405277	MCUC-76/2014	2014-04-07	Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)	1-11 Katherine Court LOGAN VILLAGE QLD 4207	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405578	MCUC-78/2014	2014-04-07	Warehouse	9-11 Calcium Court CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404871	RL-17/2014	2014-04-04	Reconfiguring a Lot (1 into 98 lots)	311-331 Logan Reserve Road LOGAN RESERVE QLD 4133	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403608	MCUC-73/2014	2014-04-02	Secondary Dwelling	80-90 Greensward Road TAMBORINE QLD 4270	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404307	MCUC-75/2014	2014-04-03	Dual Occupancy	13-15 John Davison Place CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403117	MCUC-77/2014	2014-04-02	Place of Worship	7 University Drive MEADOWBROOK QLD 4131	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403277	RL-16/2014	2014-04-02	Reconfiguring a Lot (1 into 2 lots and access easement)	11-21 Kingston Road UNDERWOOD QLD 4119	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403388	COM-1/2012	2014-04-02	Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road	1-13 Chase Close UNDERWOOD QLD 4119	2014-04-08	mailto:council@logan.qld.gov.au

Statistics

Average successful run time: 6 minutes

Total run time: 8 days

Total cpu time used: about 1 hour

Total disk space used: 10.7 MB

History

Auto ran revision c87b02c3 and completed successfully 2018-07-19.

run time 1 minute

19 records added in the database

14 pages scraped
Auto ran revision c87b02c3 and completed successfully 2018-07-18.

run time 1 minute

49 records added in the database

12 pages scraped
Auto ran revision c87b02c3 and completed successfully 2018-07-17.

run time 2 minutes

31 records added in the database

7 pages scraped
Auto ran revision c87b02c3 and completed successfully 2018-07-16.

run time 1 minute

17 records added in the database

4 pages scraped
Auto ran revision c87b02c3 and failed 2018-07-15.

run time 55 s

nothing changed in the database

3 pages scraped
...
Created on morph.io 2014-04-08

Show complete history

Scraper code

Ruby

logan / scraper.rb

git clone URL