planningalerts-scrapers / logan

MasterView 2.0 Application Master

README.md

Logan City Council scraper

Server - .NET - ApplicationMaster
Accept Terms - Yes
Cookie tracking - Yes
Pagnation - Yes
Javascript - Yes
UserAgent - No
Request actual DA page for data - No
Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth * custom = yyyy-mm (eg. 2017-01)

Enjoy

Contributors

Last run completed successfully 2018-09-21.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.0.0-p648 [1G-----> Installing dependencies using bundler 1.15.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Fetching gem metadata from https://rubygems.org/........ [1G Fetching version metadata from https://rubygems.org/. [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation. [1G Using bundler 1.15.2 [1G Fetching unf_ext 0.0.7.3 [1G Installing unf_ext 0.0.7.3 with native extensions [1G Fetching httpclient 2.8.3 [1G Installing httpclient 2.8.3 [1G Fetching mime-types-data 3.2016.0521 [1G Installing mime-types-data 3.2016.0521 [1G Fetching net-http-digest_auth 1.4.1 [1G Installing net-http-digest_auth 1.4.1 [1G Fetching net-http-persistent 2.9.4 [1G Installing net-http-persistent 2.9.4 [1G Fetching mini_portile2 2.1.0 [1G Installing mini_portile2 2.1.0 [1G Fetching ntlm-http 0.1.1 [1G Installing ntlm-http 0.1.1 [1G Fetching webrobots 0.1.2 [1G Installing webrobots 0.1.2 [1G Fetching sqlite3 1.3.13 [1G Installing sqlite3 1.3.13 with native extensions [1G Fetching unf 0.1.4 [1G Installing unf 0.1.4 [1G Fetching mime-types 3.1 [1G Installing mime-types 3.1 [1G Fetching nokogiri 1.6.8.1 [1G Installing nokogiri 1.6.8.1 with native extensions [1G Fetching sqlite_magic 0.0.6 [1G Installing sqlite_magic 0.0.6 [1G Fetching domain_name 0.5.20170404 [1G Installing domain_name 0.5.20170404 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching http-cookie 1.0.3 [1G Installing http-cookie 1.0.3 [1G Fetching mechanize 2.7.5 [1G Installing mechanize 2.7.5 [1G Bundle complete! 2 Gemfile dependencies, 18 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into ./vendor/bundle. [1G Bundle completed (28.52s) [1G Cleaning up the bundler cache. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Getting data in `thisweek`, changable via MORPH_PERIOD environment Scraping page 1... Saving record RL-126/2017 - 53-55 Fedrick Street BORONIA HEIGHTS QLD 4124 Skipping already saved record COM-34/2017 Skipping already saved record MCUI-50/2016 Skipping already saved record OW-317/2018 Skipping already saved record OW-318/2018 Skipping already saved record OWADV-37/2018 Skipping already saved record BWAP-420/2018 Skipping already saved record BWAP-421/2018 Skipping already saved record BWAP-422/2018 Skipping already saved record BWAP-423/2018 Scraping page 2... Skipping already saved record BWAP-425/2018 Skipping already saved record BWAP-426/2018 Skipping already saved record BWAP-427/2018 Skipping already saved record BWAP-428/2018 Skipping already saved record BWAP-429/2018 Skipping already saved record BWAP-430/2018 Skipping already saved record BWAP-431/2018 Skipping already saved record BWAP-432/2018 Skipping already saved record BWAP-433/2018 Skipping already saved record BWAP-434/2018 Scraping page 3... Skipping already saved record BWAP-435/2018 Saving record BWAP-436/2018 - 185 Candlebark Road GLENLOGAN QLD 4280 Saving record BWAP-437/2018 - 73 Regency Drive REGENTS PARK QLD 4118 Skipping already saved record PD-2757/2018 Skipping already saved record PD-2759/2018 Skipping already saved record PD-2761/2018 Skipping already saved record PD-2762/2018 Skipping already saved record PD-2762/2018 Skipping already saved record PD-2763/2018 Skipping already saved record PD-2765/2018 Scraping page 4... Skipping already saved record PD-2766/2018 Saving record PD-2768/2018 - 111-117 Bidyan Boulevard NEW BEITH QLD 4124 Skipping already saved record PD-2769/2018 Skipping already saved record PD-2770/2018 Skipping already saved record PD-2771/2018 Skipping already saved record PD-2772/2018 Skipping already saved record PD-2773/2018 Skipping already saved record PD-2774/2018 Skipping already saved record PD-2775/2018 Skipping already saved record PD-2776/2018 Scraping page 5... Skipping already saved record PD-2777/2018 Skipping already saved record PD-2778/2018 Skipping already saved record PD-2779/2018 Skipping already saved record PD-2780/2018 Skipping already saved record PD-2781/2018 Skipping already saved record PD-2782/2018 Skipping already saved record PD-2783/2018 Skipping already saved record PD-2784/2018 Skipping already saved record PD-2785/2018 Skipping already saved record PD-2786/2018 Scraping page 6... Skipping already saved record PD-2787/2018 Skipping already saved record PD-2788/2018 Skipping already saved record PD-2789/2018 Skipping already saved record PD-2790/2018 Skipping already saved record PD-2791/2018 Skipping already saved record PD-2792/2018 Skipping already saved record PD-2793/2018 Skipping already saved record PD-2794/2018 Skipping already saved record PD-2795/2018 Skipping already saved record PD-2796/2018 Scraping page 7... Skipping already saved record PD-2797/2018 Skipping already saved record PD-2798/2018 Skipping already saved record PD-2805/2018 Skipping already saved record PD-2808/2018 Skipping already saved record PD-2810/2018 Skipping already saved record PD-2812/2018 Skipping already saved record PD-2814/2018 Skipping already saved record PD-2817/2018 Skipping already saved record PD-2819/2018 Skipping already saved record PD-2799/2018 Scraping page 8... Skipping already saved record PD-2801/2018 Skipping already saved record PD-2802/2018 Skipping already saved record PD-2804/2018 Skipping already saved record PD-2806/2018 Skipping already saved record PD-2807/2018 Skipping already saved record PD-2809/2018 Skipping already saved record PD-2811/2018 Skipping already saved record PD-2813/2018 Skipping already saved record PD-2815/2018 Skipping already saved record PD-2816/2018 Scraping page 9... Skipping already saved record PD-2818/2018 Skipping already saved record PD-2820/2018 Skipping already saved record PD-2821/2018 Skipping already saved record PD-2822/2018 Skipping already saved record PD-2823/2018 Skipping already saved record PD-2824/2018 Skipping already saved record PD-2825/2018 Skipping already saved record PD-2826/2018 Skipping already saved record PD-2827/2018 Skipping already saved record PD-2828/2018 Scraping page 10... Skipping already saved record PD-2829/2018 Skipping already saved record PD-2830/2018 Skipping already saved record PD-2831/2018 Skipping already saved record PD-2832/2018 Skipping already saved record PD-2833/2018 Skipping already saved record PD-2835/2018 Skipping already saved record PD-2837/2018 Skipping already saved record PD-2838/2018 Skipping already saved record PD-2834/2018 Skipping already saved record PD-2836/2018 Scraping page 11... Skipping already saved record PD-2839/2018 Skipping already saved record PD-2840/2018 Skipping already saved record PD-2841/2018 Skipping already saved record PD-2842/2018 Skipping already saved record PD-2843/2018 Skipping already saved record PD-2844/2018 Skipping already saved record PD-2845/2018 Skipping already saved record PD-2846/2018 Skipping already saved record PD-2847/2018 Saving record PD-2849/2018 - 32 Kuprin Street LOGAN RESERVE QLD 4133 Scraping page 12... Saving record PD-2851/2018 - 26 Highgate Drive FLAGSTONE QLD 4280 Saving record PD-2854/2018 - 23 Roseland Avenue ROCHEDALE SOUTH QLD 4123 Saving record PD-2857/2018 - 11 Windsor Street FLAGSTONE QLD 4280 Saving record PD-2860/2018 - 43 Welford Circuit YARRABILBA QLD 4207 Saving record PD-2848/2018 - 272 Camp Cable Road JIMBOOMBA QLD 4280 Saving record PD-2850/2018 - 2-40 Koplick Road CHAMBERS FLAT QLD 4133 Saving record PD-2852/2018 - 4 Elsie Way LOGAN RESERVE QLD 4133 Saving record PD-2853/2018 - 2-40 Koplick Road CHAMBERS FLAT QLD 4133 Saving record PD-2855/2018 - 5 Anne Street PARK RIDGE QLD 4125 Saving record PD-2856/2018 - 2-40 Koplick Road CHAMBERS FLAT QLD 4133 Scraping page 13... Saving record PD-2859/2018 - 1-15 Crayfish Place NEW BEITH QLD 4124 Saving record PD-2861/2018 - 21 Wentworth Drive FLAGSTONE QLD 4280 Saving record PD-2862/2018 - 15 Regent Court BAHRS SCRUB QLD 4207 Saving record PD-2863/2018 - 20 Highgate Drive FLAGSTONE QLD 4280 Saving record PD-2864/2018 - 41 Golden Gate Avenue PARK RIDGE QLD 4125 Saving record PD-2865/2018 - 14-16 Barris Court FORESTDALE QLD 4118 Skipping already saved record BW-7459/2018 Skipping already saved record BW-7460/2018 Skipping already saved record BW-7461/2018 Skipping already saved record BW-7462/2018 Scraping page 14... Skipping already saved record BW-7464/2018 Skipping already saved record BW-7465/2018 Skipping already saved record BW-7466/2018 Skipping already saved record BW-7467/2018 Skipping already saved record BW-7468/2018 Skipping already saved record BW-7470/2018 Skipping already saved record BW-7472/2018 Skipping already saved record BW-7473/2018 Skipping already saved record BW-7474/2018 Skipping already saved record BW-7475/2018 Scraping page 15... Skipping already saved record BW-7476/2018 Skipping already saved record BW-7477/2018 Skipping already saved record BW-7478/2018 Skipping already saved record BW-7479/2018 Skipping already saved record BW-7480/2018 Skipping already saved record BW-7481/2018 Skipping already saved record BW-7482/2018 Skipping already saved record BW-7483/2018 Skipping already saved record BW-7484/2018 Skipping already saved record BW-7486/2018 Scraping page 16... Skipping already saved record BW-7487/2018 Skipping already saved record BW-7488/2018 Skipping already saved record BW-7489/2018 Skipping already saved record BW-7491/2018 Skipping already saved record BW-7493/2018 Skipping already saved record BW-7496/2018 Skipping already saved record BW-7497/2018 Skipping already saved record BW-7501/2018 Skipping already saved record BW-7504/2018 Skipping already saved record BW-7505/2018 Scraping page 17... Skipping already saved record BW-7512/2018 Skipping already saved record BW-7514/2018 Skipping already saved record BW-7515/2018 Skipping already saved record BW-7518/2018 Skipping already saved record BW-7520/2018 Skipping already saved record BW-7521/2018 Skipping already saved record BW-7523/2018 Skipping already saved record BW-7524/2018 Skipping already saved record BW-7525/2018 Skipping already saved record BW-7526/2018 Scraping page 18... Skipping already saved record BW-7528/2018 Skipping already saved record BW-7529/2018 Skipping already saved record BW-7531/2018 Skipping already saved record BW-7532/2018 Skipping already saved record BW-7535/2018 Skipping already saved record BW-7537/2018 Skipping already saved record BW-7540/2018 Skipping already saved record BW-7541/2018 Skipping already saved record BW-7542/2018 Skipping already saved record BW-7543/2018 Scraping page 19... Skipping already saved record BW-7544/2018 Skipping already saved record BW-7545/2018 Skipping already saved record BW-7546/2018 Skipping already saved record BW-7549/2018 Skipping already saved record BW-7550/2018 Skipping already saved record BW-7551/2018 Skipping already saved record BW-7552/2018 Skipping already saved record BW-7553/2018 Skipping already saved record BW-7555/2018 Skipping already saved record BW-7557/2018 Scraping page 20... Skipping already saved record BW-7558/2018 Skipping already saved record BW-7559/2018 Skipping already saved record BW-7560/2018 Skipping already saved record BW-7561/2018 Skipping already saved record BW-7562/2018 Skipping already saved record BW-7563/2018 Skipping already saved record BW-7564/2018 Skipping already saved record BW-7565/2018 Skipping already saved record BW-7566/2018 Skipping already saved record BW-7567/2018 Scraping page 21... Skipping already saved record BW-7568/2018 Skipping already saved record BW-7569/2018 Skipping already saved record BW-7570/2018 Skipping already saved record BW-7571/2018 Skipping already saved record BW-7572/2018 Skipping already saved record BW-7573/2018 Skipping already saved record BW-7574/2018 Skipping already saved record BW-7575/2018 Skipping already saved record BW-7576/2018 Skipping already saved record BW-7577/2018 Scraping page 22... Skipping already saved record BW-7578/2018 Skipping already saved record BW-7579/2018 Skipping already saved record BW-7580/2018 Skipping already saved record BW-7581/2018 Skipping already saved record BW-7582/2018 Skipping already saved record BW-7583/2018 Skipping already saved record BW-7584/2018 Skipping already saved record BW-7591/2018 Skipping already saved record BW-7597/2018 Skipping already saved record BW-7599/2018 Scraping page 23... Skipping already saved record BW-7602/2018 Skipping already saved record BW-7604/2018 Skipping already saved record BW-7613/2018 Saving record BW-7615/2018 - 59-67 Candlebark Road GLENLOGAN QLD 4280 Saving record BW-7616/2018 - 42 Haven Drive HOLMVIEW QLD 4207 Saving record BW-7618/2018 - 36-44 Wirrabara Drive GREENBANK QLD 4124 Saving record BW-7619/2018 - 123-129 Wynne Road JIMBOOMBA QLD 4280 Saving record BW-7620/2018 - 180 Beaudesert Beenleigh Road BAHRS SCRUB QLD 4207 Saving record BW-7621/2018 - 62-66 Doyle Road SOUTH MACLEAN QLD 4280 Saving record BW-7623/2018 - 2-4 Momalong Court BUCCAN QLD 4207 Scraping page 24... Saving record BW-7624/2018 - 138-140 Panitz Drive JIMBOOMBA QLD 4280 Saving record BW-7625/2018 - 44 Belyando Street HOLMVIEW QLD 4207 Saving record BW-7626/2018 - 13 Southwood Circuit YARRABILBA QLD 4207 Saving record BW-7627/2018 - 3 Winkler Court YARRABILBA QLD 4207 Saving record BW-7628/2018 - 5 Jendi Court SPRINGWOOD QLD 4127 Saving record BW-7630/2018 - 12-14 Barokee Drive TANAH MERAH QLD 4128 Saving record BW-7631/2018 - 43-45 Pepperina Drive STOCKLEIGH QLD 4280 Saving record BW-7632/2018 - 25/128-136 Murrays Road TANAH MERAH QLD 4128 Saving record BW-7633/2018 - 42A Whitaker Road CEDAR GROVE QLD 4285 Saving record BW-7634/2018 - 31 Belivah Road BELIVAH QLD 4207 Scraping page 25... Saving record BW-7635/2018 - 4 Willand Drive BEENLEIGH QLD 4207 Saving record BW-7636/2018 - 107-115 Garden Road CEDAR CREEK QLD 4207 Saving record BW-7637/2018 - 3732-3744 Mount Lindesay Highway PARK RIDGE QLD 4125 Saving record BW-7639/2018 - 12-24 Frog Hollow Court LOGAN VILLAGE QLD 4207 Saving record BW-7640/2018 - LOGAN PROFESSIONAL CENTRE 13-21 Mayes Avenue LOGAN CENTRAL QLD 4114 Saving record BW-7642/2018 - SHAILER PARK PRIMARY & PRE-SCHOOL 9-19 Bulwarna Street SHAILER PARK QLD 4128 Saving record BW-7643/2018 - 50 Bottlebrush Drive REGENTS PARK QLD 4118 Saving record BW-7644/2018 - 4 Isla Street PARK RIDGE QLD 4125 Saving record BW-7645/2018 - 128-132 Bidyan Boulevard NEW BEITH QLD 4124 Saving record BW-7647/2018 - 2-12 Springvale Circuit UNDERWOOD QLD 4119 Scraping page 26... Saving record BW-7649/2018 - 2A Glenvale Street CORNUBIA QLD 4130 Skipping already saved record SP-202/2018 Skipping already saved record SP-203/2018 Skipping already saved record SP-204/2018 Skipping already saved record OWVM-41/2018 Skipping already saved record LACA-30/2018 Skipping already saved record LACA-26/2017

Data

Downloaded 4975 times by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (11.1 MB) Use the API

rows 10 / 37007

info_url	council_reference	date_received	description	address	date_scraped	comment_url
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403056	MCUC-72/2014	2014-04-01	Light Industry	2/2-16 Beal Street MEADOWBROOK QLD 4131	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404297	MCUC-74/2014	2014-04-03	Dual Occupancy	24 John Davison Place CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405277	MCUC-76/2014	2014-04-07	Secondary Dwelling (Overlays & Reduced Road Boundary Clearance)	1-11 Katherine Court LOGAN VILLAGE QLD 4207	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1405578	MCUC-78/2014	2014-04-07	Warehouse	9-11 Calcium Court CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404871	RL-17/2014	2014-04-04	Reconfiguring a Lot (1 into 98 lots)	311-331 Logan Reserve Road LOGAN RESERVE QLD 4133	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403608	MCUC-73/2014	2014-04-02	Secondary Dwelling	80-90 Greensward Road TAMBORINE QLD 4270	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1404307	MCUC-75/2014	2014-04-03	Dual Occupancy	13-15 John Davison Place CRESTMEAD QLD 4132	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403117	MCUC-77/2014	2014-04-02	Place of Worship	7 University Drive MEADOWBROOK QLD 4131	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403277	RL-16/2014	2014-04-02	Reconfiguring a Lot (1 into 2 lots and access easement)	11-21 Kingston Road UNDERWOOD QLD 4119	2014-04-08	mailto:council@logan.qld.gov.au
http://pdonline.logan.qld.gov.au/MasterViewUI/Modules/ApplicationMaster/default.aspx?page=wrapper&key=1403388	COM-1/2012	2014-04-02	Request to Change - Multi-Unit Development (40 Units) and Reconfiguration of a Lot (1 lot into 4 lots) Plus Drainage Reserve and Road	1-13 Chase Close UNDERWOOD QLD 4119	2014-04-08	mailto:council@logan.qld.gov.au

Statistics

Average successful run time: 6 minutes

Total run time: 8 days

Total cpu time used: about 1 hour

Total disk space used: 11.1 MB

History

Auto ran revision c87b02c3 and completed successfully 2018-09-21.

run time 2 minutes

49 records added in the database

28 pages scraped
Auto ran revision c87b02c3 and completed successfully 2018-09-20.

run time 2 minutes

28 records added in the database

23 pages scraped
Auto ran revision c87b02c3 and completed successfully 2018-09-19.

run time 2 minutes

85 records added in the database

20 pages scraped
Auto ran revision c87b02c3 and completed successfully 2018-09-18.

run time 1 minute

57 records added in the database
Auto ran revision c87b02c3 and completed successfully 2018-09-17.

run time 1 minute

36 records added in the database

6 pages scraped
...
Created on morph.io 2014-04-08

Show complete history

Scraper code

Ruby

logan / scraper.rb

git clone URL