planningalerts-scrapers / burwood_development_applications

City of Burwood Development Applications

Scrapes ecouncil.burwood.nsw.gov.au

README.md

Burwood Council scraper

Accept Terms - No
Cookie tracking - Yes
Javascript - Yes
Request actual DA page for data - Yes
Clearly defined data within a row - No

Setup MORPH_PERIOD for data recovery, available options are

thisweek (default)
thismonth
lastmonth
custom = yyyy-mm (eg. 2017-01)

Contributors

Last run completed successfully 2018-04-23.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.0.0 [1G-----> Installing dependencies using bundler 1.15.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Fetching gem metadata from https://rubygems.org/........ [1G Fetching version metadata from https://rubygems.org/. [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation. [1G Using bundler 1.15.2 [1G Fetching unf_ext 0.0.7.1 [1G Installing unf_ext 0.0.7.1 with native extensions [1G Fetching httpclient 2.6.0.1 [1G Installing httpclient 2.6.0.1 [1G Fetching mime-types 2.5 [1G Installing mime-types 2.5 [1G Fetching net-http-digest_auth 1.4 [1G Installing net-http-digest_auth 1.4 [1G Fetching net-http-persistent 2.9.4 [1G Installing net-http-persistent 2.9.4 [1G Fetching mini_portile 0.6.2 [1G Installing mini_portile 0.6.2 [1G Fetching ntlm-http 0.1.1 [1G Installing ntlm-http 0.1.1 [1G Fetching webrobots 0.1.1 [1G Installing webrobots 0.1.1 [1G Fetching sqlite3 1.3.10 [1G Installing sqlite3 1.3.10 with native extensions [1G Fetching unf 0.1.4 [1G Installing unf 0.1.4 [1G Fetching nokogiri 1.6.6.2 [1G Installing nokogiri 1.6.6.2 with native extensions [1G Fetching sqlite_magic 0.0.3 [1G Installing sqlite_magic 0.0.3 [1G Fetching domain_name 0.5.24 [1G Installing domain_name 0.5.24 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching http-cookie 1.0.2 [1G Installing http-cookie 1.0.2 [1G Fetching mechanize 2.7.3 [1G Installing mechanize 2.7.3 [1G Bundle complete! 2 Gemfile dependencies, 17 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into ./vendor/bundle. [1G Bundle completed (26.60s) [1G Cleaning up the bundler cache. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Getting data in `thismonth`, changable via MORPH_PERIOD environment Skipping already saved record 194/2015 Skipping already saved record 47/2018 Skipping already saved record 41/2018 Skipping already saved record 233/2010 Skipping already saved record 93/2017 Skipping already saved record 45/2018 Skipping already saved record 39/2018 Skipping already saved record 43/2018 Skipping already saved record 36/2018 Skipping already saved record 38/2018 Skipping already saved record 46/2018 Skipping already saved record 37/2018 Skipping already saved record 35/2018 Skipping already saved record 218/2011 Skipping already saved record 142/2016 Skipping already saved record 42/2018 Skipping already saved record 40/2018 Skipping already saved record 44/2018

Data

Downloaded 1492 times by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (173 KB) Use the API

rows 10 / 469

info_url	comment_url	council_reference	date_received	address	description	date_scraped
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+84%2F2015	84/2015	2015-06-10	42 Badminton Road CROYDON NSW 2132	Dwelling - Alts & Adds	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+73%2F2015	73/2015	2015-05-25	18 Beaumaris Street ENFIELD NSW 2136	Dwelling - New	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+81%2F2015	81/2015	2015-06-04	16 Blenheim Street CROYDON PARK NSW 2133	Garage	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+78%2F2015	78/2015	2015-05-29	14 Brighton Street CROYDON NSW 2132	First Floor Addition	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+75%2F2015	75/2015	2015-05-26	59 Brighton Street CROYDON NSW 2132	Shed and Carport Shed Carport	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+65%2F2015	65/2015	2015-05-01	164 Liverpool Road ENFIELD NSW 2136	Mixed Development	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+77%2F2015	77/2015	2015-05-26	13 Morwick Street STRATHFIELD NSW 2135	Demolition of existing building Dwelling - Demolition and Residential Demolition of existing building Dwelling - Demolition Residential Flat Building	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+76%2F2015	76/2015	2015-05-26	22A-26 Parramatta Road CROYDON NSW 2132	Shop - Fitout	2015-06-17
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+83%2F2015	83/2015	2015-06-09	15 Murray Street CROYDON NSW 2132	Garage	2015-06-18
https://ecouncil.burwood.nsw.gov.au/eservice/daEnquiryInit.do?doc_typ=10&nodeNum=219	mailto:council@burwood.nsw.gov.au?subject=Development+Application+Enquiry%3A+86%2F2015	86/2015	2015-06-15	22 Church Street BURWOOD NSW 2134	Dwelling - Alts & Adds	2015-06-22

Statistics

Average successful run time: 3 minutes

Total run time: about 1 month

Total cpu time used: 25 minutes

Total disk space used: 203 KB

History

Auto ran revision 9637a203 and completed successfully 2018-04-23.

run time 2 minutes

nothing changed in the database

20 pages scraped
Auto ran revision 9637a203 and completed successfully 2018-04-22.

run time 2 minutes

nothing changed in the database

19 pages scraped
Auto ran revision 9637a203 and completed successfully 2018-04-21.

run time 2 minutes

1 record added in the database

19 pages scraped
Auto ran revision 9637a203 and completed successfully 2018-04-20.

run time 2 minutes

nothing changed in the database

18 pages scraped
Auto ran revision 9637a203 and completed successfully 2018-04-19.

run time 2 minutes

1 record added in the database

18 pages scraped
...
Created on morph.io 2015-06-17

Show complete history

Scraper code

Ruby

burwood_development_applications / scraper.rb

git clone URL