City of Adelaide Scraper

Note #1, this scraper uses brute force to query every single DA in sequential order until 10 consecutive errors

Note #2, this scraper will only obtain initial DA, any subsequent DAs like DA/1/2017/A (if any) will not be captured

  • Server - .NET - ePathway - Hate this
  • Cookie tracking - Yes
  • Pagnation - No
  • Javascript - Yes, Hate this too
  • Clearly defined data within a row - No but acceptable
  • User-Agent - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • year value (default is this year, min. 2006)

Enjoy

Contributors equivalentideas CloCkWeRX nik3daz maxious LoveMyData mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching unf_ext 0.0.7.4  Using bundler 1.15.2  Fetching httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4.1  Installing unf_ext 0.0.7.4 with native extensions  Installing httpclient 2.8.3  Fetching net-http-persistent 2.9.4  Installing net-http-digest_auth 1.4.1  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Installing mini_portile2 2.3.0  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing webrobots 0.1.2  Installing sqlite3 1.3.13 with native extensions  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.8.1  Installing nokogiri 1.8.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (26.71s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in year '2018', changable via MORPH_PERIOD environment Going to scrape 'DA' type of applications Skipping already saved record DA/1027/2018 Saving record DA/1028/2018, 313-319 Melbourne Street, NORTH ADELAIDE SA 5006 Saving record DA/1029/2018, 435 Gilles Street, ADELAIDE SA 5000 Saving record DA/1030/2018, 29 Blackburn Street, ADELAIDE SA 5000 Going to scrape 'S49' type of applications Skipping already saved record S49/31/2018 Going to scrape 'S10' type of applications Skipping already saved record S10/63/2018 Going to scrape 'HIS' type of applications Skipping already saved record HIS/38/2018 Going to scrape 'LD' type of applications Skipping already saved record LD/36/2018

Data

Downloaded 1405 times by openaustralia LoveMyData jesse-a-reilly TheZepto equivalentideas ljnicol

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (662 KB) Use the API

rows 10 / 1617

info_url comment_url council_reference on_notice_to address description date_scraped date_received
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0137%2F2016
DA/0137/2016
2016-05-26
ELLIS PARK, West Terrace, ADELAIDE SA 5000
Establish automatic weather station and fencing. (non-complying)
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0519%2F2015%2FA
DA/0519/2015/A
2016-05-12
178 Stanley Street, NORTH ADELAIDE SA 5006
Vary previous authorisation internal and external alterations and additions to existing residence including lower level garage and store, ground floor extension and extend height of boundary wall and construction of retaining wall on the north-eastern boundary - VARIATION - amended roof form (and height of wall on eastern side boundary) and use of materials.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0207%2F2016
DA/0207/2016
2016-05-12
77-78 Palmer Place, NORTH ADELAIDE SA 5006
Demolish existing building and construct two storey dwelling including undercroft parking and swimming pool.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0124%2F2016
DA/0124/2016
2016-05-19
68-70 Barnard Street, NORTH ADELAIDE SA 5006
Construct outbuilding - domestic garage on northern boundary with associated fence demolition.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0211%2F2016
DA/0211/2016
2016-05-26
21 Vincent Street, ADELAIDE SA 5000
Dwelling alterations and additions with associated roof top deck.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0034%2F2016
DA/0034/2016
2016-05-26
14-24 Lakeman Street, NORTH ADELAIDE SA 5006
Re-Notification - Demolition of existing structures and construct two detached 2 storey dwellings and remove significant tree.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0208%2F2016
DA/0208/2016
2016-05-26
60-70 Queen Street, ADELAIDE SA 5000
Demolish existing building and construct 4 level residential flat building with 12 apartments and carparking on ground floor.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0072%2F2016
DA/0072/2016
2016-05-26
128-130 Kermode Street, NORTH ADELAIDE SA 5006
Removal of regulated tree, (Agonis flexuosa - Willow Myrtle).
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0200%2F2016
DA/0200/2016
2016-05-26
10 James Street, ADELAIDE SA 5000
Construct 2nd storey rear addition to existing dwelling.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0794%2F2013%2FB
DA/0794/2013/B
2016-06-09
87 Lefevre Terrace, NORTH ADELAIDE SA 5006
Vary previous authorisation internal and external alterations and construct two storey rear addition (following demolition of portion of rear of dwelling) including double garage with deck above - VARIATION - deletion of condition 2, relating to the installation of glass screening.
2016-05-25

Statistics

Average successful run time: 1 minute

Total run time: about 21 hours

Total cpu time used: 19 minutes

Total disk space used: 699 KB

History

  • Auto ran revision 00e8dd01 and completed successfully .
    3 records added in the database
    119 pages scraped
  • Auto ran revision 00e8dd01 and completed successfully .
    nothing changed in the database
    113 pages scraped
  • Auto ran revision 00e8dd01 and completed successfully .
    3 records added in the database
    119 pages scraped
  • Auto ran revision 00e8dd01 and completed successfully .
    4 records added in the database
  • Auto ran revision 00e8dd01 and completed successfully .
    2 records added in the database
    117 pages scraped
  • ...
  • Created on morph.io

Show complete history