planningalerts-scrapers / city_of_adelaide_development_applications

City of Adelaide development applications


City of Adelaide Scraper

Note #1, this scraper uses brute force to query every single DA in sequential order until 10 consecutive errors

Note #2, this scraper will only obtain initial DA, any subsequent DAs like DA/1/2017/A (if any) will not be captured

  • Server - .NET - ePathway - Hate this
  • Cookie tracking - Yes
  • Pagnation - No
  • Javascript - Yes, Hate this too
  • Clearly defined data within a row - No but acceptable
  • User-Agent - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • year value (default is this year, min. 2006)

Enjoy

Contributors equivalentideas CloCkWeRX nik3daz maxious LoveMyData mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.4  Fetching httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.4 with native extensions  Fetching net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Installing mini_portile2 2.3.0  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Installing webrobots 0.1.2  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.8.1  Installing nokogiri 1.8.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (43.60s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in year '2018', changable via MORPH_PERIOD environment Going to scrape 'DA' type of applications Skipping already saved record DA/351/2018 Going to scrape 'S49' type of applications Skipping already saved record S49/11/2018 Going to scrape 'S10' type of applications Skipping already saved record S10/31/2018 Going to scrape 'HIS' type of applications Skipping already saved record HIS/38/2018 Going to scrape 'LD' type of applications

Data

Downloaded 1202 times by openaustralia LoveMyData TheZepto equivalentideas ljnicol

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (368 KB) Use the API

rows 10 / 881

info_url comment_url council_reference on_notice_to address description date_scraped date_received
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0137%2F2016
DA/0137/2016
2016-05-26
ELLIS PARK, West Terrace, ADELAIDE SA 5000
Establish automatic weather station and fencing. (non-complying)
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0519%2F2015%2FA
DA/0519/2015/A
2016-05-12
178 Stanley Street, NORTH ADELAIDE SA 5006
Vary previous authorisation internal and external alterations and additions to existing residence including lower level garage and store, ground floor extension and extend height of boundary wall and construction of retaining wall on the north-eastern boundary - VARIATION - amended roof form (and height of wall on eastern side boundary) and use of materials.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0207%2F2016
DA/0207/2016
2016-05-12
77-78 Palmer Place, NORTH ADELAIDE SA 5006
Demolish existing building and construct two storey dwelling including undercroft parking and swimming pool.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0124%2F2016
DA/0124/2016
2016-05-19
68-70 Barnard Street, NORTH ADELAIDE SA 5006
Construct outbuilding - domestic garage on northern boundary with associated fence demolition.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0211%2F2016
DA/0211/2016
2016-05-26
21 Vincent Street, ADELAIDE SA 5000
Dwelling alterations and additions with associated roof top deck.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0034%2F2016
DA/0034/2016
2016-05-26
14-24 Lakeman Street, NORTH ADELAIDE SA 5006
Re-Notification - Demolition of existing structures and construct two detached 2 storey dwellings and remove significant tree.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0208%2F2016
DA/0208/2016
2016-05-26
60-70 Queen Street, ADELAIDE SA 5000
Demolish existing building and construct 4 level residential flat building with 12 apartments and carparking on ground floor.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0072%2F2016
DA/0072/2016
2016-05-26
128-130 Kermode Street, NORTH ADELAIDE SA 5006
Removal of regulated tree, (Agonis flexuosa - Willow Myrtle).
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0200%2F2016
DA/0200/2016
2016-05-26
10 James Street, ADELAIDE SA 5000
Construct 2nd storey rear addition to existing dwelling.
2016-05-12
mailto:d.planner@adelaidecitycouncil.com?subject=Development+Application+Enquiry%3A+DA%2F0794%2F2013%2FB
DA/0794/2013/B
2016-06-09
87 Lefevre Terrace, NORTH ADELAIDE SA 5006
Vary previous authorisation internal and external alterations and construct two storey rear addition (following demolition of portion of rear of dwelling) including double garage with deck above - VARIATION - deletion of condition 2, relating to the installation of glass screening.
2016-05-25

Statistics

Average successful run time: 1 minute

Total run time: about 12 hours

Total cpu time used: 13 minutes

Total disk space used: 405 KB

History

  • Auto ran revision 00e8dd01 and completed successfully .
    nothing changed in the database
  • Auto ran revision 00e8dd01 and completed successfully .
    12 records added in the database
    135 pages scraped
  • Auto ran revision 00e8dd01 and completed successfully .
    4 records added in the database
    117 pages scraped
  • Auto ran revision 00e8dd01 and completed successfully .
    3 records added in the database
  • Auto ran revision 00e8dd01 and completed successfully .
    4 records added in the database
    119 pages scraped
  • ...
  • Created on morph.io

Show complete history