LoveMyData / city_of_adelaide_development_applications

City of Adelaide development applications


City of Adelaide Scraper

Note #1, this scraper uses brute force to query every single DA in sequential order until 10 consecutive errors

Note #2, this scraper will only obtain initial DA, any subsequent DAs like DA/1/2017/A (if any) will not be captured

  • Server - .NET - ePathway - Hate this
  • Cookie tracking - Yes
  • Pagnation - No
  • Javascript - Yes, Hate this too
  • Clearly defined data within a row - No but acceptable
  • User-Agent - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • year value (default is this year, min. 2006)

Enjoy

Contributors CloCkWeRX LoveMyData equivalentideas nik3daz maxious

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.4  Fetching httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.4 with native extensions  Fetching net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-digest_auth 1.4.1  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing mini_portile2 2.3.0  Installing sqlite3 1.3.13 with native extensions  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.8.1  Installing nokogiri 1.8.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (27.96s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in year '2018', changable via MORPH_PERIOD environment Going to scrape 'DA' type of applications Skipping already saved record DA/398/2018 Saving record DA/399/2018, A 1m x 1m wooden sign printed with The Foodprint Experience Logo to be attached on the Eastern side of the verandah. 2 x wooden signs with logo and cafe information to be attached to the planter box on the Franklin St side. Saving record DA/400/2018, Install bus shelter - displaying double sided illuminated advertising panels. Saving record DA/401/2018, Restoration of Heritage listed cottage with new single story extension, construction of new 2 storey 5 bed student accommodation to rear of property Saving record DA/402/2018, Install bus shelter - displaying double sided illuminated advertising panels. Saving record DA/403/2018, Install bus shelter - displaying double sided illuminated advertising panels. Saving record DA/404/2018, Install bus shelter - displaying double sided illuminated advertising panels. Saving record DA/405/2018, I am planning on ceasing this agreement with Oaks Istay in August 2018 and listing the property myself as a short term holiday rental property on such advertising platforms like AirBnB. Saving record DA/406/2018, Install bus shelter - displaying double sided illuminated advertising panels. Saving record DA/407/2018, A Temporary Digital Gateway - to be installed 19th July 2018 - Removed 25th July 2018 Saving record DA/408/2018, Signage upgrade Saving record DA/409/2018, Park Land - Tantanya Wama (Park 26) - Adelaide Convention Centre Lawn - Land 10ft Container, install 10mx18m Marquee. Park Land - Pinky Flat - Land 10ft Container Projector housing Torrens River - Aqua Screen Saving record DA/410/2018, Installation of new grease arrestor and associated sewer and vent pipework Saving record DA/411/2018, Erecting shelters and benches on the sidelines of the netball courts, installing water bubblers and court signage. Going to scrape 'S49' type of applications Skipping already saved record S49/12/2018 Going to scrape 'S10' type of applications Skipping already saved record S10/35/2018 Going to scrape 'HIS' type of applications Skipping already saved record HIS/38/2018 Going to scrape 'LD' type of applications Skipping already saved record LD/14/2018

Data

Downloaded 226 times by LoveMyData

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.87 MB) Use the API

rows 10 / 10073

council_reference address description info_url comment_url date_scraped date_received
DA/1/2017
17A Gray Street, ADELAIDE SA 5000
Internal and external alterations to existing dwelling.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-03
DA/2/2017
226 Wright Street, ADELAIDE SA 5000
Construct louvred roof over courtyard.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-03
DA/3/2017
30-32 Provost Street, NORTH ADELAIDE SA 5006
Construct upper level dwelling addition with new access at ground floor level, replace carport roof and swimming pool.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-03
DA/4/2017
54-55 South Terrace, ADELAIDE SA 5000
Change the use of existing office to child care/early learning centre, internal and external changes and construct extension on adjacent vacant land with car parking at rear.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-03
DA/5/2017
92-106 Rundle Mall, ADELAIDE SA 5000
Installation of tile and glass cladding, entry upgrades and new identification signage.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-03
DA/6/2017
War Memorial Drive, NORTH ADELAIDE SA 5006
Erect white picket memorial fence from northern side of grand stand to scoreboard.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-03
DA/7/2017
Rundle Road, ADELAIDE SA 5000
Erect various temporary structures for 'Garden of Unearthly Delights'.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-04
DA/8/2017
1-27 Moonta Street, ADELAIDE SA 5000
Fit-out (Shop 2 - Universal Travel).
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-11
DA/9/2017
141-143 Currie Street, ADELAIDE SA 5000
Repairs to north western side of balcony.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-09
DA/10/2017
21-27 Pirie Street, ADELAIDE SA 5000
Installation of two(2) bronze plaques to street facing facade.
mailto:city@cityofadelaide.com.au
2017-10-16
2017-01-10

Statistics

Average successful run time: 6 minutes

Total run time: about 18 hours

Total cpu time used: 9 minutes

Total disk space used: 3.9 MB

History

  • Auto ran revision bee60f45 and completed successfully .
    13 records added in the database
  • Auto ran revision bee60f45 and completed successfully .
    nothing changed in the database
  • Auto ran revision bee60f45 and completed successfully .
    10 records added in the database
    133 pages scraped
  • Auto ran revision bee60f45 and completed successfully .
    3 records added in the database
    119 pages scraped
  • Auto ran revision bee60f45 and completed successfully .
    nothing changed in the database
    113 pages scraped
  • ...
  • Created on morph.io

Show complete history