planningalerts-scrapers / liverpool-py

Liverpool City Council Development Applications

Scrapes eplanning.liverpool.nsw.gov.au

Terms and Conditions


Liverpool City Council Scraper

Liverpool City Council Scraper involves the followings

  • Server - .NET - XCTrack
  • XML - Yes
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • last14days (default)
  • thisweek
  • thismonth
  • lastmonth

Thanks and Enjoy

Contributors dpschramm henare LoveMyData MichaelBone

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching net-http-digest_auth 1.4.1  Installing mime-types-data 3.2018.0812  Fetching mini_portile2 2.3.0  Installing net-http-digest_auth 1.4.1  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Installing mini_portile2 2.3.0  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.8.5  Installing nokogiri 1.8.5 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (27.07s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `last14days`, changable via MORPH_PERIOD environment Skipping already saved record BC-12/2019 Skipping already saved record DA-56/2019 Skipping already saved record DA-57/2019 Skipping already saved record DA-58/2019 Skipping already saved record DA-59/2019 Skipping already saved record DA-60/2019 Skipping already saved record DA-61/2019 Skipping already saved record DA-62/2019 Skipping already saved record SC-10/2019 Skipping already saved record VPA-21 Skipping already saved record VPA-22 Skipping already saved record VPA-23 Skipping already saved record VPA-24 Skipping already saved record VPA-25 Skipping already saved record VPA-26 Skipping already saved record VPA-27 Skipping already saved record VPA-28 Skipping already saved record VPA-29 Skipping already saved record CCB-54/2019 Skipping already saved record CCB-55/2019 Skipping already saved record CCB-56/2019 Skipping already saved record CCB-57/2019 Skipping already saved record CCB-58/2019 Skipping already saved record CCB-59/2019 Skipping already saved record CCB-60/2019 Skipping already saved record CCB-61/2019 Skipping already saved record CCB-62/2019 Skipping already saved record CCB-63/2019 Skipping already saved record CCB-64/2019 Skipping already saved record CD-55/2019 Skipping already saved record CD-56/2019 Skipping already saved record CD-57/2019 Skipping already saved record CD-58/2019 Skipping already saved record CD-59/2019 Skipping already saved record CD-60/2019 Skipping already saved record CD-61/2019 Skipping already saved record CD-62/2019 Skipping already saved record DA-63/2019 Skipping already saved record DA-64/2019 Skipping already saved record DA-65/2019 Skipping already saved record DA-67/2019 Skipping already saved record DA-68/2019 Skipping already saved record DA-682/2017/A Skipping already saved record DA-694/2017/A Skipping already saved record DA-852/2018/A Skipping already saved record BC-13/2019 Skipping already saved record CCB-65/2019 Skipping already saved record CCB-66/2019 Skipping already saved record CCB-67/2019 Skipping already saved record CCB-68/2019 Skipping already saved record CCE-1/2019 Skipping already saved record CD-63/2019 Skipping already saved record DA-69/2019 Skipping already saved record VPA-30 Skipping already saved record CCB-69/2019 Skipping already saved record CCB-70/2019 Skipping already saved record CCB-71/2019 Skipping already saved record CCB-72/2019 Skipping already saved record CCB-73/2019 Skipping already saved record CCB-74/2019 Skipping already saved record CCB-75/2019 Skipping already saved record CCB-76/2019 Skipping already saved record CD-64/2019 Skipping already saved record CD-65/2019 Skipping already saved record CD-66/2019 Skipping already saved record CD-67/2019 Skipping already saved record CD-68/2019 Skipping already saved record DA-70/2019 Skipping already saved record BC-14/2019 Skipping already saved record DA-71/2019 Skipping already saved record DA-72/2019 Skipping already saved record DA-73/2019 Skipping already saved record DA-75/2019 Skipping already saved record SC-11/2019 Skipping already saved record CD-69/2019 Skipping already saved record DA-76/2019 Skipping already saved record DA-767/2017/B Skipping already saved record DA-77/2019 Skipping already saved record DA-78/2019 Skipping already saved record DA-79/2019 Skipping already saved record SC-12/2019 Skipping already saved record SPC-1/2019 Skipping already saved record ST-1/2019 Skipping already saved record BC-15/2019 Skipping already saved record CD-70/2019 Skipping already saved record DA-80/2019 Skipping already saved record DA-81/2019 Skipping already saved record SC-13/2019 Skipping already saved record BC-16/2019 Skipping already saved record CCB-77/2019 Skipping already saved record CCB-78/2019 Skipping already saved record CCB-79/2019 Skipping already saved record CCB-80/2019 Skipping already saved record CCB-81/2019 Skipping already saved record CD-1304/2018/A Skipping already saved record CD-71/2019 Skipping already saved record CD-72/2019 Skipping already saved record CD-73/2019 Skipping already saved record CD-74/2019 Skipping already saved record CD-75/2019 Skipping already saved record CD-76/2019 Skipping already saved record CD-77/2019 Skipping already saved record CD-78/2019 Skipping already saved record CD-79/2019 Skipping already saved record DA-1116/2017/B Skipping already saved record DA-210/2017/B Skipping already saved record DA-82/2019 Skipping already saved record DA-83/2019 Skipping already saved record DA-84/2019 Skipping already saved record RZ-1/2019 Skipping already saved record BC-17/2019 Skipping already saved record CD-80/2019 Skipping already saved record CD-81/2019 Skipping already saved record CD-82/2019 Skipping already saved record CD-83/2019 Skipping already saved record DA-734/2016/A Skipping already saved record DA-85/2019 Skipping already saved record DA-970/2015/A Skipping already saved record SC-14/2019 Skipping already saved record BC-18/2019 Skipping already saved record CCB-82/2019 Skipping already saved record CCB-83/2019 Skipping already saved record CCB-84/2019 Skipping already saved record CCB-85/2019 Skipping already saved record CCB-86/2019 Skipping already saved record CD-85/2019 Skipping already saved record DA-86/2019 Skipping already saved record SC-16/2019 Skipping already saved record SC-17/2019

Data

Downloaded 3990 times by slav123 openaustralia jesse-a-reilly TheZepto LoveMyData razz969 CBP1 JPHePlanning Jeremy-Charlton MikeRalphson derman232 horizons444 jayzhouny brickworksbp uncoverRiley Jedwards157 Ceebee74 henare

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (5.45 MB) Use the API

rows 10 / 11175

date_received description info_url date_scraped comment_url address council_reference
2015-06-01
two double storey semi detached dwelling, demolition and torrens title subdivision of one lot into two lots
2015-06-30
14 BRAIN AVENUE, LURNEA NSW 2170
DA-442/2015
2015-06-01
installation of a Telecommunications facility including 40m monopole, equipment shelter and ancillary equipment
2015-06-30
50 VICTOR AVENUE, KEMPS CREEK NSW 2178
DA-443/2015
2015-06-01
S(96)1A to extend retaining wall and fill
2015-06-30
48 MCGOWEN CRESCENT, LIVERPOOL NSW 2170
DA-1723/2012/C
2015-06-01
two storey dwelling and front fence
2015-06-30
LOT 40 ARDENNES AVENUE, EDMONDSON PARK NSW 2174
DA-444/2015
2015-06-01
torrens tilte subdivision of one lot into two lots
2015-06-30
15 TEMPLETON CRESCENT, MOOREBANK NSW 2170
SC-53/2015
2015-06-01
Boundary adjustment
2015-06-30
115 CROATIA AVENUE, EDMONDSON PARK NSW 2174
DA-1084/2014/A
2015-06-01
concrete hardstand and concrete on ground storage bin and relocation of property fence along eastern boundary
2015-06-30
102 JEDDA ROAD, PRESTONS NSW 2170
DA-445/2015
2015-06-02
inground swimming pool
2015-06-30
85 SADLEIR AVENUE, ASHCROFT NSW 2168
DA-446/2015
2015-06-02
demolition of existing structures, subdivision of the existing lot into two torrens tiltle lots and the construction of a double storey detached dwelling on each lot
2015-06-30
26 BOUNDARY ROAD, LIVERPOOL NSW 2170
DA-447/2015
2015-06-02
cutting hole in wall between two buildings to gain easy access for storage
2015-06-30
17 MEMORIAL AVENUE, LIVERPOOL NSW 2170
DA-448/2015

Statistics

Average successful run time: 5 minutes

Total run time: 4 days

Total cpu time used: about 1 hour

Total disk space used: 5.99 MB

History

  • Auto ran revision 81215006 and completed successfully .
    3 records added in the database
    1 page scraped
  • Auto ran revision 81215006 and completed successfully .
    17 records added in the database
    1 page scraped
  • Auto ran revision 81215006 and completed successfully .
    2 records added in the database
    1 page scraped
  • Auto ran revision 81215006 and completed successfully .
    22 records added in the database
    1 page scraped
  • Auto ran revision 81215006 and completed successfully .
    5 records added in the database
    1 page scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

liverpool-py / scraper.rb