planningalerts-scrapers / liverpool-py

Liverpool City Council Development Applications


Liverpool City Council Scraper

Liverpool City Council Scraper involves the followings

  • Server - .NET - XCTrack
  • XML - Yes
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • last14days (default)
  • thisweek
  • thismonth
  • lastmonth

Thanks and Enjoy

Contributors dpschramm henare LoveMyData MichaelBone

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Installing mini_portile2 2.3.0  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing webrobots 0.1.2  Installing sqlite3 1.3.13 with native extensions  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.8.5  Installing nokogiri 1.8.5 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (26.73s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `last14days`, changable via MORPH_PERIOD environment Skipping already saved record DA-180/2016/D Skipping already saved record DA-891/2018 Skipping already saved record DA-892/2018 Skipping already saved record DA-893/2018 Skipping already saved record DA-894/2018 Skipping already saved record RD-3/2018 Skipping already saved record SC-165/2018 Skipping already saved record SC-166/2018 Skipping already saved record SC-167/2018 Skipping already saved record SPC-58/2018 Skipping already saved record BC-120/2018 Skipping already saved record CCB-611/2015/A Skipping already saved record CCB-782/2018 Skipping already saved record CCB-783/2018 Skipping already saved record CCB-784/2018 Skipping already saved record CCB-785/2018 Skipping already saved record CCB-786/2018 Skipping already saved record CD-1216/2018 Skipping already saved record CD-1217/2018 Skipping already saved record CD-1218/2018 Skipping already saved record CD-1219/2018 Skipping already saved record CD-1220/2018 Skipping already saved record DA-519/2016/A Skipping already saved record DA-895/2018 Skipping already saved record DA-896/2018 Skipping already saved record DA-897/2018 Skipping already saved record DA-898/2018 Skipping already saved record DA-899/2018 Skipping already saved record CCB-787/2018 Skipping already saved record CCB-788/2018 Skipping already saved record CCB-789/2018 Skipping already saved record CCB-790/2018 Skipping already saved record CCB-791/2018 Skipping already saved record CCB-792/2018 Skipping already saved record CD-1221/2018 Skipping already saved record CD-1222/2018 Skipping already saved record CD-1223/2018 Skipping already saved record CD-1224/2018 Skipping already saved record CD-1225/2018 Skipping already saved record DA-900/2018 Skipping already saved record DA-901/2018 Skipping already saved record SC-168/2018 Skipping already saved record SC-169/2018 Skipping already saved record SC-170/2018 Skipping already saved record ST-28/2018 Skipping already saved record CCB-793/2018 Skipping already saved record CCB-794/2018 Skipping already saved record CD-1226/2018 Skipping already saved record CD-1227/2018 Skipping already saved record CD-1228/2018 Skipping already saved record CD-1229/2018 Skipping already saved record CD-1230/2018 Skipping already saved record CD-1231/2018 Skipping already saved record CD-1232/2018 Skipping already saved record CD-1233/2018 Skipping already saved record CD-1234/2018 Skipping already saved record DA-902/2018 Skipping already saved record SC-171/2018 Skipping already saved record BC-121/2018 Skipping already saved record CCB-795/2018 Skipping already saved record CCB-796/2018 Skipping already saved record CCB-797/2018 Skipping already saved record CCB-798/2018 Skipping already saved record CCB-799/2018 Skipping already saved record CD-1235/2018 Skipping already saved record CD-1236/2018 Skipping already saved record CD-1237/2018 Skipping already saved record CD-1238/2018 Skipping already saved record CD-1239/2018 Skipping already saved record CD-1240/2018 Skipping already saved record CD-1241/2018 Skipping already saved record CD-1242/2018 Skipping already saved record CD-1243/2018 Skipping already saved record CD-1244/2018 Skipping already saved record CD-1245/2018 Skipping already saved record CD-1246/2018 Skipping already saved record CD-1247/2018 Skipping already saved record SC-172/2018 Skipping already saved record BC-123/2018 Skipping already saved record CCB-800/2018 Skipping already saved record CCB-801/2018 Skipping already saved record CD-1249/2018 Skipping already saved record CD-1250/2018 Skipping already saved record DA-1053/2016/C Skipping already saved record DA-620/2015/C Skipping already saved record DA-903/2018 Skipping already saved record DA-905/2018 Skipping already saved record DA-906/2018 Skipping already saved record DA-907/2018 Skipping already saved record DA-908/2018 Skipping already saved record SC-173/2018 Skipping already saved record CCB-803/2018 Skipping already saved record CCB-804/2018 Skipping already saved record CCB-805/2018 Skipping already saved record CD-1251/2018 Skipping already saved record CD-1252/2018 Skipping already saved record DA-910/2018 Skipping already saved record OD-21/2018 Skipping already saved record SPC-59/2018 Skipping already saved record BC-124/2018 Skipping already saved record CCB-806/2018 Skipping already saved record CCB-807/2018 Skipping already saved record CCB-808/2018 Skipping already saved record CCB-809/2018 Skipping already saved record CCB-810/2018 Skipping already saved record CCB-811/2018 Skipping already saved record CCB-812/2018 Skipping already saved record CCB-813/2018 Skipping already saved record CCB-814/2018 Skipping already saved record CCB-815/2018 Skipping already saved record CCB-816/2018 Skipping already saved record CCB-817/2018 Skipping already saved record CCB-818/2018 Skipping already saved record CD-1253/2018 Skipping already saved record CD-1254/2018 Skipping already saved record DA-911/2018 Skipping already saved record DA-912/2018 Skipping already saved record SPC-60/2018 Skipping already saved record SPC-61/2018 Skipping already saved record BC-125/2018 Skipping already saved record BC-126/2018 Skipping already saved record CCB-819/2018 Skipping already saved record CCB-820/2018 Skipping already saved record CD-1255/2018 Skipping already saved record CD-1256/2018 Skipping already saved record CD-1257/2018 Skipping already saved record DA-914/2018 Skipping already saved record DA-915/2018 Skipping already saved record DA-916/2018 Skipping already saved record ST-29/2018 Skipping already saved record BC-127/2018 Skipping already saved record CCB-821/2018 Skipping already saved record CCB-822/2018 Skipping already saved record CCB-823/2018 Skipping already saved record CCB-824/2018 Skipping already saved record CCB-825/2018 Skipping already saved record CD-1258/2018 Skipping already saved record CD-1259/2018 Skipping already saved record CD-1260/2018 Skipping already saved record CD-1261/2018 Skipping already saved record CD-1262/2018 Skipping already saved record CD-1263/2018 Skipping already saved record CD-1264/2018 Skipping already saved record CD-1265/2018 Skipping already saved record CD-1266/2018 Skipping already saved record CD-1267/2018 Skipping already saved record CD-1268/2018 Skipping already saved record CD-1296/2017/A Skipping already saved record DA-222/2018/A Skipping already saved record DA-917/2018 Skipping already saved record DA-918/2018 Skipping already saved record SC-174/2018 Skipping already saved record SC-175/2018 Skipping already saved record SC-176/2018

Data

Downloaded 3767 times by slav123 openaustralia jesse-a-reilly TheZepto LoveMyData razz969 CBP1 JPHePlanning henare Jeremy-Charlton MikeRalphson derman232 Horizons444 jayzhouny brickworksbp uncoverRiley Ceebee74

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (5.23 MB) Use the API

rows 10 / 10646

date_received description info_url date_scraped comment_url address council_reference
2015-06-01
two double storey semi detached dwelling, demolition and torrens title subdivision of one lot into two lots
2015-06-30
14 BRAIN AVENUE, LURNEA NSW 2170
DA-442/2015
2015-06-01
installation of a Telecommunications facility including 40m monopole, equipment shelter and ancillary equipment
2015-06-30
50 VICTOR AVENUE, KEMPS CREEK NSW 2178
DA-443/2015
2015-06-01
S(96)1A to extend retaining wall and fill
2015-06-30
48 MCGOWEN CRESCENT, LIVERPOOL NSW 2170
DA-1723/2012/C
2015-06-01
two storey dwelling and front fence
2015-06-30
LOT 40 ARDENNES AVENUE, EDMONDSON PARK NSW 2174
DA-444/2015
2015-06-01
torrens tilte subdivision of one lot into two lots
2015-06-30
15 TEMPLETON CRESCENT, MOOREBANK NSW 2170
SC-53/2015
2015-06-01
Boundary adjustment
2015-06-30
115 CROATIA AVENUE, EDMONDSON PARK NSW 2174
DA-1084/2014/A
2015-06-01
concrete hardstand and concrete on ground storage bin and relocation of property fence along eastern boundary
2015-06-30
102 JEDDA ROAD, PRESTONS NSW 2170
DA-445/2015
2015-06-02
inground swimming pool
2015-06-30
85 SADLEIR AVENUE, ASHCROFT NSW 2168
DA-446/2015
2015-06-02
demolition of existing structures, subdivision of the existing lot into two torrens tiltle lots and the construction of a double storey detached dwelling on each lot
2015-06-30
26 BOUNDARY ROAD, LIVERPOOL NSW 2170
DA-447/2015
2015-06-02
cutting hole in wall between two buildings to gain easy access for storage
2015-06-30
17 MEMORIAL AVENUE, LIVERPOOL NSW 2170
DA-448/2015

Statistics

Average successful run time: 5 minutes

Total run time: 4 days

Total cpu time used: about 1 hour

Total disk space used: 5.77 MB

History

  • Auto ran revision 81215006 and completed successfully .
    nothing changed in the database
  • Auto ran revision 81215006 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 81215006 and completed successfully .
    31 records added in the database
  • Auto ran revision 81215006 and completed successfully .
    10 records added in the database
    1 page scraped
  • Auto ran revision 81215006 and completed successfully .
    14 records added in the database
    1 page scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

liverpool-py / scraper.rb