planningalerts-scrapers / liverpool-py

Liverpool City Council Development Applications


Liverpool City Council Scraper

Liverpool City Council Scraper involves the followings

  • Server - .NET - XCTrack
  • XML - Yes
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • last14days (default)
  • thisweek
  • thismonth
  • lastmonth

Thanks and Enjoy

Contributors dpschramm henare LoveMyData MichaelBone

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching connection_pool 2.2.2  Using bundler 1.15.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Installing mini_portile2 2.3.0  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Installing webrobots 0.1.2  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.8.5  Installing nokogiri 1.8.5 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (24.62s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data in `last14days`, changable via MORPH_PERIOD environment Skipping already saved record BC-50/2019 Skipping already saved record CCB-214/2019 Skipping already saved record CD-290/2019 Skipping already saved record CD-291/2019 Skipping already saved record CD-292/2019 Skipping already saved record CD-293/2019 Skipping already saved record CD-418/2018/A Skipping already saved record DA-229/2019 Skipping already saved record DA-230/2019 Skipping already saved record DA-231/2019 Skipping already saved record DA-232/2019 Skipping already saved record DA-233/2019 Skipping already saved record DA-234/2019 Skipping already saved record DA-235/2019 Skipping already saved record DA-237/2019 Skipping already saved record DA-565/2016/B Skipping already saved record DA-730/2013/A Skipping already saved record DA-775/2018/A Skipping already saved record CCB-215/2019 Skipping already saved record CCB-216/2019 Skipping already saved record CCB-217/2019 Skipping already saved record CD-294/2019 Skipping already saved record CD-296/2019 Skipping already saved record CD-814/2016/A Skipping already saved record DA-238/2019 Skipping already saved record BC-51/2019 Skipping already saved record BC-52/2019 Skipping already saved record BC-53/2019 Skipping already saved record CCB-218/2019 Skipping already saved record CCB-219/2019 Skipping already saved record CD-297/2019 Skipping already saved record CD-298/2019 Skipping already saved record CD-299/2019 Skipping already saved record CD-300/2019 Skipping already saved record CD-301/2019 Skipping already saved record CD-302/2019 Skipping already saved record CD-303/2019 Skipping already saved record CD-304/2019 Skipping already saved record CD-305/2019 Skipping already saved record CD-306/2019 Skipping already saved record CD-307/2019 Skipping already saved record CD-308/2019 Skipping already saved record BC-54/2019 Skipping already saved record CCB-134/2019/B Skipping already saved record CCB-220/2019 Skipping already saved record CCB-221/2019 Skipping already saved record CCB-222/2019 Skipping already saved record CCB-224/2019 Skipping already saved record CD-309/2019 Skipping already saved record CD-310/2019 Skipping already saved record CD-311/2019 Skipping already saved record CD-312/2019 Skipping already saved record CD-313/2019 Skipping already saved record DA-240/2019 Skipping already saved record DA-241/2019 Skipping already saved record SC-40/2019 Skipping already saved record BC-55/2019 Skipping already saved record CCB-225/2019 Skipping already saved record CCB-226/2019 Skipping already saved record CCB-227/2019 Skipping already saved record CD-314/2019 Skipping already saved record CD-315/2019 Skipping already saved record CD-316/2019 Skipping already saved record CD-317/2019 Skipping already saved record CD-318/2019 Skipping already saved record CD-319/2019 Skipping already saved record CD-320/2019 Skipping already saved record CD-321/2019 Skipping already saved record CD-322/2019 Skipping already saved record CD-323/2019 Skipping already saved record CD-324/2019 Skipping already saved record CD-325/2019 Skipping already saved record CD-92/2018/A Skipping already saved record DA-102/2019/A Skipping already saved record DA-1109/2016/A Skipping already saved record DA-1237/2015/B Skipping already saved record DA-242/2019 Skipping already saved record DA-243/2019 Skipping already saved record DA-244/2019 Skipping already saved record DA-245/2019 Skipping already saved record DA-246/2019 Skipping already saved record DA-776/2018/A Skipping already saved record CCB-228/2019 Skipping already saved record CCB-230/2019 Skipping already saved record CD-326/2019 Skipping already saved record DA-247/2019 Skipping already saved record DA-248/2019 Skipping already saved record DA-249/2019 Skipping already saved record DA-250/2019 Skipping already saved record DA-251/2019 Skipping already saved record SC-41/2019 Skipping already saved record SC-42/2019 Skipping already saved record BC-56/2019 Skipping already saved record CCB-231/2019 Skipping already saved record CCB-232/2019 Skipping already saved record CCB-233/2019 Skipping already saved record CD-327/2019 Skipping already saved record CD-328/2019 Skipping already saved record CD-329/2019 Skipping already saved record CD-330/2019 Skipping already saved record CD-331/2019 Skipping already saved record CD-332/2019 Skipping already saved record CD-682/2017/A Skipping already saved record DA-252/2019 Skipping already saved record DA-81/2018/A Skipping already saved record ST-5/2019 Skipping already saved record CCB-234/2019 Skipping already saved record CCB-235/2019 Skipping already saved record CD-333/2019 Skipping already saved record CD-334/2019 Skipping already saved record CD-335/2019 Skipping already saved record CD-336/2019 Skipping already saved record DA-1066/2015/B Skipping already saved record DA-254/2019 Skipping already saved record DA-255/2019 Skipping already saved record DA-256/2019 Skipping already saved record RD-3/2019 Skipping already saved record CD-337/2019 Skipping already saved record DA-257/2019 Skipping already saved record DA-258/2019 Skipping already saved record DA-259/2019 Skipping already saved record DA-260/2019 Skipping already saved record SC-43/2019 Skipping already saved record SC-44/2019 Skipping already saved record SPC-11/2019 Skipping already saved record CCB-236/2019 Skipping already saved record CCB-236/2019/A Skipping already saved record CCB-237/2019 Skipping already saved record CCB-238/2019 Skipping already saved record CCB-239/2019 Skipping already saved record CCB-240/2019 Skipping already saved record CCB-241/2019 Skipping already saved record CCB-242/2019 Skipping already saved record CCB-243/2019 Skipping already saved record CCB-244/2019 Skipping already saved record CCB-245/2019 Skipping already saved record CD-338/2019 Skipping already saved record CD-339/2019 Skipping already saved record CD-341/2019 Skipping already saved record CD-342/2019 Skipping already saved record CD-343/2019 Skipping already saved record CD-344/2019 Skipping already saved record CD-345/2019 Skipping already saved record CD-346/2019 Skipping already saved record CD-347/2019 Skipping already saved record CD-349/2019 Skipping already saved record CD-350/2019 Skipping already saved record CD-82/2019/A Skipping already saved record DA-261/2019

Data

Downloaded 4389 times by slav123 openaustralia WilliamYun jesse-a-reilly TheZepto razz969 LoveMyData CBP1 JPHePlanning henare Jeremy-Charlton MikeRalphson derman232 horizons444 jayzhouny brickworksbp uncoverRiley Jedwards157 Ceebee74

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (5.72 MB) Use the API

rows 10 / 11884

date_received description info_url date_scraped comment_url address council_reference
2015-06-01
two double storey semi detached dwelling, demolition and torrens title subdivision of one lot into two lots
2015-06-30
14 BRAIN AVENUE, LURNEA NSW 2170
DA-442/2015
2015-06-01
installation of a Telecommunications facility including 40m monopole, equipment shelter and ancillary equipment
2015-06-30
50 VICTOR AVENUE, KEMPS CREEK NSW 2178
DA-443/2015
2015-06-01
S(96)1A to extend retaining wall and fill
2015-06-30
48 MCGOWEN CRESCENT, LIVERPOOL NSW 2170
DA-1723/2012/C
2015-06-01
two storey dwelling and front fence
2015-06-30
LOT 40 ARDENNES AVENUE, EDMONDSON PARK NSW 2174
DA-444/2015
2015-06-01
torrens tilte subdivision of one lot into two lots
2015-06-30
15 TEMPLETON CRESCENT, MOOREBANK NSW 2170
SC-53/2015
2015-06-01
Boundary adjustment
2015-06-30
115 CROATIA AVENUE, EDMONDSON PARK NSW 2174
DA-1084/2014/A
2015-06-01
concrete hardstand and concrete on ground storage bin and relocation of property fence along eastern boundary
2015-06-30
102 JEDDA ROAD, PRESTONS NSW 2170
DA-445/2015
2015-06-02
inground swimming pool
2015-06-30
85 SADLEIR AVENUE, ASHCROFT NSW 2168
DA-446/2015
2015-06-02
demolition of existing structures, subdivision of the existing lot into two torrens tiltle lots and the construction of a double storey detached dwelling on each lot
2015-06-30
26 BOUNDARY ROAD, LIVERPOOL NSW 2170
DA-447/2015
2015-06-02
cutting hole in wall between two buildings to gain easy access for storage
2015-06-30
17 MEMORIAL AVENUE, LIVERPOOL NSW 2170
DA-448/2015

Statistics

Average successful run time: 4 minutes

Total run time: 4 days

Total cpu time used: about 1 hour

Total disk space used: 6.27 MB

History

  • Auto ran revision 81215006 and completed successfully .
    nothing changed in the database
  • Auto ran revision 81215006 and completed successfully .
    28 records added in the database
  • Auto ran revision 81215006 and completed successfully .
    7 records added in the database
  • Auto ran revision 81215006 and completed successfully .
    8 records added in the database
  • Auto ran revision 81215006 and completed successfully .
    14 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

liverpool-py / scraper.rb