Liverpool-UK / lcr_planning_applications

Planning applications from all 6 Liverpool City Region local planning authorities (bar Halton at the moment)


This is a scraper that runs on Morph. To get started see the documentation

Environment Variables

The code expects (up to) two envirvonment variables to be set to run properly:

  • MORPH_DAYS How far back to look for new planning applications
  • MORPH_POSTCODE_GEOCODE_COUNT How many new postcodes it should attempt to geocode each time the scraper runs. This is used to populate the neighbourhood_latitude and neighbourhood_longitude columns in the database, which provide a rough location of the planning application. You should aim to make the number higher than the number of new applications each day, but not too high so you aren't hammering the postcodes.io API. A few hundred seems a sensible option.

Tools

  • To geocode the planning application locations we need to find the postcode. rake find_postcodes db=name-of-your-planning-apps-database | sort | uniq

Contributors adrianshort

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.5 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/..  Fetching dependency metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/adrianshort/uk_planning_scraper.git  Fetching public_suffix 4.0.3  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.6  Installing public_suffix 4.0.3  Installing connection_pool 2.2.2  Installing unf_ext 0.0.7.6 with native extensions  Fetching http-form_data 2.2.0  Installing http-form_data 2.2.0  Fetching http_parser.rb 0.6.0  Fetching httpclient 2.8.3  Installing http_parser.rb 0.6.0 with native extensions  Installing httpclient 2.8.3  Fetching mime-types-data 3.2019.1009  Installing mime-types-data 3.2019.1009  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.4.2  Installing sqlite3 1.4.2 with native extensions  Fetching net-http-persistent 3.1.0  Installing net-http-persistent 3.1.0  Fetching addressable 2.7.0  Installing addressable 2.7.0  Fetching mime-types 3.3.1  Installing mime-types 3.3.1  Fetching nokogiri 1.10.9  Installing nokogiri 1.10.9 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20190701  Installing domain_name 0.5.20190701  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching http 3.3.0  Installing http 3.3.0  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Using uk_planning_scraper 0.4.5 from https://github.com/adrianshort/uk_planning_scraper.git (at master@624efc0)  Bundle complete! 2 Gemfile dependencies, 25 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (25.86s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... 1 of 6: Scraping Halton Checking for validated_days applications... /app/vendor/bundle/ruby/2.5.0/bundler/gems/uk_planning_scraper-624efc0ecf0a/lib/uk_planning_scraper/authority_scrape_params.rb:15: warning: constant ::Fixnum is deprecated Planning system not supported for Halton at URL: https://webapp.halton.gov.uk/planningapps/ Checking for decided_days applications... Planning system not supported for Halton at URL: https://webapp.halton.gov.uk/planningapps/ 2 of 6: Scraping Knowsley /app/vendor/bundle/ruby/2.5.0/bundler/gems/uk_planning_scraper-624efc0ecf0a/lib/uk_planning_scraper/authority_scrape_params.rb:43: warning: constant ::Fixnum is deprecated Checking for validated_days applications... Using Idox scraper. Getting: https://planapp.knowsley.gov.uk/online-applications/search.do?action=advanced SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get local issuer certificate) Checking for decided_days applications... Using Idox scraper. /app/vendor/bundle/ruby/2.5.0/bundler/gems/uk_planning_scraper-624efc0ecf0a/lib/uk_planning_scraper/authority_scrape_params.rb:43: warning: constant ::Fixnum is deprecated Getting: https://planapp.knowsley.gov.uk/online-applications/search.do?action=advanced SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get local issuer certificate) 3 of 6: Scraping Liverpool /app/vendor/bundle/ruby/2.5.0/bundler/gems/uk_planning_scraper-624efc0ecf0a/lib/uk_planning_scraper/authority_scrape_params.rb:15: warning: constant ::Fixnum is deprecated Checking for validated_days applications... Using Northgate scraper. I, [2022-08-16T19:55:24.443053 #17] INFO -- : Form variables: {"csbtnSearch"=>"Search", "txtProposal"=>nil, "cboSelectDateValue"=>"DATE_VALID", "rbGroup"=>"rbRange", "dateStart"=>"2022-08-14", "dateEnd"=>"2022-08-16"} D, [2022-08-16T19:55:24.443109 #17] DEBUG -- : HTTP request headers: D, [2022-08-16T19:55:24.443129 #17] DEBUG -- : {"Origin"=>"http://northgate.liverpool.gov.uk", "Referer"=>"http://northgate.liverpool.gov.uk/PlanningExplorer17/GeneralSearch.aspx"} D, [2022-08-16T19:55:24.443142 #17] DEBUG -- : GET: http://northgate.liverpool.gov.uk/PlanningExplorer17/GeneralSearch.aspx D, [2022-08-16T19:55:25.387619 #17] DEBUG -- : Response code: HTTP 200 D, [2022-08-16T19:55:25.537091 #17] DEBUG -- : POST: http://northgate.liverpool.gov.uk/PlanningExplorer17/GeneralSearch.aspx D, [2022-08-16T19:55:27.206694 #17] DEBUG -- : Response code: HTTP 302 D, [2022-08-16T19:55:27.206777 #17] DEBUG -- : Location: /PlanningExplorer17/Generic/StdResults.aspx?PT=Planning Applications On-Line&SC=Date Validated is between 14 August 2022 and 16 August 2022&FT=Planning Application Search Results&XMLSIDE=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/Menus/PL.xml&XSLTemplate=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/xslt/PL/PLResults.xslt&PS=10&XMLLoc=/PlanningExplorer17/generic/XMLtemp/epx2xncu1abm5svwytogtxmv/0f3bfcd6-5097-4998-b317-16485db70a8e.xml D, [2022-08-16T19:55:27.206941 #17] DEBUG -- : GET: http://northgate.liverpool.gov.uk/PlanningExplorer17/Generic/StdResults.aspx?PT=Planning%20Applications%20On-Line&SC=Date%20Validated%20is%20between%2014%20August%202022%20and%2016%20August%202022&FT=Planning%20Application%20Search%20Results&XMLSIDE=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/Menus/PL.xml&XSLTemplate=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/xslt/PL/PLResults.xslt&PS=99999&XMLLoc=/PlanningExplorer17/generic/XMLtemp/epx2xncu1abm5svwytogtxmv/0f3bfcd6-5097-4998-b317-16485db70a8e.xml D, [2022-08-16T19:55:27.995333 #17] DEBUG -- : Response code: HTTP 200 I, [2022-08-16T19:55:28.009423 #17] INFO -- : Found -1 applications in search results. Liverpool: 0 application(s) saved. Checking for decided_days applications... Using Northgate scraper. /app/vendor/bundle/ruby/2.5.0/bundler/gems/uk_planning_scraper-624efc0ecf0a/lib/uk_planning_scraper/authority_scrape_params.rb:43: warning: constant ::Fixnum is deprecated I, [2022-08-16T19:55:28.009920 #17] INFO -- : Form variables: {"csbtnSearch"=>"Search", "txtProposal"=>nil, "cboSelectDateValue"=>"DATE_DECISION", "rbGroup"=>"rbRange", "dateStart"=>"2022-08-14", "dateEnd"=>"2022-08-16"} D, [2022-08-16T19:55:28.009943 #17] DEBUG -- : HTTP request headers: D, [2022-08-16T19:55:28.009956 #17] DEBUG -- : {"Origin"=>"http://northgate.liverpool.gov.uk", "Referer"=>"http://northgate.liverpool.gov.uk/PlanningExplorer17/GeneralSearch.aspx"} D, [2022-08-16T19:55:28.009965 #17] DEBUG -- : GET: http://northgate.liverpool.gov.uk/PlanningExplorer17/GeneralSearch.aspx D, [2022-08-16T19:55:28.389792 #17] DEBUG -- : Response code: HTTP 200 D, [2022-08-16T19:55:28.539503 #17] DEBUG -- : POST: http://northgate.liverpool.gov.uk/PlanningExplorer17/GeneralSearch.aspx D, [2022-08-16T19:55:29.972770 #17] DEBUG -- : Response code: HTTP 302 D, [2022-08-16T19:55:29.972864 #17] DEBUG -- : Location: /PlanningExplorer17/Generic/StdResults.aspx?PT=Planning Applications On-Line&SC=Decision Date is between 14 August 2022 and 16 August 2022&FT=Planning Application Search Results&XMLSIDE=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/Menus/PL.xml&XSLTemplate=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/xslt/PL/PLResults.xslt&PS=10&XMLLoc=/PlanningExplorer17/generic/XMLtemp/zjfoes5yjnvyleamoxawffj0/81ce27c1-5c8f-45fa-8978-3f470d66559d.xml D, [2022-08-16T19:55:29.972983 #17] DEBUG -- : GET: http://northgate.liverpool.gov.uk/PlanningExplorer17/Generic/StdResults.aspx?PT=Planning%20Applications%20On-Line&SC=Decision%20Date%20is%20between%2014%20August%202022%20and%2016%20August%202022&FT=Planning%20Application%20Search%20Results&XMLSIDE=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/Menus/PL.xml&XSLTemplate=/PlanningExplorer17/SiteFiles/Skins/Liverpool_WIP/xslt/PL/PLResults.xslt&PS=99999&XMLLoc=/PlanningExplorer17/generic/XMLtemp/zjfoes5yjnvyleamoxawffj0/81ce27c1-5c8f-45fa-8978-3f470d66559d.xml D, [2022-08-16T19:55:30.294832 #17] DEBUG -- : Response code: HTTP 200 I, [2022-08-16T19:55:30.296910 #17] INFO -- : Found -1 applications in search results. Liverpool: 0 application(s) saved. 4 of 6: Scraping Sefton /app/vendor/bundle/ruby/2.5.0/bundler/gems/uk_planning_scraper-624efc0ecf0a/lib/uk_planning_scraper/authority_scrape_params.rb:15: warning: constant ::Fixnum is deprecated Checking for validated_days applications... Using Idox scraper. Getting: https://pa.sefton.gov.uk/online-applications/search.do?action=advanced SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get local issuer certificate) Checking for decided_days applications... Using Idox scraper. Getting: https://pa.sefton.gov.uk/online-applications/search.do?action=advanced SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get local issuer certificate) 5 of 6: Scraping St. Helens Checking for validated_days applications... Using Idox scraper. Getting: https://publicaccess.sthelens.gov.uk/online-applications/search.do?action=advanced Found 3 apps on this page. 1 of 3: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RGNHT9PE02I00&activeTab=summary 2 of 3: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RFTNHMPEI4Q00&activeTab=summary 3 of 3: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RFPY74PEI3J00&activeTab=summary St. Helens: 3 application(s) saved. Checking for decided_days applications... Using Idox scraper. Getting: https://publicaccess.sthelens.gov.uk/online-applications/search.do?action=advanced Found 10 apps on this page. Getting: https://publicaccess.sthelens.gov.uk/online-applications/pagedSearchResults.do?action=page&searchCriteria.page=2 Found 1 apps on this page. 1 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=REJT9KPE02I00&activeTab=summary 2 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RDX4TNPEHM400&activeTab=summary 3 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RDVWGHPEHM000&activeTab=summary 4 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RDTFIEPEHKV00&activeTab=summary 5 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RDRYT6PEHKD00&activeTab=summary 6 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RDS4ECPEHKJ00&activeTab=summary 7 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RDIE9QPEHHS00&activeTab=summary 8 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RCHO5XPEH9K00&activeTab=summary 9 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=RC4VCZPE02J00&activeTab=summary 10 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=R98XCIPEGGE00&activeTab=summary 11 of 11: https://publicaccess.sthelens.gov.uk/online-applications/applicationDetails.do?keyVal=R7C69OPEFVZ00&activeTab=summary St. Helens: 11 application(s) saved. 6 of 6: Scraping Wirral Checking for validated_days applications... Using Idox scraper. Getting: https://planning.wirral.gov.uk/online-applications/search.do?action=advanced SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get local issuer certificate) Checking for decided_days applications... Using Idox scraper. Getting: https://planning.wirral.gov.uk/online-applications/search.do?action=advanced SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (unable to get local issuer certificate) Extracting postcodes...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Data

Downloaded 472 times by amcewen m410ne

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (24.3 MB) Use the API

rows 10 / 26178

scraped_at authority_name council_reference date_received date_validated status decision date_decision info_url address description documents_count documents_url alternative_reference appeal_status appeal_decision neighbourhood_postcode neighbourhood_latitude neighbourhood_longitude
2019-03-27T19:09:19
Sefton
DC/2019/00274
2019-02-08
Decided
Prior Approval Not Required
2019-03-21
84 Blundell Road Hightown Liverpool L38 9EQ
Prior approval submission for a proposed rear extension projecting 5.0 metres from the rear wall of the original dwellinghouse with a height of 2.67 metres at the eaves and a maximum height of 3.70 metres (Valid 08/02/2019)
7
Not Available
Unknown
Not Available
L38 9EQ
53.51795
-3.056965
2020-01-23T10:33:05
Sefton
DC/2019/02229
2019-11-26
Decided
Approve with Conditions
2020-01-17
38 Blundell Road Hightown Liverpool L38 9EQ
Erection of a two storey extension to the rear following demolition of the existing conservatory and part two storey part single storey extension to the side following removal of an existing dormer.
5
PP-08323944
Unknown
Not Available
L38 9EQ
53.51795
-3.056965
2019-06-13T00:05:01
Sefton
DC/2019/00571
2019-03-22
Decided
Approve with Conditions
2019-06-06
Land Adjacent To 16 Moorhouses Hightown Liverpool L38 9ER
Erection of a detached dwellinghouse with associated access and landscaping
16
PP-07391034
Unknown
Not Available
L38 9ER
53.521023
-3.06119
2019-02-19T20:42:05
Sefton
DC/2018/02265
2018-12-18
Decided
Approve with Conditions
2019-02-13
2 Old Acre Hightown Liverpool L38 9EW
Erection of a part two storey/ part single storey extension to the rear and single storey extensions to the front and side of the dwellinghouse
6
PP-07502299
Unknown
Not Available
L38 9EW
53.521204
-3.059007
2018-11-27T17:12:29
Sefton
DC/2018/00881
2018-05-10
Decided
Approve with Conditions
2018-11-21
Land Adjacent To Blundellsands Sailing Club Thornbeck Avenue Hightown Liverpool L38 9EY
Demolition of 37m of concrete outfall and timber piles; Erection of a new headwall, concrete apron and sailing basin with flap valve and grill; relocation of existing rock armour groyne and a temporary contractors compound (retrospective application)
17
PP-06061978
Unknown
Not Available
L38 9EY
53.518743
-3.05982
2020-01-23T10:29:10
Sefton
DC/2019/02322
2019-12-11
Decided
Approve
2020-01-17
22 Alton Close Hightown Liverpool L38 9GE
Approval of details reserved by conditions 3 and 4 attached to planning permission DC/2017/00116 approved on 17/03/2017
4
PP-08360880
Unknown
Not Available
L38 9GE
53.518412
-3.058545
2020-02-12T09:25:56
Sefton
DC/2019/02235
2019-12-06
Decided
Approve with Conditions
2020-02-06
32 Alton Close Hightown Liverpool L38 9GE
Erection of a first floor extension to the side of the dwellinghouse.
18
PP-08331057
Unknown
Not Available
L38 9GE
53.518412
-3.058545
2019-06-29T06:38:10
Sefton
DC/2019/00818
2019-04-29
Decided
Approve with Conditions
2019-06-24
6 Richmond Close Hightown Liverpool L38 9GH
Extension at 1st floor to side of house over existing room.
4
PP-07784129
Unknown
Not Available
L38 9GH
53.516158
-3.060419
2019-05-29T06:02:34
Sefton
DC/2019/00430
2019-03-19
Decided
Approve with Conditions
2019-05-23
7 Brentwood Close Hightown Liverpool L38 9GJ
Erection of a free standing first floor balcony to the rear of the dwellinghouse
9
IAP00035750-001
Unknown
Not Available
L38 9GJ
53.517151
-3.059856
2020-01-08T02:21:18
Liverpool
19F/2925
2019-11-12
FINAL DECISION
Approve with Conditions
117 Walton Breck Road Liverpool L4 0RD
Change of Use from A3 into A5 alterations to the shop front and installation of extraction ducting to the rear.
L4 0RD
53.430949
-2.966453

Statistics

Average successful run time: about 1 hour

Total run time: 3 months

Total cpu time used: about 2 hours

Total disk space used: 24.4 MB

History

  • Auto ran revision e82f2006 and completed successfully .
    1981 records added, 1972 records removed in the database
  • Auto ran revision e82f2006 and completed successfully .
    1972 records added, 1969 records removed in the database
  • Auto ran revision e82f2006 and completed successfully .
    1970 records added, 1970 records removed in the database
  • Auto ran revision e82f2006 and completed successfully .
    1974 records added, 1972 records removed in the database
  • Auto ran revision e82f2006 and completed successfully .
    1976 records added, 1970 records removed in the database
  • ...
  • Created on morph.io

Show complete history