This is a scraper that runs on Morph. To get started see the documentation

Contributors woodbine blablupcom

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 1))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:387: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  SNIMissingWarning,  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:142: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Collecting lxml==3.4.4  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:142: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning,  Downloading lxml-3.4.4.tar.gz (3.5 MB)  Collecting cssselect==0.9.1  Downloading cssselect-0.9.1.tar.gz (32 kB)  Collecting beautifulsoup4  Downloading beautifulsoup4-4.9.1-py2-none-any.whl (111 kB)  Collecting requests[security]  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting soupsieve<2.0  Downloading soupsieve-1.9.6-py2.py3-none-any.whl (33 kB)  Collecting idna<3,>=2.5  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)  Collecting chardet<4,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)  Collecting cryptography>=1.3.4; extra == "security"  Downloading cryptography-3.0-cp27-cp27m-manylinux2010_x86_64.whl (2.7 MB)  Collecting pyOpenSSL>=0.14; extra == "security"  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)  Collecting backports.functools-lru-cache; python_version < "3"  Downloading backports.functools_lru_cache-1.6.1-py2.py3-none-any.whl (5.7 kB)  Collecting cffi!=1.11.3,>=1.8  Downloading cffi-1.14.1-cp27-cp27m-manylinux1_x86_64.whl (388 kB)  Collecting enum34; python_version < "3"  Downloading enum34-1.1.10-py2-none-any.whl (11 kB)  Collecting six>=1.4.1  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)  Collecting ipaddress; python_version < "3"  Downloading ipaddress-1.0.23-py2.py3-none-any.whl (18 kB)  Collecting pycparser  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)  Building wheels for collected packages: lxml, cssselect, dumptruck  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27m-linux_x86_64.whl size=2989850 sha256=c4568548b0d6ecabe9b3cd44a65ccaea8d492fe3e769798c0ac0181865e96b1a  Stored in directory: /tmp/pip-ephem-wheel-cache-no2BD3/wheels/d6/de/81/11ae6edd05c75aac677e67dd154c85da758ba6f3e8e80e962e  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-py2-none-any.whl size=26993 sha256=0f4955622ac355ade60a151cd22be966b723f372e9e21a46133689859fb95a6a  Stored in directory: /tmp/pip-ephem-wheel-cache-no2BD3/wheels/85/fe/00/b94036d8583cec9791d8cda24c184f2d2ac1397822f7f0e8d4  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11843 sha256=c722beebcc222d105c824b4a2fca0696777504613812ac66dd74c9923718f484  Stored in directory: /tmp/pip-ephem-wheel-cache-no2BD3/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Successfully built lxml cssselect dumptruck  Installing collected packages: dumptruck, idna, chardet, urllib3, certifi, pycparser, cffi, enum34, six, ipaddress, cryptography, pyOpenSSL, requests, scraperwiki, lxml, cssselect, backports.functools-lru-cache, soupsieve, beautifulsoup4  Running setup.py develop for scraperwiki  Successfully installed backports.functools-lru-cache-1.6.1 beautifulsoup4-4.9.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 cryptography-3.0 cssselect-0.9.1 dumptruck-0.1.6 enum34-1.1.10 idna-2.10 ipaddress-1.0.23 lxml-3.4.4 pyOpenSSL-19.1.0 pycparser-2.20 requests-2.24.0 scraperwiki six-1.15.0 soupsieve-1.9.6 urllib3-1.25.10   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/.heroku/python/lib/python2.7/site-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release. CryptographyDeprecationWarning, E5037_ELBC_gov_2020_05 Error validating URL. E5037_ELBC_gov_2020_04 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/monthly-transactions-over-250---april-2020-business-and-licensing.xlsx E5037_ELBC_gov_2020_03 Error validating URL. E5037_ELBC_gov_2020_02 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-feb-2020-business-and-licensing.xlsx Error validating URL. E5037_ELBC_gov_2020_01 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-jan-2020-business-and-licensing.xlsx E5037_ELBC_gov_2019_12 Error validating URL. E5037_ELBC_gov_2019_11 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-nov-19-business-and-licensing.xlsx Error validating URL. E5037_ELBC_gov_2019_10 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-oct-19-business-and-licensing.xlsx E5037_ELBC_gov_2019_09 Error validating URL. E5037_ELBC_gov_2019_08 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-aug-19-business-and-licensing.xlsx E5037_ELBC_gov_2019_07 Error validating URL. E5037_ELBC_gov_2019_06 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-june-19-business-and-licensing.xlsx E5037_ELBC_gov_2019_05 Error validating URL. E5037_ELBC_gov_2019_04 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-april-19-business-and-licensing.xlsx Error validating URL. E5037_ELBC_gov_2019_03 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-march-19-business-and-licensing.xlsx Error validating URL. E5037_ELBC_gov_2019_02 *Error: Invalid URL* https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-february-19-business-and-licensing.xlsx Traceback (most recent call last): Error validating URL. File "scraper.py", line 133, in <module> E5037_ELBC_gov_2019_01 *Error: Invalid URL* raise Exception("%d errors occurred during scrape." % errors) Exception: 11 errors occurred during scrape. https://new.enfield.gov.uk/services/business-and-licensing/transparency-reports/monthly-report-for-transactions-over-250/monthly-transactions-over-250-january-19-business-and-licensing.xlsx

Data

Downloaded 972 times by SimKennedy woodbine MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (98 KB) Use the API

rows 10 / 106

d f l
2018-11-06 02:17:22.420368
E5037_ELBC_gov_2018_05
2018-11-06 02:17:25.192999
E5037_ELBC_gov_2018_04
2018-11-06 02:17:27.844237
E5037_ELBC_gov_2018_03
2018-11-06 02:17:33.062551
E5037_ELBC_gov_2018_01
2018-11-06 02:17:38.362668
E5037_ELBC_gov_2017_11
2018-11-06 02:17:43.337277
E5037_ELBC_gov_2017_10
2018-11-06 02:17:45.904206
E5037_ELBC_gov_2017_09
2018-11-06 02:17:51.374071
E5037_ELBC_gov_2017_08
2018-11-06 02:17:54.147686
E5037_ELBC_gov_2017_07
2018-11-06 02:18:03.601167
E5037_ELBC_gov_2017_05

Statistics

Average successful run time: 2 minutes

Total run time: 17 days

Total cpu time used: about 1 hour

Total disk space used: 127 KB

History

  • Auto ran revision 2873b8f0 and failed .
    6 records added, 6 records removed in the database
  • Auto ran revision 2873b8f0 and completed successfully .
    17 records added, 17 records removed in the database
  • Auto ran revision 2873b8f0 and completed successfully .
    17 records added, 17 records removed in the database
  • Auto ran revision 2873b8f0 and completed successfully .
    17 records added, 17 records removed in the database
  • Auto ran revision 2873b8f0 and completed successfully .
    17 records added, 17 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

sp_E5037_ELBC_gov / scraper.py