blablupcom / E0101_BNESC_gov

spending

Scrapes www.bathnes.gov.uk

Bathnes | Making Bath and North East Somerset an even better place to live, work and visit

README.md

This is a scraper that runs on Morph. To get started see the documentation

Contributors blablupcom

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... -----> Python app detected -----> Stack changed, re-installing runtime -----> Installing runtime (python-2.7.9) -----> Installing dependencies with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to morph_defaults) to ./.heroku/src/scraperwiki  Collecting lxml==3.4.4 (from -r requirements.txt (line 8))  Downloading lxml-3.4.4.tar.gz (3.5MB)  Building lxml version 3.4.4.  Building without Cython.  Using build configuration of libxslt 1.1.28  /app/.heroku/python/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'  warnings.warn(msg)  Collecting cssselect==0.9.1 (from -r requirements.txt (line 9))  Downloading cssselect-0.9.1.tar.gz  Collecting beautifulsoup4 (from -r requirements.txt (line 10))  Downloading beautifulsoup4-4.4.0-py2-none-any.whl (81kB)  Collecting python-dateutil (from -r requirements.txt (line 11))  Downloading python_dateutil-2.4.2-py2.py3-none-any.whl (188kB)  Collecting dumptruck>=0.1.2 (from scraperwiki->-r requirements.txt (line 6))  Downloading dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r requirements.txt (line 6))  Downloading requests-2.7.0-py2.py3-none-any.whl (470kB)  Collecting six>=1.5 (from python-dateutil->-r requirements.txt (line 11))  Downloading six-1.9.0-py2.py3-none-any.whl  Installing collected packages: six, requests, dumptruck, python-dateutil, beautifulsoup4, cssselect, lxml, scraperwiki    Running setup.py install for dumptruck    Running setup.py install for cssselect  Running setup.py install for lxml  Building lxml version 3.4.4.  Building without Cython.  Using build configuration of libxslt 1.1.28  /app/.heroku/python/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'  warnings.warn(msg)  building 'lxml.etree' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-OhtDA_/lxml/src/lxml/includes -I/app/.heroku/python/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w  gcc -pthread -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7/lxml/etree.so  building 'lxml.objectify' extension  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-OhtDA_/lxml/src/lxml/includes -I/app/.heroku/python/include/python2.7 -c src/lxml/lxml.objectify.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -w  gcc -pthread -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7/lxml/objectify.so  Running setup.py develop for scraperwiki  Creating /app/.heroku/python/lib/python2.7/site-packages/scraperwiki.egg-link (link to .)  Adding scraperwiki 0.3.7 to easy-install.pth file  Installed /app/.heroku/src/scraperwiki  Successfully installed beautifulsoup4-4.4.0 cssselect-0.9.1 dumptruck-0.1.6 lxml-3.4.4 python-dateutil-2.4.2 requests-2.7.0 scraperwiki six-1.9.0  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/.heroku/python/lib/python2.7/site-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently. To get rid of this warning, change this: BeautifulSoup([your markup]) to this: BeautifulSoup([your markup], "lxml") markup_type=markup_type)) E0101_BNESC_gov_2015_01 E0101_BNESC_gov_2015_02 E0101_BNESC_gov_2015_03 E0101_BNESC_gov_2014_01 E0101_BNESC_gov_2014_02 E0101_BNESC_gov_2014_03 E0101_BNESC_gov_2014_04 E0101_BNESC_gov_2014_05 E0101_BNESC_gov_2014_06 E0101_BNESC_gov_2014_07 E0101_BNESC_gov_2014_08 E0101_BNESC_gov_2014_09 E0101_BNESC_gov_2014_10 E0101_BNESC_gov_2014_11 E0101_BNESC_gov_2014_12 E0101_BNESC_gov_2013_01 E0101_BNESC_gov_2013_02 E0101_BNESC_gov_2013_03 E0101_BNESC_gov_2013_04 E0101_BNESC_gov_2013_05 E0101_BNESC_gov_2013_06 E0101_BNESC_gov_2013_07 E0101_BNESC_gov_2013_08 E0101_BNESC_gov_2013_09 E0101_BNESC_gov_2013_10 E0101_BNESC_gov_2013_11 E0101_BNESC_gov_2013_12 E0101_BNESC_gov_2012_12 E0101_BNESC_gov_2012_11 E0101_BNESC_gov_2012_10 E0101_BNESC_gov_2012_09 E0101_BNESC_gov_2012_08 E0101_BNESC_gov_2012_07 E0101_BNESC_gov_2012_06 E0101_BNESC_gov_2012_05 E0101_BNESC_gov_2012_04 E0101_BNESC_gov_2012_03 E0101_BNESC_gov_2012_02 E0101_BNESC_gov_2012_01 E0101_BNESC_gov_2011_12 E0101_BNESC_gov_2011_11 E0101_BNESC_gov_2011_10 E0101_BNESC_gov_2011_09 E0101_BNESC_gov_2011_08 E0101_BNESC_gov_2011_07 E0101_BNESC_gov_2011_06 E0101_BNESC_gov_2011_05 E0101_BNESC_gov_2011_04 E0101_BNESC_gov_2011_03 E0101_BNESC_gov_2011_02 E0101_BNESC_gov_2011_01 E0101_BNESC_gov_2010_12

Data

Downloaded 2 times by MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (25 KB) Use the API

rows 10 / 52

d l f
2015-07-13 19:56:56.999097
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_jan15.csv
E0101_BNESC_gov_2015_01
2015-07-13 19:57:02.730554
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_feb15.csv
E0101_BNESC_gov_2015_02
2015-07-13 19:57:09.971377
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_mar15.csv
E0101_BNESC_gov_2015_03
2015-07-13 19:57:15.199474
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_jan14.csv
E0101_BNESC_gov_2014_01
2015-07-13 19:57:19.365597
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_feb14.csv
E0101_BNESC_gov_2014_02
2015-07-13 19:57:23.476591
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_mar14.csv
E0101_BNESC_gov_2014_03
2015-07-13 19:57:27.633661
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_apr14.csv
E0101_BNESC_gov_2014_04
2015-07-13 19:57:32.331746
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_may14.csv
E0101_BNESC_gov_2014_05
2015-07-13 19:57:37.536449
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_jun14.csv
E0101_BNESC_gov_2014_06
2015-07-13 19:57:43.399829
http://www.bathnes.gov.uk/sites/default/files/sitedocuments/Your-Council/Data-Protection-and-freedom-of-information/spend_over_500_jul14.csv
E0101_BNESC_gov_2014_07

Statistics

Average successful run time: 8 minutes

Total run time: 8 minutes

Total cpu time used: less than 5 seconds

Total disk space used: 49.7 KB

History

  • Manually ran revision 56f908c6 and completed successfully .
    52 records added in the database
    53 pages scraped
  • Created on morph.io

Scraper code

Python

E0101_BNESC_gov / scraper.py