wdiv-scrapers / DC-PollingStations-Dundee

Scrapes inspire.dundeecity.gov.uk

Under the INSPIRE Regulations Dundee City Council are obliged to publish information about the environment for use in Geographical Information Systems.  We continue to work with the Improvement Service and Scottish Government to identify data covered by the regulations, and have already made some data layers available;


Scrape polling station and district data from Dundee City Council

This is a scraper that runs on Morph. To get started see the documentation

Contributors chris48s

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.4.3  $ pip install -r requirements.txt  Collecting git+https://github.com/chris48s/arcgis2geojson.git (from -r requirements.txt (line 1))  Cloning https://github.com/chris48s/arcgis2geojson.git to /tmp/pip-ln5qfzds-build  Collecting git+https://github.com/wdiv-scrapers/dc-base-scrapers.git (from -r requirements.txt (line 2))  Cloning https://github.com/wdiv-scrapers/dc-base-scrapers.git to /tmp/pip-ubgwg_o8-build  Collecting lxml==3.6.4 (from dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading lxml-3.6.4-cp34-cp34m-manylinux1_x86_64.whl (4.2MB)  Collecting retry==0.9.2 (from dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading retry-0.9.2-py2.py3-none-any.whl  Collecting scraperwiki==0.5.1 (from dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading scraperwiki-0.5.1.tar.gz  Collecting py<2.0.0,>=1.4.26 (from retry==0.9.2->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading py-1.4.32-py2.py3-none-any.whl (82kB)  Collecting decorator>=3.4.2 (from retry==0.9.2->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading decorator-4.0.11-py2.py3-none-any.whl  Collecting requests (from scraperwiki==0.5.1->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading requests-2.12.4-py2.py3-none-any.whl (576kB)  Collecting six (from scraperwiki==0.5.1->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading six-1.10.0-py2.py3-none-any.whl  Collecting sqlalchemy (from scraperwiki==0.5.1->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading SQLAlchemy-1.1.4.tar.gz (5.1MB)  Collecting alembic (from scraperwiki==0.5.1->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading alembic-0.8.9.tar.gz (977kB)  Collecting Mako (from alembic->scraperwiki==0.5.1->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading Mako-1.0.6.tar.gz (575kB)  Collecting python-editor>=0.3 (from alembic->scraperwiki==0.5.1->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading python-editor-1.0.3.tar.gz  Collecting MarkupSafe>=0.9.2 (from Mako->alembic->scraperwiki==0.5.1->dc-base-scrapers==0.0.0->-r requirements.txt (line 2))  Downloading MarkupSafe-0.23.tar.gz  Installing collected packages: arcgis2geojson, lxml, py, decorator, retry, requests, six, sqlalchemy, MarkupSafe, Mako, python-editor, alembic, scraperwiki, dc-base-scrapers  Running setup.py install for arcgis2geojson: started  Running setup.py install for arcgis2geojson: finished with status 'done'  Running setup.py install for sqlalchemy: started  Running setup.py install for sqlalchemy: finished with status 'done'  Running setup.py install for MarkupSafe: started  Running setup.py install for MarkupSafe: finished with status 'done'  Running setup.py install for Mako: started  Running setup.py install for Mako: finished with status 'done'  Running setup.py install for python-editor: started  Running setup.py install for python-editor: finished with status 'done'  Running setup.py install for alembic: started  Running setup.py install for alembic: finished with status 'done'  Running setup.py install for scraperwiki: started  Running setup.py install for scraperwiki: finished with status 'done'  Running setup.py install for dc-base-scrapers: started  Running setup.py install for dc-base-scrapers: finished with status 'done'  Successfully installed Mako-1.0.6 MarkupSafe-0.23 alembic-0.8.9 arcgis2geojson-1.0.0 dc-base-scrapers-0.0.0 decorator-4.0.11 lxml-3.6.4 py-1.4.32 python-editor-1.0.3 requests-2.12.4 retry-0.9.2 scraperwiki-0.5.1 six-1.10.0 sqlalchemy-1.1.4   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... found 89 stations Traceback (most recent call last): File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 469, in do_execute cursor.execute(statement, parameters) sqlite3.DatabaseError: database disk image is malformed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "scraper.py", line 10, in <module> stations_scraper.scrape() File "/app/.heroku/python/lib/python3.4/site-packages/dc_base_scrapers/geojson_scraper.py", line 36, in scrape truncate(self.table) File "/app/.heroku/python/lib/python3.4/site-packages/dc_base_scrapers/common.py", line 11, in truncate scraperwiki.sqlite.execute("DROP TABLE IF EXISTS %s;" % table) File "/app/.heroku/python/lib/python3.4/site-packages/scraperwiki/sql.py", line 138, in execute connection = _State.connection() File "/app/.heroku/python/lib/python3.4/site-packages/scraperwiki/sql.py", line 79, in connection cls.reflect_metadata() File "/app/.heroku/python/lib/python3.4/site-packages/scraperwiki/sql.py", line 97, in reflect_metadata cls.metadata.reflect() File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/sql/schema.py", line 3773, in reflect bind.engine.table_names(schema, connection=conn)) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2119, in table_names return self.dialect.get_table_names(conn, schema) File "<string>", line 2, in get_table_names File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/reflection.py", line 42, in cache return fn(self, con, *args, **kw) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1162, in get_table_names rs = connection.execute(s) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 939, in execute return self._execute_text(object, multiparams, params) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_text statement, parameters File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception exc_info File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 185, in reraise raise value.with_traceback(tb) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/app/.heroku/python/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 469, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"]

Statistics

Average successful run time: 16 minutes

Total run time: about 1 month

Total cpu time used: 2 minutes

Total disk space used: 250 KB

History

  • Auto ran revision 60b6034e and failed .
    1 page scraped
  • Auto ran revision 60b6034e and failed .
  • Auto ran revision 60b6034e and failed .
    427 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    73 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    34 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    3 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    nothing changed in the database
    3 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 260 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    4 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    3 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    3 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 60b6034e and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Manually ran revision 60b6034e and completed successfully .
    180 records added in the database
    2 pages scraped
  • Auto ran revision e1411a9d and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision e1411a9d and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision e1411a9d and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision e1411a9d and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision e1411a9d and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Manually ran revision e1411a9d and completed successfully .
    180 records added in the database
    2 pages scraped
  • Auto ran revision f7090635 and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision f7090635 and completed successfully .
    2 records added, 178 records updated in the database
  • Auto ran revision f7090635 and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision f7090635 and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision f7090635 and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Manually ran revision f7090635 and completed successfully .
    180 records added in the database
    3 pages scraped
  • Auto ran revision 42c97a15 and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 42c97a15 and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 42c97a15 and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 42c97a15 and completed successfully .
    2 records added, 188 records updated in the database
  • Auto ran revision 42c97a15 and completed successfully .
    nothing changed in the database
    60 pages scraped
  • Auto ran revision 42c97a15 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 14df13bd and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 14df13bd and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 14df13bd and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 14df13bd and completed successfully .
    2 records added, 178 records updated in the database
    2 pages scraped
  • Auto ran revision 14df13bd and completed successfully .
    180 records added in the database
    2 pages scraped
  • Manually ran revision 14df13bd and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Manually ran revision 14df13bd and completed successfully .
    nothing changed in the database
  • Created on morph.io

Scraper code

Python

DC-PollingStations-Dundee / scraper.py