wdiv-scrapers / dc-meta-scraper

Yo dawg, I heard you like scrapers so I made a scraper to scrape your scrapers


Contributors chris48s

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.4.3  $ pip install -r requirements.txt  Collecting git+https://github.com/polling-bot-4000/polling-bot.git (from -r /tmp/build/requirements.txt (line 2))  Cloning https://github.com/polling-bot-4000/polling-bot.git to /tmp/pip-tijqx_y2-build  Collecting scraperwiki==0.5.1 (from -r /tmp/build/requirements.txt (line 1))  Downloading scraperwiki-0.5.1.tar.gz  Collecting requests<3,>=2 (from polling-bot==0.0.0->-r /tmp/build/requirements.txt (line 2))  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)  Collecting six (from scraperwiki==0.5.1->-r /tmp/build/requirements.txt (line 1))  Downloading six-1.11.0-py2.py3-none-any.whl  Collecting sqlalchemy (from scraperwiki==0.5.1->-r /tmp/build/requirements.txt (line 1))  Downloading SQLAlchemy-1.1.14.tar.gz (5.2MB)  Collecting alembic (from scraperwiki==0.5.1->-r /tmp/build/requirements.txt (line 1))  Downloading alembic-0.9.6.tar.gz (998kB)  Collecting idna<2.7,>=2.5 (from requests<3,>=2->polling-bot==0.0.0->-r /tmp/build/requirements.txt (line 2))  Downloading idna-2.6-py2.py3-none-any.whl (56kB)  Collecting urllib3<1.23,>=1.21.1 (from requests<3,>=2->polling-bot==0.0.0->-r /tmp/build/requirements.txt (line 2))  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests<3,>=2->polling-bot==0.0.0->-r /tmp/build/requirements.txt (line 2))  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting certifi>=2017.4.17 (from requests<3,>=2->polling-bot==0.0.0->-r /tmp/build/requirements.txt (line 2))  Downloading certifi-2017.7.27.1-py2.py3-none-any.whl (349kB)  Collecting Mako (from alembic->scraperwiki==0.5.1->-r /tmp/build/requirements.txt (line 1))  Downloading Mako-1.0.7.tar.gz (564kB)  Collecting python-editor>=0.3 (from alembic->scraperwiki==0.5.1->-r /tmp/build/requirements.txt (line 1))  Downloading python-editor-1.0.3.tar.gz  Collecting python-dateutil (from alembic->scraperwiki==0.5.1->-r /tmp/build/requirements.txt (line 1))  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)  Collecting MarkupSafe>=0.9.2 (from Mako->alembic->scraperwiki==0.5.1->-r /tmp/build/requirements.txt (line 1))  Downloading MarkupSafe-1.0.tar.gz  Installing collected packages: idna, urllib3, chardet, certifi, requests, six, sqlalchemy, MarkupSafe, Mako, python-editor, python-dateutil, alembic, scraperwiki, polling-bot  Running setup.py install for sqlalchemy: started  Running setup.py install for sqlalchemy: finished with status 'done'  Running setup.py install for MarkupSafe: started  Running setup.py install for MarkupSafe: finished with status 'done'  Running setup.py install for Mako: started  Running setup.py install for Mako: finished with status 'done'  Running setup.py install for python-editor: started  Running setup.py install for python-editor: finished with status 'done'  Running setup.py install for alembic: started  Running setup.py install for alembic: finished with status 'done'  Running setup.py install for scraperwiki: started  Running setup.py install for scraperwiki: finished with status 'done'  Running setup.py install for polling-bot: started  Running setup.py install for polling-bot: finished with status 'done'  Successfully installed Mako-1.0.7 MarkupSafe-1.0 alembic-0.9.6 certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 polling-bot-0.0.0 python-dateutil-2.6.1 python-editor-1.0.3 requests-2.18.4 scraperwiki-0.5.1 six-1.11.0 sqlalchemy-1.1.14 urllib3-1.22   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scraper: data Scraper: DC-PollingStations-Ashfield Scraper: DC-PollingStations-Barnet Scraper: DC-PollingStations-BarrowInFurness Scraper: DC-PollingStations-Bristol Scraper: DC-PollingStations-Calderdale Scraper: DC-PollingStations-Camden Scraper: DC-PollingStations-Canterbury Scraper: DC-PollingStations-Carmarthenshire Scraper: DC-PollingStations-CityOfLondon Scraper: DC-PollingStations-Daventry Scraper: DC-PollingStations-Doncaster Scraper: DC-PollingStations-Dover Scraper: DC-PollingStations-Dundee Scraper: DC-PollingStations-Ealing Scraper: DC-PollingStations-EastNorthamptonshire Scraper: DC-PollingStations-Eden Scraper: DC-PollingStations-Edinburgh Scraper: DC-PollingStations-EpsomAndEwell Scraper: DC-PollingStations-Erewash Scraper: DC-PollingStations-Fife Scraper: DC-PollingStations-Guildford Scraper: DC-PollingStations-Hackney Scraper: DC-PollingStations-Halton Scraper: DC-PollingStations-Hastings Scraper: DC-PollingStations-Kingston Scraper: DC-PollingStations-Lambeth Scraper: DC-PollingStations-Lancaster Scraper: DC-PollingStations-Leeds Scraper: DC-PollingStations-Lichfield Scraper: DC-PollingStations-Mid-Sussex Scraper: DC-PollingStations-Newham Scraper: DC-PollingStations-Pembrokeshire Scraper: DC-PollingStations-Portsmouth Scraper: DC-PollingStations-Runnymede Scraper: DC-PollingStations-Salford Scraper: DC-PollingStations-Sandwell Scraper: DC-PollingStations-Scarborough Scraper: DC-PollingStations-Shepway Scraper: DC-PollingStations-Southampton Scraper: DC-PollingStations-StAlbans Scraper: DC-PollingStations-StHelens Scraper: DC-PollingStations-Thanet Scraper: DC-PollingStations-TonbridgeMalling Scraper: DC-PollingStations-TunbridgeWells Scraper: DC-PollingStations-Uttlesford Scraper: DC-PollingStations-Wellingborough Scraper: DC-PollingStations-Westminster Scraper: DC-PollingStations-Wolverhampton Scraper: DC-PollingStations-York

Data

Downloaded 97 times by symroe wdiv-scrapers chris48s

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (17 KB) Use the API

rows 10 / 86

scraper started_polling changes entity council_id last_changed
DC-PollingStations-Ashfield
2016-12-18T11:39:51+00:00
2
stations
E07000170
2017-05-08T19:36:21+00:00
DC-PollingStations-Ashfield
2016-12-18T11:39:52+00:00
0
districts
E07000170
2016-12-18T11:39:52+00:00
DC-PollingStations-BarrowInFurness
2016-11-19T17:08:25+00:00
1
stations
E07000027
2017-03-27T19:28:43+00:00
DC-PollingStations-BarrowInFurness
2016-11-19T17:08:36+00:00
1
districts
E07000027
2017-03-27T19:28:50+00:00
DC-PollingStations-Bristol
2016-11-19T18:16:32+00:00
7
stations
E06000023
2017-06-08T12:11:23+00:00
DC-PollingStations-Bristol
2016-11-19T18:16:44+00:00
1
districts
E06000023
2017-06-16T19:50:37+00:00
DC-PollingStations-Calderdale
2017-05-12T16:46:50+00:00
3
stations
E08000033
2017-10-05T23:27:15+00:00
DC-PollingStations-Calderdale
2017-05-12T16:46:51+00:00
3
districts
E08000033
2017-10-05T23:27:16+00:00
DC-PollingStations-Camden
2016-11-19T17:13:43+00:00
4
stations
E09000007
2017-06-09T23:01:38+00:00
DC-PollingStations-Camden
2016-11-19T17:13:45+00:00
0
districts
E09000007
2016-11-19T17:13:45+00:00

Statistics

Average successful run time: 4 minutes

Total run time: about 21 hours

Total cpu time used: 20 minutes

Total disk space used: 44.2 KB

History

  • Auto ran revision 61aaac1d and completed successfully .
    86 records updated in the database
  • Auto ran revision 61aaac1d and completed successfully .
    86 records updated in the database
  • Auto ran revision 61aaac1d and completed successfully .
    86 records updated in the database
  • Auto ran revision 61aaac1d and completed successfully .
    86 records updated in the database
  • Auto ran revision 61aaac1d and completed successfully .
    86 records updated in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

dc-meta-scraper