soit-sk / siemens-sk-utilities-scraper

data about utilities (public ligting, etc.) managed by Siemens in some Slovak municipalities

README.md

This is a scraper that runs on Morph. There you can also access data collected by the scraper.

It collects information about utilities (public lighting, etc.) managed by Siemens in some Slovak municipalities.

Current status: - only information about lighting is collected for now - ...

Contributors hanecak

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.13, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.13 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  Collecting lxml==3.4.4 (from -r /tmp/build/requirements.txt (line 8))  Downloading https://files.pythonhosted.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz (3.5MB)  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)  Collecting idna<2.9,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Building wheels for collected packages: lxml, dumptruck  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27mu-linux_x86_64.whl size=2987180 sha256=6de01ade07165bb8c3033797ca1820ff06c924d15ab027565970edcfc83a98ab  Stored in directory: /tmp/pip-ephem-wheel-cache-PdT7cM/wheels/f6/df/7b/af9cace9baf95a6e4a2b5790e30da55fc780ddee598314d1ed  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-cp27-none-any.whl size=11845 sha256=94051dde378dd3eaf5f0bbc94925ef7849fa460b8276f8d0a1b518a05d34ee5b  Stored in directory: /tmp/pip-ephem-wheel-cache-PdT7cM/wheels/57/df/83/32654ae89119876c7a7db66829bbdb646caa151589dbaf226e  Successfully built lxml dumptruck  Installing collected packages: dumptruck, certifi, urllib3, idna, chardet, requests, scraperwiki, lxml  Running setup.py develop for scraperwiki  Successfully installed certifi-2019.6.16 chardet-3.0.4 dumptruck-0.1.6 idna-2.8 lxml-3.4.4 requests-2.22.0 scraperwiki urllib3-1.25.3 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... ### connecting: done ### parsing response: Traceback (most recent call last): File "scraper.py", line 83, in <module> tree = lxml.etree.parse(response) File "lxml.etree.pyx", line 3310, in lxml.etree.parse (src/lxml/lxml.etree.c:72517) File "parser.pxi", line 1812, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:106204) File "parser.pxi", line 1832, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:106464) File "parser.pxi", line 1727, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:105354) File "parser.pxi", line 1146, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:100481) File "parser.pxi", line 580, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:94350) File "parser.pxi", line 690, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:95786) File "parser.pxi", line 620, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:94853) lxml.etree.XMLSyntaxError: Document is empty, line 1, column 1

Data

Downloaded 2 times by hanecak

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (7.3 MB) Use the API

rows 10 / 63501

position_lat position_lon aktivne scrap_time smid smcislo
17.585005
48.371367
true
2017-08-13T21:00:59
173939
100/005x
17.58551434
48.37117228
true
2017-08-13T21:01:01
90203
100/008x
17.591147
48.372266
true
2017-08-13T21:01:02
173969
100/022x
17.593191
48.372956
true
2017-08-13T21:01:03
173979
100/027x
17.585168
48.371562
true
2017-08-13T21:01:03
173940
100/055x
17.585450548
48.371133817
false
2017-08-13T21:01:46
90250
700/974
17.587186
48.371894
true
2017-08-13T21:01:53
173951
100/048x
17.583525
48.370882
true
2017-08-13T21:02:48
173928
100/061x
17.583793
48.371013
true
2017-08-13T21:02:48
173930
100/060x
17.584216
48.371033
true
2017-08-13T21:02:48
173933
100/002x

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (7.3 MB) Use the API

rows 1 / 1

value_blob type name
2018-11-12T09:11:08
text
last_run

Statistics

Average successful run time: 4 minutes

Total run time: 1 day

Total cpu time used: about 2 hours

Total disk space used: 7.34 MB

History

  • Auto ran revision 67fbd53a and failed .
    nothing changed in the database
  • Auto ran revision 67fbd53a and failed .
    nothing changed in the database
  • Auto ran revision 67fbd53a and failed .
    nothing changed in the database
  • Auto ran revision 67fbd53a and failed .
    nothing changed in the database
  • Auto ran revision 67fbd53a and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history