Wolframko / hltv

HLTV matches


This is a scraper that runs on Morph. To get started see the documentation

This scraper

Aims to collect CSGO matches from HLTV results page.

To get to know what's scraped see hltv.models.

License

This software is available under The MIT License, see the LICENSE file for more information.

Contributors miedzinski

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.9 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.  Collecting py-dateutil==2.2 (from -r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/73/30/e9e22e80fc592803edb23d89588ccaa184f14bb67d9c321f705edbfdd95b/py-dateutil-2.2.tar.gz (64kB)  Collecting Scrapy==1.0.3 (from -r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/ae/09/2c4395d5d0f0a2e6610d70d13cb6e2037050a61b79acd4180ccc9b5bf350/Scrapy-1.0.3-py2-none-any.whl (290kB)  Collecting SQLAlchemy==1.0.9 (from -r /tmp/build/requirements.txt (line 3))  Downloading https://files.pythonhosted.org/packages/54/70/7f9924a10611d1a21e7b77422364e5dce763d99aab7543747e1c4e632673/SQLAlchemy-1.0.9.tar.gz (4.7MB)  Collecting six (from py-dateutil==2.2->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl  Collecting lxml (from Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/81/e1/830a5ace40867959b22b83a99aa0c2f2768cf4cc5fe514e3a72e8c101deb/lxml-4.3.3-cp27-cp27m-manylinux1_x86_64.whl (5.6MB)  Collecting service-identity (from Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl  Collecting pyOpenSSL (from Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)  Collecting queuelib (from Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl  Collecting cssselect>=0.9 (from Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/7b/44/25b7283e50585f0b4156960691d951b05d061abf4a714078393e51929b30/cssselect-1.0.3-py2.py3-none-any.whl  Collecting Twisted>=10.0.0 (from Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/f8/2b/a80a70f71eb2b86992ffa5aaae41457791ae67faa70927fd16b76127c2b7/Twisted-19.2.0.tar.bz2 (3.1MB)  Collecting w3lib>=1.8.0 (from Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/81/43/9dcf92a77f5f0afe4f4df2407d7289eea01368a08b64bda00dd318ca62a6/w3lib-1.20.0-py2.py3-none-any.whl  Collecting pyasn1-modules (from service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/91/f0/b03e00ce9fddf4827c42df1c3ce10c74eadebfb706231e8d6d1c356a4062/pyasn1_modules-0.2.5-py2.py3-none-any.whl (74kB)  Collecting pyasn1 (from service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)  Collecting ipaddress; python_version < "3.3" (from service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl  Collecting cryptography (from service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/73/f3/84b8b24181a66f14de04497031a99b5e6f558d391dc8635cbfd03e5baee7/cryptography-2.6.1-cp27-cp27m-manylinux1_x86_64.whl (2.3MB)  Collecting attrs>=16.0.0 (from service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl  Collecting zope.interface>=4.4.2 (from Twisted>=10.0.0->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/b0/74/084cc7f111480c4ad60eecd02ea7830bd39259e29e40dc6a69fbed4b6f09/zope.interface-4.6.0-cp27-cp27m-manylinux1_x86_64.whl (164kB)  Collecting constantly>=15.1 (from Twisted>=10.0.0->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/b9/65/48c1909d0c0aeae6c10213340ce682db01b48ea900a7d9fce7a7910ff318/constantly-15.1.0-py2.py3-none-any.whl  Collecting incremental>=16.10.1 (from Twisted>=10.0.0->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/f5/1d/c98a587dc06e107115cf4a58b49de20b19222c83d75335a192052af4c4b7/incremental-17.5.0-py2.py3-none-any.whl  Collecting Automat>=0.3.0 (from Twisted>=10.0.0->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/a3/86/14c16bb98a5a3542ed8fed5d74fb064a902de3bdd98d6584b34553353c45/Automat-0.7.0-py2.py3-none-any.whl  Collecting hyperlink>=17.1.1 (from Twisted>=10.0.0->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/7f/91/e916ca10a2de1cb7101a9b24da546fb90ee14629e23160086cf3361c4fb8/hyperlink-19.0.0-py2.py3-none-any.whl  Collecting PyHamcrest>=1.9.0 (from Twisted>=10.0.0->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl (52kB)  Collecting enum34; python_version < "3" (from cryptography->service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl  Collecting asn1crypto>=0.21.0 (from cryptography->service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)  Collecting cffi!=1.11.3,>=1.8 (from cryptography->service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/32/47/94da1c6cba7d1352be11035133e5e0f17180df6e11f088642ecf9c48b156/cffi-1.12.3-cp27-cp27m-manylinux1_x86_64.whl (415kB)  Collecting idna>=2.5 (from hyperlink>=17.1.1->Twisted>=10.0.0->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)  Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->service-identity->Scrapy==1.0.3->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)  Building wheels for collected packages: py-dateutil, SQLAlchemy, Twisted, pycparser  Building wheel for py-dateutil (setup.py): started  Building wheel for py-dateutil (setup.py): finished with status 'done'  Stored in directory: /tmp/pip-ephem-wheel-cache-cHlgg8/wheels/8f/75/a0/567f85f33b24864ed7a16b3b9be16c31d593700b3c755cb3ad  Building wheel for SQLAlchemy (setup.py): started  Building wheel for SQLAlchemy (setup.py): finished with status 'done'  Stored in directory: /tmp/pip-ephem-wheel-cache-cHlgg8/wheels/2a/de/9f/49ccde938965f7f59d496b348f47139de9515a5fa0d4af194f  Building wheel for Twisted (setup.py): started  Building wheel for Twisted (setup.py): finished with status 'done'  Stored in directory: /tmp/pip-ephem-wheel-cache-cHlgg8/wheels/57/c0/6c/874eb1b6a5ab9ac5434ab9d2700cf78fed3c151ea33cf668af  Building wheel for pycparser (setup.py): started  Building wheel for pycparser (setup.py): finished with status 'done'  Stored in directory: /tmp/pip-ephem-wheel-cache-cHlgg8/wheels/f2/9a/90/de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511  Successfully built py-dateutil SQLAlchemy Twisted pycparser  Installing collected packages: six, py-dateutil, lxml, pyasn1, pyasn1-modules, ipaddress, enum34, asn1crypto, pycparser, cffi, cryptography, attrs, service-identity, pyOpenSSL, queuelib, cssselect, zope.interface, constantly, incremental, Automat, idna, hyperlink, PyHamcrest, Twisted, w3lib, Scrapy, SQLAlchemy  Successfully installed Automat-0.7.0 PyHamcrest-1.9.0 SQLAlchemy-1.0.9 Scrapy-1.0.3 Twisted-19.2.0 asn1crypto-0.24.0 attrs-19.1.0 cffi-1.12.3 constantly-15.1.0 cryptography-2.6.1 cssselect-1.0.3 enum34-1.1.6 hyperlink-19.0.0 idna-2.8 incremental-17.5.0 ipaddress-1.0.22 lxml-4.3.3 py-dateutil-2.2 pyOpenSSL-19.0.0 pyasn1-0.4.5 pyasn1-modules-0.2.5 pycparser-2.19 queuelib-1.5.0 service-identity-18.1.0 six-1.12.0 w3lib-1.20.0 zope.interface-4.6.0 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... 2019-05-03 15:27:50 [scrapy] ERROR: Error downloading <GET http://www.hltv.org/results/> Traceback (most recent call last): File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/utils/defer.py", line 45, in mustbe_deferred result = f(*args, **kw) File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 41, in download_request return handler(request, spider) File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 44, in download_request return agent.download_request(request) File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 211, in download_request d = agent.request(method, url, headers, bodyproducer) File "/app/.heroku/python/lib/python2.7/site-packages/twisted/web/client.py", line 1725, in request parsedURI.originForm) File "/app/.heroku/python/lib/python2.7/site-packages/twisted/web/client.py", line 1503, in _requestWithEndpoint d = self._pool.getConnection(key, endpoint) File "/app/.heroku/python/lib/python2.7/site-packages/twisted/web/client.py", line 1388, in getConnection return self._newConnection(key, endpoint) File "/app/.heroku/python/lib/python2.7/site-packages/twisted/web/client.py", line 1400, in _newConnection return endpoint.connect(factory) File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 924, in connect EndpointReceiver, self._hostText, portNumber=self._port File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/_resolver.py", line 189, in resolveHostName onAddress = self._simpleResolver.getHostByName(hostName) File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/resolver.py", line 21, in getHostByName d = super(CachingThreadedResolver, self).getHostByName(name, timeout) File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/base.py", line 282, in getHostByName timeoutDelay = sum(timeout) TypeError: 'float' object is not iterable

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (6 KB) Use the API

rows 0 / 0

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (6 KB) Use the API

rows 0 / 0

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (6 KB) Use the API

rows 0 / 0

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (6 KB) Use the API

rows 0 / 0

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (6 KB) Use the API

rows 0 / 0

Statistics

Average successful run time: 1 minute

Total run time: 1 minute

Total cpu time used: less than 5 seconds

Total disk space used: 43.8 KB

History

  • Manually ran revision c176fe24 and completed successfully .
    nothing changed in the database
  • Created on morph.io

Scraper code

Python

hltv / scraper.py