soit-sk / scraper-shmu-observations

Environment observation data from SHMU

README.md

SHMU Environment Observation Data

SHMU made available for DanubeHack 3.0 some environment observation data at:

http://meteo.shmu.sk/customer/home/opendata/

We can say it is a REST API with CSV payload. Thus this scraper is not "the usual one" (thanks to REST and CSV, its job would be much easier compared to other scrapers):

  1. crawl all data
  2. (optionally) keep local copy of CSV snapshots
  3. compile all snapshots into one SQLite database

Data

  • source: http://meteo.shmu.sk/customer/home/opendata/
  • data license: CC-BY
  • reference: https://data.gov.sk/dataset/atmosfericke-podmienky-a-meteorologicke-geograficke-prvky

Status

works

How to run

python3 -m venv .venv

source .venv/bin/activate

pip install -U pip # optional step

pip install -r requirements.txt

python scraper.py

License

Scraper is licensed under BSD license, see full text of license.

Scraper is on Morph

This is a scraper that runs on Morph. To get started see the documentation

TODO

  • ...

Contributors hanecak

Last run failed with status code 255.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Python app detected [1G-----> Installing python-2.7.14 [1G ! Requested runtime (python-2.7.14) is not available for this stack (cedar-14). [1G ! Aborting. More info: https://devcenter.heroku.com/articles/python-support

Data

Downloaded 12874 times by Petrus2929 MartinTuchyna klimeto designdnt MartinRobomaze cowjen01 hanecak adrian-tester heavyfrog775 janskulavik michalgeci gajdy valachy jozivaq11 mlandauer sgi320 tetraedrit pnosko 07spider70 janikmilodar Sibyx Gogis0 simonsaisgtfo JakubCiesko minazbc kovkus fragileqt Grolicek maroskomlbfm bullettime96 hablakan hemledra Dzankou franko14 MarekBasista Romnov8 michalfunket martinweb1 stkubica GreyWolfSk KathiSchleidt qweaq osgewo 100fino

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (521 MB) Use the API

rows 1 / 1

value_blob type name
2022-07-24T00:37:37
text
latest_observation

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (521 MB) Use the API

rows 10 / 2930205

ws_avg rh lat elev pa cccc scrap_time obs_stn name wd_avg date ta_2m lon pr_1h
1.6
87.0
48.167780
283.0
993.4
ASBA
2017-12-17T04:53:10
11:11813
BRATISLAVA - KOLIBA
51.2
2016-01-01T00:00:00Z
-5.3
17.105830
0.00
1.5
81.0
48.170280
128.0
1013.3
ASIB
2017-12-17T04:53:10
11:11816
BRATISLAVA - LETISKO
60
2016-01-01T00:00:00Z
-5.3
17.207500
0.00
1
95.3
48.486700
176.0
1008.0
ASJB
2017-12-17T04:53:10
11:11819
JASLOVSKE BOHUNICE
111
2016-01-01T00:00:00Z
-3.9
17.670800
0.00
1.1
62.0
48.613100
163.0
1009.6
ASPP
2017-12-17T04:53:10
11:11826
PIESTANY
281
2016-01-01T00:00:00Z
-3.1
17.832800
0.00
0.7
86.0
49.231900
309.0
991.6
ASZI
2017-12-17T04:53:10
11:11841
ZILINA - DOLNY HRICOV
101
2016-01-01T00:00:00Z
-10.0
18.617800
0.00
48.280560
132.0
ASNI
2017-12-17T04:53:10
11:11855
NITRA
2016-01-01T00:00:00Z
18.135560
1.1
100.0
48.289400
261.0
997.8
ASMO
2017-12-17T04:53:10
11:11856
MOCHOVCE
81
2016-01-01T00:00:00Z
-8.2
18.456100
0.00
47.872500
115.0
ASHU
2017-12-17T04:53:10
11:11858
HURBANOVO
2016-01-01T00:00:00Z
18.193060
1
89.9
48.767500
256.0
998.1
ASPE
2017-12-17T04:53:10
11:11867
PRIEVIDZA
133.9
2016-01-01T00:00:00Z
-9.1
18.593060
0.00
1
98.2
48.169200
139.0
1013.7
ASDU
2017-12-17T04:53:10
11:11880
DUDINCE
98
2016-01-01T00:00:00Z
-9.4
18.876100
0.00

Statistics

Average successful run time: 10 minutes

Total run time: 12 days

Total cpu time used: about 2 hours

Total disk space used: 521 MB

History

  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

scraper-shmu-observations / scraper.py