soit-sk / scraper-shmu-observations

Environment observation data from SHMU

README.md

SHMU Environment Observation Data

SHMU made available for DanubeHack 3.0 some environment observation data at:

http://meteo.shmu.sk/customer/home/opendata/

We can say it is a REST API with CSV payload. Thus this scraper is not "the usual one" (thanks to REST and CSV, its job would be much easier compared to other scrapers):

  1. crawl all data
  2. (optionally) keep local copy of CSV snapshots
  3. compile all snapshots into one SQLite database

Data

  • source: http://meteo.shmu.sk/customer/home/opendata/
  • data license: CC-BY
  • reference: https://data.gov.sk/dataset/atmosfericke-podmienky-a-meteorologicke-geograficke-prvky

Status

works

How to run

python3 -m venv .venv

source .venv/bin/activate

pip install -U pip # optional step

pip install -r requirements.txt

python scraper.py

License

Scraper is licensed under BSD license, see full text of license.

Scraper is on Morph

This is a scraper that runs on Morph. To get started see the documentation

TODO

  • ...

Contributors hanecak

Last run failed with status code 255.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Python app detected [1G-----> Installing python-2.7.14 [1G ! Requested runtime (python-2.7.14) is not available for this stack (cedar-14). [1G ! Aborting. More info: https://devcenter.heroku.com/articles/python-support

Data

Downloaded 13390 times by Petrus2929 MartinTuchyna klimeto designdnt MartinRobomaze cowjen01 hanecak adrian-tester janskulavik michalgeci heavyfrog775 gajdy valachy jozivaq11 mlandauer sgi320 xssxsr tetraedrit pnosko 07spider70 janikmilodar Sibyx Gogis0 simonsaisgtfo JakubCiesko Romnov8 michalfunket martinweb1 stkubica GreyWolfSk KathiSchleidt qweaq osgewo 100fino minazbc kovkus fragileqt Grolicek maroskomlbfm bullettime96 hablakan hemledra Dzankou franko14 MarekBasista

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (521 MB) Use the API

rows 1 / 1

value_blob type name
value_blob
2022-07-24T00:37:37
type
text
name
latest_observation

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (521 MB) Use the API

rows 10 / 2930205

ws_avg rh lat elev pa cccc scrap_time obs_stn name wd_avg date ta_2m lon pr_1h
ws_avg
1.6
rh
87.0
lat
48.167780
elev
283.0
pa
993.4
cccc
ASBA
scrap_time
2017-12-17T04:53:10
obs_stn
11:11813
name
BRATISLAVA - KOLIBA
wd_avg
51.2
date
2016-01-01T00:00:00Z
ta_2m
-5.3
lon
17.105830
pr_1h
0.00
ws_avg
1.5
rh
81.0
lat
48.170280
elev
128.0
pa
1013.3
cccc
ASIB
scrap_time
2017-12-17T04:53:10
obs_stn
11:11816
name
BRATISLAVA - LETISKO
wd_avg
60
date
2016-01-01T00:00:00Z
ta_2m
-5.3
lon
17.207500
pr_1h
0.00
ws_avg
1
rh
95.3
lat
48.486700
elev
176.0
pa
1008.0
cccc
ASJB
scrap_time
2017-12-17T04:53:10
obs_stn
11:11819
name
JASLOVSKE BOHUNICE
wd_avg
111
date
2016-01-01T00:00:00Z
ta_2m
-3.9
lon
17.670800
pr_1h
0.00
ws_avg
1.1
rh
62.0
lat
48.613100
elev
163.0
pa
1009.6
cccc
ASPP
scrap_time
2017-12-17T04:53:10
obs_stn
11:11826
name
PIESTANY
wd_avg
281
date
2016-01-01T00:00:00Z
ta_2m
-3.1
lon
17.832800
pr_1h
0.00
ws_avg
0.7
rh
86.0
lat
49.231900
elev
309.0
pa
991.6
cccc
ASZI
scrap_time
2017-12-17T04:53:10
obs_stn
11:11841
name
ZILINA - DOLNY HRICOV
wd_avg
101
date
2016-01-01T00:00:00Z
ta_2m
-10.0
lon
18.617800
pr_1h
0.00
ws_avg
rh
lat
48.280560
elev
132.0
pa
cccc
ASNI
scrap_time
2017-12-17T04:53:10
obs_stn
11:11855
name
NITRA
wd_avg
date
2016-01-01T00:00:00Z
ta_2m
lon
18.135560
pr_1h
ws_avg
1.1
rh
100.0
lat
48.289400
elev
261.0
pa
997.8
cccc
ASMO
scrap_time
2017-12-17T04:53:10
obs_stn
11:11856
name
MOCHOVCE
wd_avg
81
date
2016-01-01T00:00:00Z
ta_2m
-8.2
lon
18.456100
pr_1h
0.00
ws_avg
rh
lat
47.872500
elev
115.0
pa
cccc
ASHU
scrap_time
2017-12-17T04:53:10
obs_stn
11:11858
name
HURBANOVO
wd_avg
date
2016-01-01T00:00:00Z
ta_2m
lon
18.193060
pr_1h
ws_avg
1
rh
89.9
lat
48.767500
elev
256.0
pa
998.1
cccc
ASPE
scrap_time
2017-12-17T04:53:10
obs_stn
11:11867
name
PRIEVIDZA
wd_avg
133.9
date
2016-01-01T00:00:00Z
ta_2m
-9.1
lon
18.593060
pr_1h
0.00
ws_avg
1
rh
98.2
lat
48.169200
elev
139.0
pa
1013.7
cccc
ASDU
scrap_time
2017-12-17T04:53:10
obs_stn
11:11880
name
DUDINCE
wd_avg
98
date
2016-01-01T00:00:00Z
ta_2m
-9.4
lon
18.876100
pr_1h
0.00

Statistics

Average successful run time: 10 minutes

Total run time: 12 days

Total cpu time used: about 2 hours

Total disk space used: 521 MB

History

  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • Auto ran revision c4174e0f and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

scraper-shmu-observations / scraper.py