rednerrus / noaa_historical_weather_data_csv_converter

NOAA Historical weather data CSV converter


NOAA publishes an amazing trove of historical weather data from stations all over the world – http://www.ncdc.noaa.gov/cgi-bin/res40.pl – but in a rather awkward format: each year is a separate folder, in which each station is one file, identified by two arbitrary ref numbers. Each file is gzipped text in a rather non-standard format with some inconvenient features like NULL tokens varying per field, and numbers being concatenated together with flags that give interpretation information: ftp://ftp.ncdc.noaa.gov/pub/data/gsod/readme.txt

This script lets the user specify a station and a number of years to download. It then iterates through, downloading enough years, parsing them into a much more standard CSV format read for use in (e.g.) Excel or Tableau, and concatenating them into one continuous file per station.

Here’s an example of what you can make with the output: http://eldan.co.uk/2012/10/rain-redux/

Forked from ScraperWiki

Contributors rednerrus

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... How many years of data would you like to download for each station? Traceback (most recent call last): File "scraper.py", line 64, in <module> maxyears = int(raw_input("How many years of data would you like to download for each station?\n")) EOFError: EOF when reading a line

Statistics

Total run time: 2 minutes

Total cpu time used: less than 10 seconds

Total disk space used: 46.8 KB

History

  • Manually ran revision 6fac418b and failed .
    nothing changed in the database
  • Manually ran revision 9ffb74c6 and failed .
    nothing changed in the database
  • Manually ran revision 70a8d86d and failed .
    nothing changed in the database
  • Manually ran revision 70a8d86d and failed .
    nothing changed in the database
  • Manually ran revision 70a8d86d and failed .
    nothing changed in the database
  • Forked from ScraperWiki