Solved – How to fetch all historically available hourly weather data

datasettime series

Where can I obtain all hourly weather data available?

Notes:

but am sure there's a better way.

  • I'd love per-minute/etc data too, though I don't think anyone kept
    measurements better than hourly back then.

Best Answer

This may be simplistic, but if you have a consistent directory structure on the NOAA site (they usually are), you can recursive wget the entire thing, then sort through it at your leisure.

wget -r http://weather.noaa.gov/pub/SL.us008001/DF.an/

This will grab everything recursively from that URL and deeper. It's what I normally use when I want a huge whack of data from some (typically government site) and they do silly things like store it in 1-file-per-hour and I need 12 years worth.

As an aside, I presume you only want historical weather data for the United States? If you're working on this on a slightly longer scale and more global, the Berkeley Earth Surface Temperature Project should be releasing their raw data set in the next few months: see here.