[GIS] “Error 4: xxxxx.dat: not recognized as a supported file format.”

gdalgdal-translatenoaa

I am trying to convert a .dat file to a .tif file using gdal_translate. The .dat file comes with an associated .Hdr file of the same name with the metadata (these are geospatial data from SNODAS). Here are the instructions:

Unzip and untar all data if you have not already:
gunzip *.gz should do the trick if you work in a linux environment. Then, tar -zxvf *.tar.
Using a text editor, create an ENVI header file with the following information for masked data¹ (files beginning with 'us'):

ENVI
samples = 6935
lines = 3351
bands = 1
header offset = 0
file type = ENVI Standard
data type = 2
interleave = bsq
byte order = 1

Save the header file using the exact same file name as the data file you are converting, but with a .hdr extension.

For example, the "us_ssmv01025SlL01T0024TTNATS2004010105DP001.dat" filename is used to create "us_ssmv01025SlL01T0024TTNATS2004010105DP001.hdr".

Ensure your working directory contains both the .hdr file and the .dat file, then issue the following commands:

GeoTIFF

gdal_translate -of GTiff -a_srs '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs' -a_nodata -9999 -a_ullr  -124.73333333 52.87500000 -66.94166667 24.95000000 <input.dat> <output.tif>

A common error that can be thrown after issuing the 'gdal_translate …' command is Error 4: not recognized as a supported file format. Depending on your system, GDAL might be confused between the .hdr you created and the .Hdr which came with the data. If this occurs, try storing the .Hdr files outside of the working directory and trying again.

¹For unmasked data, you will need to open the .Hdr files delivered with the data and adjust the ENVI header and bounding coordinates in the GDAL string accordingly.

Appendix 1. Conversion example for unmasked SNODAS data.

gdal_translate -of GTiff -a_srs '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs' -a_nodata -9999 -a_ullr -130.516666666661 58.2333333333310 -62.2499999999975 24.0999999999990 34.dat 34.tif

and header file should be:

ENVI
samples=8192
lines=4096
bands=1
header offset=0
file type=ENVI Standard
data type=2
interleave=bsq
byte order=1

So, here is my input code:

(gdal30) Lauras-iMac:SWE Casey$ gdal_translate -of GTiff -a_srs '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs' -a_nodata -9999 -a_ullr  -124.73333333 52.87500000 -66.94166667 24.95000000 us_ssmv11034tS__T0001TTNATS2018120805HP001.dat us_ssmv11034tS__T0001TTNATS2018120805HP001.tif

And here is the returned error:

ERROR 4: `us_ssmv11034tS__T0001TTNATS2018120805HP001.dat' not recognized as a supported file format.

Now, per the warning, I do in fact have the .Hdr file outside of the cwd, and only the .dat file and the .hdr file located within:

(gdal30) Lauras-iMac:SWE Casey$ ls
us_ssmv11034tS__T0001TTNATS2018120805HP001.dat  us_ssmv11034tS__T0001TTNATS2018120805HP001.hdr

Additionally, here is an overview of what I'm running:

(gdal30) Lauras-iMac:SWE Casey$ conda --version
conda 4.6.14
(gdal30) Lauras-iMac:SWE Casey$ gdalinfo --version
GDAL 3.0.0, released 2019/05/05
(gdal30) Lauras-iMac:SWE Casey$ python --version
Python 3.7.3

I performed this action pretty easily about a month ago, and have since then had a software update and am running MacOS Mojave 10.14.5. I was actually performing the task in a virtual environment in the terminal, but now have Anaconda installed. I have tried many different versions of python and gdal, completely wiped conda/python/gdal from my HD, all to no avail.

Also, gdal_translate does work for other actions, such as converting a geotif to a jpg:

(gdal30) Lauras-iMac:SNODAS_20181201 Casey$ gdal_translate -of JPEG -co QUALITY=70 -co PROGRESSIVE=ON -outsize 1400 0 -r bilinear CANYrelief1-geo.tif CANYrelief1-geo.jpg
Input file size is 2800, 2800
0...10...20...30...40...50...60...70...80...90...100 - done.

I'm a bit of a rookie and have two full days put into trying to figure this out. It's hard to understand how this worked a month ago with no problems but now refuses to work.

Here is the return from gdalinfo –formats :

GenBin -raster- (rov): Generic Binary (.hdr Labelled)
ENVI -raster- (rw+v): ENVI .hdr Labelled
EHdr -raster- (rw+v): ESRI .hdr Labelled

This is to confirm the Envi .hdr Labelled…

And…

(gdal30) Lauras-iMac:SWE Casey$ ls
34.dat  34.hdr
(gdal30) Lauras-iMac:SWE Casey$ gdalinfo 34.dat
ERROR 4: `34.dat' not recognized as a supported file format.
gdalinfo failed - unable to open '34.dat'.

Also, here is a link of the data source: ftp://sidads.colorado.edu/DATASETS/NOAA/G02158/masked/2018/12_Dec/

Each .tar file is a single day. The file contains 8 pairs, a .dat and a .Hdr file. For anyone who wishes to give it a try, just take this download and follow the given instructions.

My thought thus far is that GDAL changed, and that I'm not finding the appropriate steps to fix this.

Best Answer

I downloaded sample data and took one .dat/.Hdr pair from the archive. Then I had a try with GDAL 3.1.0dev on Windows. This version from the gisinternals comes with a specific driver

gdalinfo --formats |find "SNODAS"
SNODAS -raster- (rov): Snow Data Assimilation System

There is some documentation about the driver at https://gdal.org/drivers/raster/snodas.html

Everything seems to go smoothly with this driver when I just tell to use the .Hdr file as input.

gdalinfo  us_ssmv01025SlL00T0024TTNATS2018120105DP001.Hdr
Driver: SNODAS/Snow Data Assimilation System
Files: us_ssmv01025SlL00T0024TTNATS2018120105DP001.Hdr
       us_ssmv01025SlL00T0024TTNATS2018120105DP001.dat
Size is 6935, 3351
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-124.733333333332993,52.875000000000000)
Pixel Size = (0.008333333333333,-0.008333333333333)
Metadata:
  Data_Units=Kilograms per square meter / 10.000000
  Description=Scaled Non-snow accumulation, 24-hour total
  Stop_Date=2018/12/01 06:00:00
Corner Coordinates:
Upper Left  (-124.7333333,  52.8750000) (124d44' 0.00"W, 52d52'30.00"N)
Lower Left  (-124.7333333,  24.9500000) (124d44' 0.00"W, 24d57' 0.00"N)
Upper Right ( -66.9416667,  52.8750000) ( 66d56'30.00"W, 52d52'30.00"N)
Lower Right ( -66.9416667,  24.9500000) ( 66d56'30.00"W, 24d57' 0.00"N)
Center      ( -95.8375000,  38.9125000) ( 95d50'15.00"W, 38d54'45.00"N)
Band 1 Block=6935x1 Type=Int16, ColorInterp=Undefined
  Min=0.000 Max=892.000
  NoData Value=-9999



gdal_translate -of GTiff us_ssmv01025SlL00T0024TTNATS2018120105DP001.Hdr us_ssmv01025SlL00T0024TTNATS2018120105DP001.tif
Input file size is 6935, 3351
0...10...20...30...40...50...60...70...80...90...100 - done.

Related Solutions

[GIS] How to read/write ENVI metadata using GDAL

"Starting with GDAL 1.10..."

"I am using the Python bindings with GDAL 1.9.2..."

GDAL 1.10 hasn't been released yet. Beta 1 was released a short while ago or if you're using Windows, you can grab a build of the current trunk (1.10dev) from GISInternals.

If you're stuck with 1.9.2 for a while, here's some code to parse envi headers (envi.py)

Some more code that uses regexes to parse a header:

import re

def get_envi_header_dict(hdr):
    #Get all "key = {val}" type matches
    regex=re.compile(r'^(.+?)\s*=\s*({\s*.*?\n*.*?})$',re.M|re.I)
    matches=regex.findall(hdr)

    #Remove them from the header
    subhdr=regex.sub('',hdr)

    #Get all "key = val" type matches
    regex=re.compile(r'^(.+?)\s*=\s*(.*?)$',re.M|re.I)
    matches.extend(regex.findall(subhdr))

    return dict(matches)

if __name__=='__main__':
    hdr='''ENVI
description = {
  RPC Orthorectification Result [Mon Aug 13 13:38:09 2012] [Mon Aug 13
  13:38:09 2012]}
samples = 27856
lines   = 30016
bands   = 1
header offset = 0
file type = ENVI Standard
data type = 12
interleave = bsq
sensor type = WorldView
byte order = 0
map info = {UTM, 1.000, 1.000, 723000.000, 8129434.000, 5.0000000000e-001, 5.0000000000e-001, 55, South, WGS-84, units=Meters}
coordinate system string = {PROJCS["UTM_Zone_55S",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",10000000.0],PARAMETER["Central_Meridian",147.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]}
wavelength units = Micrometers
band names = {
 Orthorectified (Band 1)}
wavelength = {
 0.625000}
'''
    print get_envi_header_dict(hdr)

[GIS] Translating Modis HDF To GeoTIFF

I would recommend using the MODIS Reprojection Tool (MRT), which allows you to convert, reproject, and mosaic MODIS HDF files. Here's a screenshot:

MODIS reprojection tool

There's even a web-based version called MRTWeb so you don't have to download each of the individual HDF you want to process. Here's another screenshot:

MRTWeb

Best Answer

Related Solutions

[GIS] How to read/write ENVI metadata using GDAL

[GIS] Translating Modis HDF To GeoTIFF

Related Question