I think that the original poster may want to calculate for each cell the height above the first stream cell that would be reached by water flowing from the cell. So the 'nearest stream' is calculated along the downslope flow path, not euclidian distance. The references for this Height Above Nearest Drainage (HAND) are:
Rennó, C. D., Nobre, A. D., Cuartas, L. A., Soares, J. V., Hodnett, M.
G., Tomasella, J. and Waterloo, M. J. (2008) HAND, a new terrain
descriptor using SRTM-DEM: Mapping terra-firme rainforest environments
in Amazonia. Remote Sensing of Environment 112, 3469-3481.
Nobre, A. D., Cuartas, L. A., Hodnett, M., Rennó, C. D., Rodrigues,
G., Silveira, A., Waterloo, M. and Saleska, S. (2011) Height Above the
Nearest Drainage - a hydrologically relevant new terrain model. J.
Hydrol. 404, 13-29.
My rather kludgy implementation was:
1) Create a flow direction raster and flow accumulation raster from the DEM. The help files can walk you through this.
2) Create a stream raster from a flow accumulation raster by setting a threshold value for what is considered a stream (in my case 250 m^2). Merge with a sink raster because Australia is full of hydrologically relevant closed depressions. This is the drainage raster so set all values to a HAND of 0.
3) Calculate a raster of height above the next downslope cell for the entire area (flow elevation difference). This will be used in lots of iterations but only needs to be calculated once. You need to store text files for each irregular neighbourhood direction, e.g. for flow direction 64:
DirCode64.txt -
3 3
0 1 0
0 0 0
0 0 0
The calculation is performed once for each determinate value in the flow direction raster (powers of 2). For the odd-ball values that are not powers of 2, I used the height above the minimum elevation cell in the surrounding 3x3 neighbourhood.
4) Calculate the height above drainage iterating out from the drainage lines as:
flow elevation difference + HAND of next downslope cell.
This is performed once for each flow direction, added to the HAND raster, then iterating until the number of null cells in the HAND raster stops changing. I included an escape if the number of iterations got too high and save the output periodically so I can restart if/when the process crashes. The saving seems to be a slow step so I don't do it every iteration.
Hope this is clear enough. I'm sure the code could be cleaned up but I stopped when it was working. Thanks for help from other threads that pointed me in the right direction. Here is my code:
import arcpy
from arcpy import env
from arcpy.sa import *
arcpy.CheckOutExtension("Spatial")
env.workspace = "C:/Data/GIS_Data/DEM"
# Starting HAND raster with 0 for streams/sinks
outHandRaster = Raster("hnd20strsnk")
# use below for restarting iterations
#outHandRaster = Raster("hand20gh2")
inElevRaster = Raster("dtm20m")
inFDirRaster = Raster("fdir20m")
lstDirection = [1,2,4,8,16,32,64,128]
# Count the number of null cells in the initial stream raster
nullOutRaster = Con(IsNull(outHandRaster),1)
nullOutRaster.save("handNull")
cursor = arcpy.da.SearchCursor("handNull","Count")
nullCount = cursor.next()[0] # Count of null cells in outHandRaster
print "nullCount = ", nullCount
nullDif = 1 # anything but 0
# Calculation of floweldif raster – contains the elevation difference
# between each cell and the cell in the downslope direction.
# This block only needs to be calculated once for the area
for idx in lstDirection:
focalMaskFile = "C:/Data/GIS_Data/DEM/FocalStatNeighbor/" + "DirCode" + str(idx) + ".txt"
outElDifRaster = Con(inFDirRaster == idx, inElevRaster - FocalStatistics(inElevRaster, NbrIrregular(focalMaskFile), "MINIMUM"))
else:
# Calculate values for indeterminate flow direction cells
outElDifRaster = Con(inFDirRaster,inElevRaster - FocalStatistics(inElevRaster, NbrRectangle(3,3), "MINIMUM"))
outElDifRaster.save("floweldif")
# Iterative calculation of HAND raster
inElDiffRaster = Raster("floweldif")
maxIter = 100
i = 0 # iteration counter limits number of loops for testing
while nullDif != 0:
for idx in lstDirection:
focalMaskFile = "C:/Data/GIS_Data/DEM/FocalStatNeighbor/" + "DirCode" + str(idx) + ".txt"
outHandRaster = Con(inFDirRaster == idx,Con(IsNull(outHandRaster),inElDiffRaster + FocalStatistics(outHandRaster, NbrIrregular(focalMaskFile), "MAXIMUM"),outHandRaster), outHandRaster)
else:
# Calculate values for indeterminate flow direction cells
outHandRaster = Con(IsNull(outHandRaster),inElDiffRaster + FocalStatistics(outHandRaster, NbrRectangle(3,3), "MINIMUM"),outHandRaster)
i += 1
print(str(i) + " iterations complete")
if i % 5 == 0:
outHandRaster.save("hand20")
nullOutRaster = Con(IsNull(outHandRaster),1)
nullOutRaster.save("handNull")
cursor = arcpy.da.SearchCursor("handNull","Count")
newCount = cursor.next()[0]
print "nullcount = ", nullCount, "newCount = ", newCount
nullDif = nullCount - newCount
nullCount = newCount
print "nullDif = ", nullDif
cursor.reset()
if i >= maxIter:
outHandRaster.save("hand20gh2") #restart file
break
ArcMap uses the D8 routing algorithm for determining flow directions between cells, with no convergence exponent since it is a single direction algorithm, while the default in SAGA GIS (I believe - at least it is in mine) is MFD with the convergence you specify.
These methods differ and therefore will generate different catchment area results.
You could run SAGA GIS with the D8 algorithm (and a convergence = 1), and this should make the results match more closely, but I have the feeling that even then they will not be identical. Different implementations of the same algorithm (differences in the way they are coded up) should still lead to differences in the results.
Hope this helps.
Edit: See the comment from @WhiteBoxDev below as well.
Best Answer
Its possible the negatives are the result of real depressions, so it wouldn't be wrong to just reset any negative values to 0, IMO. Map up the location of the negatives and compare with your input DEM and a decent satellite image before you make a decision.
That aside, the key thing with HAND is that it traces a flowpath downhill to the nearest stream cell and then calculates the elevation difference. Avoiding a Euclidean calculation is the whole point.
It took me a while to track down the actual software that Renno et al used for HAND since they didn't bother quoting it in their paper ( >:-( ). Its a bit clunky, but it does the job. Start with the TerraHidro 0.5 console app from http://www.dpi.inpe.br/terrahidro/doku.php and follow the instructions to get d8 grid, contributing area, and flow accumulation rasters using a hydrologically corrected DEM, e.g.
then get Terraview 4.2.2 from http://www.dpi.inpe.br/terraview/php/dow.php?body=DowFiles and TerraHidro 0.4.2 from http://wiki.dpi.inpe.br/doku.php?id=download and install them sequentially. Now:
I'm recommending TerraHidro 0.5 for generating the inputs because it can handle larger datasets and is miles faster. It just doesn't have a HAND command.
You can get a very similar output to 'official' HAND by chaining r.stream.extract and r.stream.distance in GRASS 7. The bonus is that you can use dInf, not just d8 if you want, but it does produce some negatives, just as your procedure has.