[GIS] Out of memory when calculating loop in ArcGIS 10.2 with Python 2.7

arcgis-10.2arcpyloopmemory

I am quite new to scripting for ArcGIS, so don't mind the styling and layout.

I am trying to calculate 600.000 routes by looping a network analyst route analysis. The script extracts two pointID's from a database, selects their spatial component from a feature dataset based on this pointID, calculates the route and writes it to a layer file by using an append.

When I run the file (in ArcGIS Python Window), I get an out of memory error after 400+ files. When I examine the memory-use of the process, I can see a definite build-up until 2.5GB usage until the error.

This is my code:

# step 1: centroids geographical area
arcpy.AddMessage("Performing Feature to Point")
arcpy.FeatureToPoint_management(geographicalArea, outGeographicalAreaCentroids)
arcpy.MakeFeatureLayer_management(outGeographicalAreaCentroids, outGeographicalAreaCentroidsLyr)

# step 2: create new empty feature class to append routes into
arcpy.CreateFeatureclass_management(outLocation, routesFC, "POLYLINE", templateRoute)

# step 3: go through table and perform routing for each row
cursor = arcpy.SearchCursor(dbf)

for row in cursor:
    outRouteLayer = arcpy.na.MakeRouteLayer(networkDataset, outRouteLayerName, "Minutes", hierarchy = "USE_HIERARCHY")
    outRouteLayer = outRouteLayer.getOutput(0)
    subLayerNames = arcpy.na.GetNAClassNames(outRouteLayer)
    stopsLayerName = subLayerNames["Stops"]
    herkomst = row.getValue("Herkomstzo")
    expression1 = "\"Z\" = %d" % herkomst
    arcpy.SelectLayerByAttribute_management(outGeographicalAreaCentroidsLyr, "NEW_SELECTION", expression1)
    bestemming = row.getValue("Bestemming")
    expression2 = "\"Z\" = %d" % bestemming
    arcpy.SelectLayerByAttribute_management(outGeographicalAreaCentroidsLyr, "ADD_TO_SELECTION", expression2)
    arcpy.CopyFeatures_management(outGeographicalAreaCentroidsLyr, outGeographicalAreaCentroidsSelection)
    arcpy.MakeFeatureLayer_management(outGeographicalAreaCentroidsSelection, outGeographicalAreaCentroidsSelectionLyr)
    arcpy.na.AddLocations(outRouteLayer, stopsLayerName, outGeographicalAreaCentroidsSelectionLyr)
    if herkomst == bestemming:
        arcpy.AddMessage("Origin zone is equal to the destination zone")
    else:
        arcpy.na.Solve(outRouteLayer)
        outputRoute = "C:\Users\Koos\Documents\Werk\Onderzoek\Data\WORKSPACE\Routes\outRouteRoutes"
        arcpy.MakeFeatureLayer_management(r"route\Routes", outputRoute)
        arcpy.AddField_management(outputRoute, "Orig", "DOUBLE")
        arcpy.CalculateField_management(outputRoute, "Orig", herkomst)
        arcpy.AddField_management(outputRoute, "Dest", "DOUBLE")
        arcpy.CalculateField_management(outputRoute, "Dest", bestemming)
        arcpy.Append_management(outputRoute, routesFCLocation, "NO_TEST")
    # delete layers from memory
    arcpy.Delete_management(outRouteLayer)
    arcpy.Delete_management(herkomst)
    arcpy.Delete_management(expression1)
    arcpy.Delete_management(bestemming)
    arcpy.Delete_management(expression2)
    arcpy.Delete_management(outGeographicalAreaCentroidsSelection)
    arcpy.Delete_management(outGeographicalAreaCentroidsSelectionLyr)
    arcpy.Delete_management(outputRoute)
    del row
del cursor

There are more steps, but they are not relevant.

I read a lot of topics about the out of memory problems and memory leaks, etc. But I can't seem to solve this problem.

First I tried to remove every file made in the proces with the arcpy.Delete_management function. This seems to drop the memory use a little bit every step, but not enough, so the memory usage still builds up. Next, I tried to run my script in PythonWin and IDLE (thinking the memory issue would maybe not occur), but then I got errors saying the spatial network analyst (solve tool) is not licensed (a whole other issue).

Are there any suggestions as to how I can solve this issue and calculate the routes? I read on one topic that it is possible to extract the code in the loop to a seperate script and then summon this script in the loop, would this be an option? And if yes, how is this done?

Edits:
For a similar 'out of memory'-problem when calculating service areas, chunking the calculation has proved a valuable solution (the memory resets after every chunk). However, this still not resolves the routing issue, since chunking the route calculation will not help. Is there a possibility to partition the loop, thus resetting the memory after every 'loop-chunk'?

Code used to check license:

class LicenseError(Exception):
    pass

try:
    if arcpy.CheckExtension("Spatial") == "Available":
        arcpy.CheckOutExtension("Spatial")
        print "Checked out \"Spatial\" Extension"
    else:
        raise LicenseError
except LicenseError:
    print "Spatial license is unavailable"
except:
    print arcpy.GetMessages(2)

Code used to determine input variables (example):

inputMap = "C://Users//Koos//Documents//Werk//Data script//Inputs"
facility = inputMap + "//Totale_opvang_Vlaanderen.shp"
geographicalArea = inputMap + "//BASISZONERING_OVL.shp"
networkDataset = inputMap + "//belbel________nw_ND.nd"

Best Answer

Couple of general suggestions. Not really a Network Analyst expert, but hopefully these apply.

  • really look at what needs to within the looping structure. I'm probably not understanding all the logic in there, but it does seem like there are some variables and operations that would yield the same result with each iteration (particularly at the beginning of the loop).
  • is there a way to avoid using the Append operation inside your looping? This is a relatively expensive operation. In the past, I've created a numpy array within a loop and then used arcpy.da.numpyArrayToFeatureClass to ingest all of that information at once (see my response to another question for an example).
  • I'm guessing the arcpy.na.Solve operation is pretty expensive. Does this need to be solved iteratively or can it be done once all the arcpy.na.AddLocations operations are done? For that matter, if you collect all the locations within the loop, can the arcpy.na.AddLocations be done below the loop (could use numpy arrays, or even just a list of point geometries)?
  • is it necessary to create outGeographicalAreaCentroidsSelection? Can the selection on the original feature layer be used as input to AddLocations instead?
  • can the two SelectLayerByAttribute_management operations be merged into one?
Related Question