[GIS] Python crashes during large cursor process

arcgis-10.0python

In a current project, I am attempting to extract several fields for each record of a table in a GDB that has ~8.5 million records. My current strategy is to create a search cursor, fetch the data I want from each row, and pass it to a csv.writerow to write a line with the desired information. Unfortunately, after about 340k records, I get a message from Windows telling me Pythonw.exe has stopped working, and the process stops.

Why would this happen? Is there a better method for extracting large amounts of data to text? DBF isn't an option because these files could become larger than the 2GB limit.

Interestingly enough, after a few more tests, it appears that it always crashes on the 341051st record, and always at the same point on the line.

Edit Now attempting to split up the cursor to read only a subset of the data each time. It is apparently a memory issue. I created a list for each subset query eg: 

squery = ['OBJECTID >=0 AND OBJECTID <=300000',
'OBJECTID >=300001 AND OBJECTID <=600000']

And run the search cursor as an iteration: 

for query in squery:
    InRows = gp.SearchCursor(InFC,query)
    for row in InRows:
        val = row.val
        val2 = row.val2
        items = [val,val2]
        OutCsv.writerow(items)
    del row
    del InRows

Best Answer

So the problem is solved with some help from the kindly Mapperz and DavidF. I deleted the SearchCursor Object after each iteration of a subset of the input features, passing the subset to the csv filed using csv.writerow. Interestingly enough, I found that references to a string object do not require extra quotes to read as a single argument ( 'OBJECTID >=0 AND OBJECTID <=300000' vs. '"OBJECTID >=0 AND OBJECTID <=300000"'. At any rate, thank you Mapperz and DavidF for your help working through this!

Note: The above post features the code that eventually succeded.

Related Question