I have over 180,000 shapefiles that I have to merge (or append) into a final shapefile.
I have written a python script to do so which uses the acrpy.append_management tool. However the runtime is too long.
I was thinking of maybe testing out the merge tool but I am skeptical that it will accept this many shapefiles and merge them together in a reasonable amount of time.
Does anyone have any experience in merging large datasets?
Is there an efficient way to merge large amounts of shapefiles?
Best Answer
Consider using Dissolve. This will write the output to a new feature class but it may be more efficient. In my experience dissolve operations have taken less time than merge. Run a benchmark to determine which is faster.
There may be some issue with merge/dissolve wrt to performance and dissolving circular arcs (such as results of buffering points, polylines). Densifying input geometry prior to running merge/dissolve may increase performance.
You may also want to consider using software other that ESRI. Some users experience increase persomrmace using Postgres for example (see linked threads)
Measuring performance difference between Merge and Append in ArcGIS Desktop?
Improving upon ArcGIS Buffer/Dissolve performance?