[GIS] Merge all shp files in a folder into one with a new field populated with the source filename

pythonqgis

Want to merge all shp files in a folder into one with a new field populated with the source filename. Not getting any errors but not getting any outputs.

Code originally taken from (How to add field with filename when merging shapefiles with ogr2ogr?)

# merge_shps.py
import os    
path = "C:\TEMP\SWS_SUDS\Cadastral_Maps"  # path to your folder of .shp files
merge = "SWS_INSPIRE"                         # this will be the name of your merged result
directory = os.listdir(path)
count = 0
for filename in directory:
    print "\n" + filename + "\n"
    if ".SHP" in filename.upper() and not ".XML" in filename.upper():
        # On the first pass, create a clone and add the filename column.
        if count == 0:
             # Make a clone (matt wilkie)..
             cmd = 'ogr2ogr ' + path + '\\' + merge + '.shp ' + path + '\\' + filename + ' -where "FID < 0"'
             print "\n" + cmd + "\n"
             os.system(cmd)
             # Add the field (j03lar50n)..
             cmd = 'ogrinfo ' + path + '\\' + merge + '.shp -sql "ALTER TABLE ' + merge + ' ADD COLUMN filename character(50)"'
             print "\n" + cmd + "\n"
             os.system(cmd)
        # Now populate the data (capooti)..
        print "Merging: " + str(filename)
        # You'll need the filename without the .shp extension for the OGR_SQL..
        filenameNoExt = filename.replace(".shp","")
        cmd = 'ogr2ogr -f "esri shapefile" -update -append "' + \
         path + '\\' + merge + '.shp" "' + \
         path + '\\' + filename +'"' \
         ' -sql "SELECT \'' + filename + '\' AS filename, * FROM ' + filenameNoExt + '"'
        # Uncomment this line to spit the ogr2ogr sentence to the terminal..
        print "\n" + cmd + "\n"
        os.system(cmd)
        count += 1

Best Answer

I'd suggest you rather go with ogr2ogr in a terminal script directly.

In summary (using the syntax from the linked post), to merge all .shp into merged.shp (both in CWD), with the filename (without extension) added as a column, run from within

  • Bash (Linux):

    for file in *.shp
do
  if [ -f  merged.shp ]
    then
      ogr2ogr -f "ESRI Shapefile" merged.shp $file -update -append -dialect "SQLite" -sql "SELECT '${file%.*}' AS filename, * FROM ${file%.*}"
    else
      ogr2ogr -f "ESRI Shapefile" merged.shp $file -dialect "SQLite" -sql "SELECT '${file%.*}' AS filename, * FROM ${file%.*}"
  fi
done

  • CMD (Windows Command Line):

    for %F in (*.shp) do (
  if not exists merged.shp (
    ogr2ogr -f "ESRI Shapefile" merged.shp %F -dialect "SQLite" -sql "SELECT '%~nF' AS filename, * FROM %~nF"
  ) else (
    ogr2ogr -f "ESRI Shapefile" merged.shp %F -update -append -dialect "SQLite" -sql "SELECT '%~nF' AS filename, * FROM %~nF"
  )
)

And to add up to this, there's actually no need to catch the non-existing file case, ogr2ogr will (at least in recent versions) create the file even in -append mode:

  • Bash (Linux):

    for file in *.shp
do
  ogr2ogr -f "ESRI Shapefile" merged.shp $file -update -append -dialect "SQLite" -sql "SELECT '${file%.*}' AS filename, * FROM ${file%.*}"
done

  • CMD (Windows Command Line):

    for %F in (*.shp) do (
  ogr2ogr -f "ESRI Shapefile" merged.shp %F -update -append -dialect "SQLite" -sql "SELECT '%~nF' AS filename, * FROM %~nF"
)
Related Question