PyQGIS QGIS 3 – Running Processing Algorithms on Multiple Input Files

pyqgisqgis-3qgis-processing

I'm new to QGIS(3.20.2), and have even less experience with python. I've been working on automating a simple task and have been having difficulty tying together the pieces of information that I've found on the forum. What I'm looking to do is create a standalone script that runs without opening QGIS, takes all the text files from a specified folder, and creates output polygon files from each of these text files and places them in another specified folder. So far, I've mostly been able to accomplish this task within the python console of QGIS, but am having trouble looping the processing algorithms so they run on all the files in a folder, and running this script outside of QGIS. My script so far does the following:

Creates delimited text layers from all the text files in a folder and adds them to the map. The text file format is shown below:

ID	X	Y	Sort
1	-77.1	36.4	1
2	-77.1	36.3	1
3	-77.09	36.2	2
4	-77.09	36.3	2

Runs the Point to Path algorithm on an individual input file using the start and end xy coordinates, which has field_4 (sort) as the path group expression.
Runs the Polygonize algorithm on the input file from the previous step and exports it to a .kml file in a specified folder.

The code that I'm using to do this is as follows:

import os.path, glob
from qgis.core import QgsProject, QgsVectorLayer
layers=[]
for file in glob.glob('C:/Users/1134377/Documents/ASD/ModelBuilder/*.txt'): # Change this base path and change backslashes to forward slashes
  uri = "file:///" + file + "?delimiter=%s&xField=%s&yField=%s&useHeader=no&crs=epsg:4326" % (",","field_2","field_3") #(Delimeter, XField, YField) #1st column in file is column 1
  vlayer = QgsVectorLayer(uri, os.path.basename(file), "delimitedtext")
  vlayer.setFieldAlias(1,'X') #these order of columns go: 0, 1, 2 etc
  vlayer.setFieldAlias(2,'Y')
  layers.append(vlayer)

QgsProject.instance().addMapLayers(layers)
processing.runAndLoadResults("native:pointstopath", {'INPUT':'delimitedtext://file:///C:/Users/1134377/Documents/ASD/ModelBuilder\\*.txt?delimiter=,&xField=field_2&yField=field_3&useHeader=no&crs=epsg:4326','CLOSE_PATH':False,'ORDER_EXPRESSION':'','NATURAL_SORT':False,'GROUP_EXPRESSION':'\"field_4\"','OUTPUT':'C:/MyFolder/Path.shp'})
processing.runAndLoadResults("native:polygonize", {'INPUT':'C:/MyFolder/Path.shp','KEEP_FIELDS':False,'OUTPUT':'C:/Users/1134377/Documents/ASD/ModelBuilder/Outputs/Poly.kml'})

The limitation of this script is that it doesn't run the algorithms (point to path & polygonize) on all the text layers that are added to the map using the beginning portion of code. It also needs to be run within the python console within QGIS, and not as a standalone script.

Best Answer

Building on recipes from the two answers here and here, you can try the script below. You will need to edit at least a couple of lines. Depending on how you installed QGIS (whether you used the standalone installer or OsGeo4W installer) your QgsApplication prefix path and the path to the processing plugin may be different from what I used in the script.

Also, you will need to edit the in_folder and out_folder paths to match your own file system.

import sys

import os

from qgis.core import (QgsApplication, QgsVectorLayer)
from qgis.analysis import QgsNativeAlgorithms

# See https://gis.stackexchange.com/a/155852/4972 for details about the prefix 
QgsApplication.setPrefixPath('C:/OSGeo4W/apps/qgis', True)
qgs = QgsApplication([], False)
qgs.initQgis()

# Append the path where processing plugin can be found
sys.path.append('C:\\OSGeo4W\\apps\\qgis\\python\\plugins')

import processing
from processing.core.Processing import Processing
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())

in_folder = 'C:\\Users\\Ben\\Desktop\\TEMP\\txt_files'

out_folder = 'C:\\Users\\Ben\\Desktop\\TEMP\\kml_files'

def save_as_kml(in_file, save_location):
    out_layer = os.path.join(save_location, f'{in_file.name.replace(".txt", ".kml")}')
    uri = 'file:///{}?delimiter={}&crs=epsg:4326&xField={}&yField={}'.format(in_file.path, '\\t','X', 'Y')

    vlayer = QgsVectorLayer(uri, os.path.basename(in_file), 'delimitedtext')
    paths = processing.run("qgis:pointstopath",
                {'INPUT':vlayer,
                'CLOSE_PATH':True,
                'ORDER_FIELD':'Sort',
                'GROUP_FIELD':'',
                'DATE_FORMAT':'',
                'OUTPUT':'TEMPORARY_OUTPUT'})
    processing.run("native:polygonize",
                {'INPUT':paths['OUTPUT'],
                'KEEP_FIELDS':False,
                'OUTPUT':out_layer})

src_dir = os.scandir(in_folder)
for file in src_dir:
    if file.name.endswith('.txt'):
        save_as_kml(file, out_folder)

I'm not sure how you run your standalone scripts, but if you have trouble, here is how I do it:

Save the script above as a .py file (e.g. save_as_kml.py)

Next, create a batch file with the following content:

@echo off
SET OSGEO4W_ROOT=C:\OSGeo4W
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
@echo off
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
path %PATH%;C:\OSGeo4W\apps\Qt5\bin
path %PATH%;C:\OSGeo4W\apps\Python39\Scripts
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT:\=/%/apps/qgis
set GDAL_FILENAME_IS_UTF8=YES
rem Set VSI cache to be used as buffer, see #6448
set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000
set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python
set PYTHONHOME=%OSGEO4W_ROOT%\apps\Python39
set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt5\plugins

cmd.exe

Again, the OSGEO4W_ROOT may be different than mine depending on your installation, just make sure it points to your main installation directory which contains the bin folder.

Save the batch file in the same location as your .py file. Then you can just double click the batch to launch it and, at the prompt, type: python save_as_kml.py to run your script.

Related Solutions

PyQGIS – Loading Non-Spatial Data via CSV for Standalone Script in Python 2.7

Try to reverse engineer (deduce from UI behaviour the expected parameters).

According to the return error, you have an issue with delimiter in your CSV data source before processing.

How can you do it?

You should try to go in Layers > Add Layer > Add Delimited Text Layer... and add your CSV file like below :

You will be able to debug visually with this.

Then, go in your layer Properties and in tab General, look at Layer source. You should have similar result as the following image. You can from this, deduce additional arguments required to add your CSV layer with PyQGIS without issues.

I forgot to mention additional information: the documentation of each parameters for Delimited Text Layer within the QGIS API is available from this QGIS API page (link for QGIS 2.14).

QGIS – Utilizing QGIS Processing Tools in a Standalone Script

The arguments are:

runAlgorithm(algOrName, onFinish, *args, **kwargs)

Set the onFinish argument to None (i.e. no post-processing)

However, by the looks of it you are just wanting to run GDAL Warp. May I suggest you go for a simpler approach for any of the GDAL functions? QGIS is a wrapper around GDAL so for a stand-alone process why involve QGIS at all? Cut out the middle-man and use the GDAL bindings directly in Python. This is not only simpler, it is also more portable. Bundled in GDAL (and therefore QGIS) you will find a standalone exe version of GDAL Warp (and others) ready to go and that may be all you need.

In a standard install, you'll find the GDAL Python libraries and standalone exe files in the bin folder (e.g. C:/OSGeo4W64/bin on my machine). On a machine without QGIS you can install the GDAL bindings from here (I recommend the GISInternals site for Windows). You only need to do this is you move your code to a machine without QGIS of course. Numpy and SciPy are also similarly available and usable without the QGIS wrapper.

In the case of GDAL Warp (or any of the GDAL exe processes) you can call the exe as a subprocess from Python or use it directly on the commandline.

Best Answer

Related Solutions

PyQGIS – Loading Non-Spatial Data via CSV for Standalone Script in Python 2.7

QGIS – Utilizing QGIS Processing Tools in a Standalone Script

Related Question