[GIS] Issues with QgsTask and task manager

Question

I am trying to use the task manager to run a long operation (uploading layer to a GeoServer instance, for the Geoserver Explorer plugin), but I am finding some issues.

Here is the code that I am using:

def _publishLayers(catalog, layers, layersUploaded):
    task = PublishLayersTask(catalog, layers) 
    task.taskCompleted.connect(layersUploaded)
    QgsApplication.taskManager().addTask(task)

class PublishLayersTask(QgsTask):

    def __init__(self, catalog, layers):
        QgsTask.__init__(self, "Publish layers")
        self.layers = layers
        self.catalog = catalog
        self.exception = None
        self.errortrace = None

    def canCancel(self):
        return False

    def run(self):
        try:
            //Upload layers here
            return True
        except Exception as e:            
            self.exception = e
            self.errortrace = traceback.format_exc()
            return False

    def finished(self, ok):
        if ok:
            //show confirmation message
        else:
            //show error message and log error details

The issues I have are mainly two:

1) After adding the task to the task manager, it doesnt start. I have to add some call that refreshes the UI, such as logging a dummy message to the QGIS log or calling processEvents() to have it started. I have tried in Win and Mac, and in both cases the task is not triggered. Docs say that the task manager will take care of running it when it can, but that never happens, no matter what I do on the QGIS UI.

2) The finished() method is not always run when the task is finished. The layersUploaded method that is connected to the taskCompleted signal is always run when the task is correctly finished, but the finished method of the task is only called sometimes. That doesn't depend on the success of the task. If it fails, the finished method is only run sometimes, and I could not find any pattern in it.

Any ideas or suggestions?

Answer 1

QGIS version 3.4.2 If you define a task and add it to the task manager within a function scope, it will cause the task to either not execute properly or, at worst, crash QGIS. Consider the following code.

import random
from time import sleep
from qgis.core import QgsApplication, QgsTask, QgsMessageLog, Qgis


MESSAGE_CATEGORY = 'Wasting time'

def do_task(task, wait_time):
    """
    Raises an exception to abort the task.
    Returns a result if success.
    The result will be passed together with the exception (None in
    the case of success) to the on_finished method
    """
    QgsMessageLog.logMessage('Started task {}'.format(task.description()),
                             MESSAGE_CATEGORY, Qgis.Info)
    wait_time = wait_time / 100
    total = 0
    iterations = 0
    for i in range(100):
        sleep(wait_time)
        # use task.setProgress to report progress
        task.setProgress(i)
        arandominteger = random.randint(0, 500)
        total += arandominteger
        iterations += 1
        # check task.isCanceled() to handle cancellation
        if task.isCanceled():
            stopped(task)
            return None
        # raise an exception to abort the task
        if arandominteger == 42:
            raise Exception('bad value!')
    return {'total': total, 'iterations': iterations,
            'task': task.description()}

def stopped(task):
    QgsMessageLog.logMessage(
        'Task "{name}" was canceled'.format(
            name=task.description()),
        MESSAGE_CATEGORY, Qgis.Info)

def completed(exception, result=None):
    """This is called when do_task is finished.
    Exception is not None if do_task raises an exception.
    Result is the return value of do_task."""
    if exception is None:
        if result is None:
            QgsMessageLog.logMessage(
                'Completed with no exception and no result '\
                '(probably manually canceled by the user)',
                MESSAGE_CATEGORY, Qgis.Warning)
        else:
            QgsMessageLog.logMessage(
                'Task {name} completed\n'
                'Total: {total} ( with {iterations} '
                'iterations)'.format(
                    name=result['task'],
                    total=result['total'],
                    iterations=result['iterations']),
                MESSAGE_CATEGORY, Qgis.Info)
    else:
        QgsMessageLog.logMessage("Exception: {}".format(exception),
                                 MESSAGE_CATEGORY, Qgis.Critical)
        raise exception


# Create and execute a few tasks
task1 = QgsTask.fromFunction(u'Waste cpu 1', do_task,
                             on_finished=completed, wait_time=4)
task2 = QgsTask.fromFunction(u'Waste cpu 2', do_task,
                             on_finished=completed, wait_time=3)
QgsApplication.taskManager().addTask(task1)
QgsApplication.taskManager().addTask(task2)

If you put this in the script editor in QGIS and execute it, it will run just fine. The user can cancel. The exceptions are raised, occasionally. All is well. Now if you replace the last block of the code to be within a function scope, like so:

def task_create_and_execute():
    # Create and execute a few tasks
    task1 = QgsTask.fromFunction(u'Waste cpu 1', do_task,
                                 on_finished=completed, wait_time=4)
    task2 = QgsTask.fromFunction(u'Waste cpu 2', do_task,
                                 on_finished=completed, wait_time=3)
    QgsApplication.taskManager().addTask(task1)
    QgsApplication.taskManager().addTask(task2)

task_create_and_execute()

Now when you run the script you will find that either task1 will appear to run, but the completed() function will never be reached (task2 will NOT run) or QGIS will crash. If you run it twice, QGIS will most surely crash. This will also happen when using the class method, as in OP's code. This is likely a bug and I may link to this answer and report it. A possible workaround would be to take the task creation and execution out of scope, as in my first example, and run it from the script editor.

Update: I have reported this issue to QGIS, however it may not be a bug and just a matter of scope. If you have the following code, in a module named task_example.py, you will find it to work correctly for both the subclassed and fromFunction methods:

import random
from time import sleep
from qgis.core import QgsApplication, QgsTask, QgsMessageLog, Qgis

MESSAGE_CATEGORY = 'Task Subclass'

class TaskSubclass(QgsTask):
    """This shows how to subclass QgsTask"""
    def __init__(self, description, duration):
        super().__init__(description, QgsTask.CanCancel)
        self.duration = duration
        self.total = 0
        self.iterations = 0
        self.exception = None

    def run(self):
        """Here you implement your heavy lifting.
        Should periodically test for isCanceled() to gracefully
        abort.
        This method MUST return True or False.
        Raising exceptions will crash QGIS, so we handle them
        internally and raise them in self.finished
        """
        QgsMessageLog.logMessage('Started task "{}"'.format(
                                     self.description()),
                                 MESSAGE_CATEGORY, Qgis.Info)
        wait_time = self.duration / 100
        for i in range(100):
            sleep(wait_time)
            # use setProgress to report progress
            self.setProgress(i)
            arandominteger = random.randint(0, 500)
            self.total += arandominteger
            self.iterations += 1
            # check isCanceled() to handle cancellation
            if self.isCanceled():
                return False
            # # simulate exceptions to show how to abort task
            # if arandominteger == 42:
                # # DO NOT raise Exception('bad value!')
                # # this would crash QGIS
                # self.exception = Exception('bad value!')
                # return False
        return True

    def finished(self, result):
        """
        This function is automatically called when the task has
        completed (successfully or not).
        You implement finished() to do whatever follow-up stuff
        should happen after the task is complete.
        finished is always called from the main thread, so it's safe
        to do GUI operations and raise Python exceptions here.
        result is the return value from self.run.
        """
        if result:
            QgsMessageLog.logMessage(
                'Task "{name}" completed\n' \
                'Total: {total} (with {iterations} '\
              'iterations)'.format(
                  name=self.description(),
                  total=self.total,
                  iterations=self.iterations),
              MESSAGE_CATEGORY, Qgis.Success)
        else:
            if self.exception is None:
                QgsMessageLog.logMessage(
                    'Task "{name}" not successful but without '\
                    'exception (probably the task was manually '\
                    'canceled by the user)'.format(
                        name=self.description()),
                    MESSAGE_CATEGORY, Qgis.Warning)
            else:
                QgsMessageLog.logMessage(
                    'Task "{name}" Exception: {exception}'.format(
                        name=self.description(),
                        exception=self.exception),
                    MESSAGE_CATEGORY, Qgis.Critical)
                raise self.exception

    def cancel(self):
        QgsMessageLog.logMessage(
            'Task "{name}" was canceled'.format(
                name=self.description()),
            MESSAGE_CATEGORY, Qgis.Info)
        super().cancel()


class DoStuff:
    def __init__(self):
        """Example of the class that needs to get tasks done. """
        """Subclassed examples. """
        self.longtask = TaskSubclass('class method long', 20)
        self.shorttask = TaskSubclass('class method short', 10)
        self.minitask = TaskSubclass('class method mini', 5)
        # Subtasks to do.
        self.shortsubtask = TaskSubclass('class method subtask short', 5)
        self.longsubtask = TaskSubclass('class method subtask long', 10)
        self.shortestsubtask = TaskSubclass('class method subtask shortest', 4)
        # Add a subtask (shortsubtask) to shorttask that must run after
        # minitask and longtask has finished
        self.shorttask.addSubTask(self.shortsubtask,
                                  [self.minitask, self.longtask])
        # Add a subtask (longsubtask) to longtask that must be run
        # before the parent task
        self.longtask.addSubTask(self.longsubtask, [], 
                                 TaskSubclass.ParentDependsOnSubTask)
        # Add a subtask (shortestsubtask) to longtask
        self.longtask.addSubTask(self.shortestsubtask)

        """From function examples. """
        self.MESSAGE_CATEGORY = 'Task from Function'

        self.task1 = QgsTask.fromFunction(u'function task', self.internal_task,
                                     on_finished=self.completed, wait_time=4)
        self.task2 = QgsTask.fromFunction(u'function task 2', self.internal_task,
                                     on_finished=self.completed, wait_time=3)

    def do_subclassed_tasks(self):
        """Do tasks using QgsTask subclass. """
        QgsApplication.taskManager().addTask(self.longtask)
        QgsApplication.taskManager().addTask(self.shorttask)
        QgsApplication.taskManager().addTask(self.minitask)

    def do_from_function_tasks(self):
        """Do tasks from a function. """
        QgsApplication.taskManager().addTask(self.task1)
        QgsApplication.taskManager().addTask(self.task2)

    def internal_task(self, task, wait_time):
        """
        Raises an exception to abort the task.
        Returns a result if success.
        The result will be passed together with the exception (None in
        the case of success) to the on_finished method
        """
        QgsMessageLog.logMessage('Started task {}'.format(task.description()),
                                 self.MESSAGE_CATEGORY, Qgis.Info)
        wait_time = wait_time / 100
        total = 0
        iterations = 0
        for i in range(100):
            sleep(wait_time)
            # use task.setProgress to report progress
            task.setProgress(i)
            arandominteger = random.randint(0, 500)
            total += arandominteger
            iterations += 1
            # check task.isCanceled() to handle cancellation
            if task.isCanceled():
                stopped(task)
                return None
            # # raise an exception to abort the task
            # if arandominteger == 42:
                # raise Exception('bad value!')
        return {'total': total, 'iterations': iterations,
                'task': task.description()}

    def stopped(self, task):
        QgsMessageLog.logMessage(
            'Task "{name}" was canceled'.format(
                name=task.description()),
            self.MESSAGE_CATEGORY, Qgis.Info)

    def completed(self, exception, result=None):
        """This is called when do_task is finished.
        Exception is not None if do_task raises an exception.
        Result is the return value of do_task."""
        if exception is None:
            if result is None:
                QgsMessageLog.logMessage(
                    'Completed with no exception and no result '\
                    '(probably manually canceled by the user)',
                    self.MESSAGE_CATEGORY, Qgis.Warning)
            else:
                QgsMessageLog.logMessage(
                    'Task {name} completed\n'
                    'Total: {total} ( with {iterations} '
                    'iterations)'.format(
                        name=result['task'],
                        total=result['total'],
                        iterations=result['iterations']),
                    self.MESSAGE_CATEGORY, Qgis.Info)
        else:
            QgsMessageLog.logMessage("Exception: {}".format(exception),
                                     MESSAGE_CATEGORY, Qgis.Critical)
            raise exception

You can now, in the python console (or another py file), make the following executions:

import task_example

okay = task_example.DoStuff()
okay.do_subclassed_tasks()
okay.do_from_function_tasks()

Please note that I commented out the parts of the code that cause an exception when the random integer is 42. I was tired of the answer to the ultimate question of life, the universe, and everything throwing exceptions at me.