[GIS] Multiprocessing error in QGIS with Python on Windows

parallel processingpythonqgis

I have this error when I try to use Manager() from python multiprocessing library.
(Just start QGIS open Python console and execute the two following lines)

>>> import multiprocessing
>>> mgr = multiprocessing.Manager()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\DEV\osgeo4w_dev\apps\Python27\lib\multiprocessing\__init__.py", line 99, in Manager
    m.start()
  File "C:\DEV\osgeo4w_dev\apps\Python27\lib\multiprocessing\managers.py", line 524, in start
    self._process.start()
  File "C:\DEV\osgeo4w_dev\apps\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\DEV\osgeo4w_dev\apps\Python27\lib\multiprocessing\forking.py", line 252, in __init__
    cmd = get_command_line() + [rhandle]
  File "C:\DEV\osgeo4w_dev\apps\Python27\lib\multiprocessing\forking.py", line 339, in get_command_line
    if process.current_process()._identity==() and is_forking(sys.argv):
AttributeError: 'module' object has no attribute 'argv'
>>>

It happens in both 1.8 and 1.9 dev version on Windows. Of course everything is ok using QGIS on Linux!… 😛

It seems something related to the main Python/QGis process startup and the way the library tries to start a new process…. but I can't figure out any workaround.

Any idea? (already posted this on qgis dev list but had no response…)

Best Answer

I was wrong in my comment to original post. Suggested workaround does work. In case someone else has that issue here is what needs to be done in QGIS 2.0 before instantiating Manager().

# OSGeo4W does not bundle python in exec_prefix for python
path = os.path.abspath(os.path.join(sys.exec_prefix, '../../bin/pythonw.exe'))
mp.set_executable(path)
sys.argv = [ None ]

Note that this cannot be tested from python console as Windows lacks fork() and all multiprocessing statements shall be isolated.

If one wants to play around with multiprocessing and embedded python from OSGeo4W bundle outside of QGIS here is the code.

tst.py

import multiprocessing as mp
import sys, os

print("Non-isolated statement")

if __name__ == '__main__':
    print("I'm in main module")
    path = os.path.abspath(os.path.join(sys.exec_prefix, '../../bin/pythonw.exe'))
    mp.set_executable(path)
    print("Setting executable path to {:s}".format(path))
    sys.argv = [ None ]               # '../tst.py' __file__
    mgr = mp.Manager()
    print("I'm past Manager()")

tst.c

#include <Python.h>
#include <stdio.h>

int main(int argc, char * argv[]) {
  char buf[10240] = {0};
  size_t sz, res;
  FILE *f;
  Py_SetProgramName(argv[0]);  /* optional but recommended */
  Py_Initialize();

  f = fopen("../tst.py", "r");
  // obtain file size:
  fseek(f, 0 , SEEK_END);
  sz = ftell(f);
  rewind(f);
  res = fread(buf, 1, sz, f);
  fclose(f);
  PyRun_SimpleString(buf);
  getchar();
  /* PyRun_SimpleString("from time import time,ctime\n" */
  /*                    "print 'Today is',ctime(time())\n"); */
  Py_Finalize();
  return 0;
}

CMakeLists.txt

cmake_minimum_required(VERSION 2.8)
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_DIRS})

add_executable(tst tst.c)
target_link_libraries(tst ${PYTHON_LIBRARY})

if(MSVC)
  set_target_properties(tst PROPERTIES LINK_FLAGS -NODEFAULTLIB:python27_d)
endif()
  1. From your OSGeo4W console, initialize your vcvarsall.bat building environment
  2. Create subfolder build (or alike) and cd to it
  3. Use cmake-gui .. to generate jom/nmake makefiles in build folder provided all files are saved in the parent one
  4. Use nmake or jom to build tst.exe
  5. Try to run tst or python ..\tst.py
Related Question