[GIS] Conflict between ArcPy and GDAL/OGR

arcgis-10.0arcpygdalimporterrorogr

TL;DR

I can't get an installation free version of GDAL/OGR to work from within ArcMap 10.0. I get an ImportError: DLL load failed on _mod = imp.load_module('_gdal', fp, pathname, description).

Below, more details on the context and a procedure on how to reproduce the error.

Someone in the comments specified that arcpy and GDAL work fine together on his machine. What could be the differences between our configurations (software installed, environment variables, …)?

Description of my issue

What I am trying to do…

I want to deploy an ArcMap 10.0 ToolBox with scripts that use external libraries. I would like to avoid having any manipulation done on the user side: ideally, no additional installation/manipulation should be required for the user. For that reason, I embed the libraries files inside the ToolBox.

What my problem is…

When I try to use gdal/ogr from osgeo along with arcpy, I get the following error:

Traceback (most recent call last):
  File "path_to_script\debugDepl.py", line 32, in <module>
    from osgeo import ogr
  File "path_to_script\osgeo\__init__.py", line 30, in <module>
    _gdal = swig_import_helper()
  File "path_to_script\osgeo\__init__.py", line 26, in swig_import_helper
    _mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: DLL load failed: The specified procedure could not be found.

There are some binding errors that I don't understand. It seems that there is a conflict between arcpy and gdal/ogr (more information below)

Related:

How to reproduce the error

If you can help me achieve want I want, it might be useful if I explain every step that lead to this error. I tried to detail as much as I could so that you could reproduce it.

1/ On a Windows 7 64bits machine, Install ArcGIS Desktop 10.0 (32 bits)

2/ Download GDAL‑1.11.0.win32‑py2.6.exe Windows binary

3/ Install that library with your Python installation

4/ Move the associated library files (path_to_python_install\Python26\ArcGIS10.0\Lib\site-packages) to another new folder (osgeo folder + several Python files)

5/ Create a Python script in that folder and edit it in your favorite Python editor and check that you can import gdal/ogr

from osgeo import gdal
from osgeo import ogr

6/ Try importing arcpy from the script before and after importing gdal/ogr

from osgeo import ogr
from osgeo import gdal 
import arcpy

That should work, now try the other way around

import arcpy
# Maybe some environment variable settings
from osgeo import ogr
from osgeo import gdal

And that version should create the error.

7/ Create a toolbox (or call the script from within ArcMap) with the version that worked standalone. And try to run it within ArcMap (should fail).

Notes on the steps

1/ I've found some official Python bindings but only for ArcMap 10.1

2/ This distribution includes a complete GDAL installation. It is a 32bit version for my 32bit version of Python26.

4/ As specified in these recommendations for scripts deployment, I should copy the library files directly. I want to avoid the user to have to install things as much as possible

5/ This may (or may not) require some environment variable settings:

import os
environList = os.environ['PATH'].split(';')
environList.insert(0, r'path_to_folder\osgeo')
os.environ['PATH'] = ';'.join(environList)
os.environ['GDAL_DATA'] = r'path_to_folder\osgeo\data\gdal'

GDAL Documentation specifies that we should:

  • Add the installation directory bin folder to your system PATH, remember to put a semicolon in front of it before you add to the existing path.
  • Create a new user or system variable with the data folder from your installation.

Developers on this blog page seemed to have that error when the environment variables were incorrect.

6/ It seems that for some reason if you import arcpy first and ogr/gdal second, gdal is no longer able to find the dll's that it requires. This could have been due to some environment variable settings but the modification I proposed at step 5) didn’t change anything.

7/ But then my guess is that arcpy is automatically imported (first) so we’re back to the 2nd version of 6)

Some more information about my attempt to understand what is going on

From PyScripter:

>>> from osgeo import gdal
>>> gdal.__file__
'C:\\Python26\\ArcGIS10.0\\lib\\site-packages\\osgeo\\gdal.pyc'
>>> import arcpy
>>> arcpy.__file__
'C:\\Program Files (x86)\\ArcGIS\\Desktop10.0\\arcpy\\arcpy\\__init__.pyc'

Alright, the library files were found where they are supposed to be. But then…

>>> import arcpy
>>> from osgeo import gdal
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python26\ArcGIS10.0\lib\site-packages\osgeo\__init__.py", line 29, in <module>
    _gdal = swig_import_helper()
  File "C:\Python26\ArcGIS10.0\lib\site-packages\osgeo\__init__.py", line 25, in swig_import_helper
    _mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: DLL load failed: The specified procedure could not be found.

Maybe something happened to the environment variables or to the python path when importing Arcpy? Well, it doesn't seem like it…

>>> import os
>>> import sys
>>> envBefore = os.environ
>>> pyPathBefore = sys.path
>>> import arcpy
>>> envAfter = os.environ
>>> pyPathAfter = sys.path
>>> envBefore == envAfter
True
>>> pyPathBefore == pyPathAfter
True

Best Answer

I suspect the issue is the GDAL 1.11 bindings you're fetching from gohlke do not match the internal GDAL inside of Arc*. Your bindings need to be compiled against ESRI's GDAL to work reliably. It's possible they can be made to work, but it is going to be a lot of headache.

In short, ESRI needs to provide the gdal_i.lib stub file that GDAL generates as part of its compilation process, and you need to build the Python bindings against that. I don't know that ESRI provides this, however.

Maybe ESRI could be requested to provide a gdal.py and GDAL bindings as part of ArcPy? Alternatively, maybe you could use Fiona and rasterio to replicate the GDAL bindings parts you need. These have the benefit of not needing to link against the ESRI GDAL and instead are able to use it by doing C API calls directly into the DLL.

Related Question