Esri Python – Installing Pandas with Esri’s Python

arcgis-10.2arcpynumpypandaspython

Trying to get pandas to import, but I'm getting what looks like an error caused by a failed dependency of the version of my Esri-provided numpy's compile, based on this stackoverflow posting. Here's what I get.

>>> import pandas
numpy.dtype has the wrong size, try recompiling
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python27\ArcGIS10.2\lib\site-packages\pandas\__init__.py", line 6, in <module>
    from . import hashtable, tslib, lib
  File "numpy.pxd", line 157, in init pandas.hashtable (pandas\hashtable.c:22315)
ValueError: numpy.dtype has the wrong size, try recompiling

I check my numpy version:

>>> import numpy
>>> numpy.version.full_version
'1.6.1'

Based on this, I believe that this is the right version of numpy for the version of pandas I'm using (tried 13.1, 14.0, and 14.1).

I'm running the standard ESRI-shipped python for ArcGIS 10.2 (Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32).

>>> import arcpy
>>> arcpy.__file__
'C:\\Program Files (x86)\\ArcGIS\\Desktop10.2\\arcpy\\arcpy\\__init__.py'

Has anyone succeeded in getting pandas going this way or have ideas on what's wrong with my environment?

Should I update my numpy, which is supposed to be backwards compatible (according to user333700's text at the bottom of the stackoverflow posting), or is that likely to break my ArcGIS functionality?

Best Answer

I was able to get pandas working in ArcMap 10.1 using pandas packages already built for Anaconda.

One of the nice things about Anaconda (besides being free) is that it's easy to create environments with specific versions of python and numpy.

Try this:

Open a 32-bit Anaconda command prompt and type:

conda create -n esri101 python=2.7 numpy=1.6 pandas
activate esri101

Then create a file called conda.pth in your ArcGIS10.1 site-packages folder. On my machine this is C:\Python27\ArcGIS10.1\Lib\site-packages\conda.pth

In this file add one line that points to your Anaconda site-packages for the esri101 environment you created. On my machine this is:

C:\programs\Anaconda\envs\esri101\Lib\site-packages

Then fire up ArcMap and try import pandas.

The only unfortunate thing is that it looks like the latest pandas release anyone built for windows with python=2.7 and numpy=1.6 is pandas=0.10.1. I might try building a later version using this stackoverflow answer

Related Solutions

[GIS] Avoiding import arcpy error (RuntimeError: NotInitialized) when using IDLE

Looking at your posts here and on the ESRI Forums, I think you have some muddled environment settings.

You want to put C:\Python26\ArcGIS10.0 in your system PATH variable, that gets you to the correct executables from IDLE, PythonWin or the command line.

Your PYTHONPATH could be empty. Or, you could just include the 3 stanzas of your Desktop10.pth file: C:\Program Files (x86)\ArcGIS\Desktop10.0\bin C:\Program Files; (x86)\ArcGIS\Desktop10.0\arcpy; C:\Program Files (x86)\ArcGIS\Desktop10.0\ArcToolbox\Scripts--those paths will then take precedence over the the default search for modules (below the python.exe or pythonw.exe being called).

Finally, since you've uninstalled and installed several versions of Python. At this point, there should be nothing under the C:\Python26 folder except the ArcGIS10.0 folder of the ESRI provided Python 2.6.5 32-bit install. If there is a Lib file there it is a residual of an earlier install and will interfere. It would be a bigger problem if any or your prior Python26 installs were 64-bit.

Also, since you're on 64-bit Windows 7, the Registry path the to 32-bit Python keys are HKLM\SOFTWARE\Wow6432Node\Python If you have a HKLM\SOFTWARE\Python, they would be Keys for a 64-bit install. They also would interfere with the 32-bit install.

[GIS] Importing ArcPy taking many seconds and code is really slow for GeoProcessing Tools 64-bit (ArcGIS 10.1 SP1)

I also create ArcPy geoprocessing tools using PyDev for Eclipse. Generally, I run my scripts as in-process geoprocessing tools in a custom toolbox (.tbx) from within ArcMap so that the import arcpy happens instantly. If I run a script from outside of ArcMap, the import arcpy step usually takes about 5 seconds on a Sandy Bridge Core i5.

I think your trouble may lie in the use of 64-bit Python and the ArcGIS 64-bit background geoprocessing package. I don't know why this would take so much longer to import compared to 32-bit ArcPy. If you want confirmation, you could iteratively add/remove Python libraries from the Python Interpreters menu you have and see if a specific folder is causing the excessive import times.

Personally, I haven't found a compelling need to use 64-bit geoprocessing. Even in some of the massive Spatial Analyst workflows I've had to implement, geoprocessing usually crashes before I hit memory limits. I prefer to divide my inputs before processing and merge them together at the end. This also ensures better compatibility with 3rd part modules, most of which are already compiled for 32-bit Python.

I also think there's a bunch of tweaks you can make to your code to make it faster, besides the import arcpy issue and 64- vs. 32-bit Python. Creating a new searchCursor object for each iteration of your fc_nodes features is really slow. This means you are creating 6290 * 6290 = 39,564,100 search cursor objects. Creating each cursor object has processing overhead, so it's not surprising that this is taking many minutes to run. You want to avoid this code structure with nested cursors.

It appears that you are trying to find the counts of features in fc_links with the value of the TONODENO field equal to each possible value of the NO field in fc_nodes. Instead of using these nested cursors, it would be much easier to use the arcpy.Statistics_analysis() tool, with the TONODENO as the case field:

fc_links = "Rede_Cbr_MM_2008_WithoutMetro"
fieldname = "TONODENO"
statisticsTable = "OutputStatisticsTable"
arcpy.Statistics_analysis(fc_links, statisticsTable, [[fieldname,"COUNT"]], fieldname)

This is similar to do a Group By query in SQL, or a Totals query in Excel. In the output table, the FREQUENCY column gives the number of features in fc_links that have that value for the TONODENO field. You can join this back table to the fc_nodes table, or store it in a dictionary or something like that.

Best Answer

Related Solutions

[GIS] Avoiding import arcpy error (RuntimeError: NotInitialized) when using IDLE

[GIS] Importing ArcPy taking many seconds and code is really slow for GeoProcessing Tools 64-bit (ArcGIS 10.1 SP1)

Related Question