ArcGIS Desktop – Troubleshooting Slow Data Access to Shapefiles

arcgis-9.3arcgis-desktop

We have just upgraded from ArcGIS Desktop 9.2 to 9.3.1

For a ~250MB .shp in 9.2 it took 1 minute to render, at 9.3.1 it now takes 50 minutes.

The format has to stay as shapefile.
The data is stored on both NetApp and Dell Servers – problem consistent on both.
Only an issue with GIS data, in particular shapefiles and only in 9.3.1.
Have already have tested during the day and night and the times returned are consistent showing that network traffic is having no or very little effect on the poor performance.
Also tested on servers that do not have virus protection to eliminate that from the equation.
Already tried testing on several servers with differing switches (100Mbit and 1Gbit) but the results have been the same.
Have tested on a newly built pc with a vanilla install of ArcGIS so it can’t be incorrectly uninstalled components or third party products causing the issue.

I am putting together some further test-cases to carry out, to try and pinpoint the problem further before taking it to Esri:

Run the tests at 9.2 with a large ESRI sample dataset. Run tests with it as a shp and as a file GDB.
Re-test with 9.3.1 (+SPs) and again with v10
Run tests in both ArcMap, ArcGIS Explorer and ArcReader (to see if it is specific to an application)
Compare these results with how long it takes to open when stored locally?

From what I know, there have not been any changes made to the underlying code and the way that ArcGIS connects to a network between 9.3.1 and 9.2. I also don't think that Esri changed anything in the way that ArcGIS reads shapefiles between these versions?

Other than the above, the only other suggestion I can think of is to run WireShark to try and see how long it takes to send/receive packets from the client machine & the server.

This will tell me two things:

1) How many packets are going both directions

2) how long each packet takes to be transmitted.

My thinking is, if it is related to 9.3.1, then you would most likely see an increase in the number of packets. It could be a security configuration that at 9.3.1 we have encountered, but at 9.2 we did not.
It would be best to capture a wireshark trace for both 9.2 and 9.3.1 accessing the same shape file on the same server.

Has anyone experienced anything similar to this?

Does anyone have any other ideas on how to troubleshoot this?

Best Answer

We had this as well unfortunately - I believe it is a known bug. The only way around it we found was to clone data across the network to a local drive, but that isn't much of a solution I know. It has been fixed in 10 however...

Found two possible solutions on the ESRI forum - the one that worked for us was to look at the registry settings, as shown here (it reduced the draw speed from a large mxd from ~3mins to <10 secs).

Alternatively you could try manually building a spatial index on your shapefiles, as explained here.

And the ESRI bug report, which mentions the issue, but not much in the way of fixing it.

Hope this helps!

Related Solutions

[GIS] Editing with auxiliary features (lines, points etc.) in ArcGIS Desktop

Your Draftsman could use the Direction Distance Tool, that you find "under" the Sketch Tool button. The symbol on the button is a circle with a crossing line.

To use the tool:

Set your Target to you Lighting Post and then:

Set snap (edge/vertex) to your curb layer (Editor button -> Snapping).
Choose the Direction Distance Tool.
Click the position on the curb, where the lighting post should be placed.
Now a line shows up on the screen, and by moving the mouse the direction of the line can be changed.
When the right direction is represented by the line, the left mouse button is clicked.
Now click once more on the same place on the curb as in bullet 3.
Now a circle shows up on the screen.
Press "R" on the keyboard and state the radius of the circle (in other words how long you want the the post placed from the curb) - here 4 meters.
Press Enter on the keyboard.
Now choose which of the two possible intersections between the line and the circle you want to use.

That's it.

I know that the accuracy for the direction isn't that good, since you are unable to use the Perpendicular function, but maybe thats ok?

ArcGIS 10 has a new Place Point at the end of a Line tool, that will do exactly what you want, and you can use Perpendicular and so on too, but sadly it is not available in 9.3.1.

ArcGIS Desktop – Finding Parcel Polygon Features with Frontage on Multiple Streets

If your road network is divided by segments, dissolve the buffers based on road name or some common attribute so that individual segments of roads do not count as two separate roads in the next step.
Buffer your street centerlines by a standard amount - more than half the typical ROW width, but no so large that it would overlap parcels that do not front on it. You may want to choose different buffers for different classes of roads. When you buffer, don't merge the polygons.
Spatially join the parcels to the road buffers, choosing "one to many". The "count" field on the results should allow you to identify all parcels with two frontages.

Now you have a table with rows for every overlap between a parcel and a road. You could easily summarize this to find all parcels overlapping two or more roads using either GIS or a database. But eliminating corner lots will be more tricky. Here's one broad method:

Set up topology for the buffer layer with a rule "must not intersect".
Export all errors to a new feature class of intersections.
Spatial join the intersections to the buffer layer, choosing "one to many".

At this point you have two tables: one with all frontages and one with all intersections between roads. It won't be easy, but you can write a SQL query that selects finds all parcels that have two frontages where those frontages do not intersect. The results would be the set of all parcels that have multiple frontages that do not intersect.

One advantage of this method is that it avoids choosing parcels with edges that are not on roads, such as where a parcel is adjacent to an alley, a water features, a railroad ROW, etc. A disadvantage is that the buffering won't be perfect, and you'll still have to visually check it for errors.

I'm not sure how to write the SQL - perhaps someone else can chime in on that.

Edit Thoughts on the SQL:

It would be fairly simple to solve for parcels with two frontages as below. Then you might need to manually look at parcels with more than two frontages, but hopefully there would be few, and most would be corner lots anyway.

This might work for selecting only non-corner parcels with two frontages:

select ParcelID from parcels
where parcelID not in (
    ((select Parcels1.parcelID, intersections.intersectionID from
     intersections left join
        (Select first(RoadID) as Road1, last(RoadID) as Road2, count(*) as frontages, ParcelID
        from Parcels
        where count(*) = 2
        group by ParcelID) as Parcels1
    on intersections.RoadID = Parcels1.Road1) as int1
inner join
    (select Parcels2.parcelID, intersections.intersectionID from
    intersections left join
        (Select first(RoadID) as Road1, last(RoadID) as Road2, count(*) as frontages, ParcelID
        from Parcels
        where count(*) = 2
        group by ParcelID) as Parcels2
    on intersections.RoadID = Parcels2.Road2) as int2
on int1.intersectionID on int2.intersectionID))

Best Answer

Related Solutions

[GIS] Editing with auxiliary features (lines, points etc.) in ArcGIS Desktop

ArcGIS Desktop – Finding Parcel Polygon Features with Frontage on Multiple Streets

Related Question