intermediate ArcMap user here. I'm trying to calculate the geometry for a shapefile such that I can add lat/long columns to my attribute table. While I have no trouble calculating the latitude, the longitude keeps coming out a decimal place off.
I know that the accurate longitude for this shapefile should be -74.067, -74.068, etc. I'm not sure if the negative sign is screwing up the significant digits? I looked around StackExchnage and no one seems to have this problem.
For reference, when calculating the Latitude, for "Type", I used "float" with precision = 6 and scale = 5. This worked fine for Latitude but still gives me the incorrect result for Longitude.
Also for reference:
Projected Coordinate System: PCS_CarMAGBOG
Projection: Transverse_Mercator
False_Easting: 92334.87900000
False_Northing: 109320.96500000
Central_Meridian: -74.14659167
Scale_Factor: 1.00000000
Latitude_Of_Origin: 4.68048611
Linear Unit: Meter
Has anyone faced this problem before, and if so, how did you resolve it? Is the problem in calculate geometry or in the projection itself?
Best Answer
You should use a 'double' field type instead of 'float' as per Esri's recommendations. "When you create float and double fields and specify a precision and scale, if your precision is greater than 6, use a double" (http://resources.arcgis.com/en/help/main/10.1/index.html#//001700000047000000).
If you change your Precision and Scale to 8 and 5, respectively, then 'Calculate Geometry' in ArcMap will return the correct coordinates. Values of 7 and 5 will actually round the longitudes to integers. I believe the negative sign is treated counts towards the precision limit.
I downloaded your projection here (http://www.spatialreference.org/ref/sr-org/6873/) and actually tested this on a set of points I created with the same long/lat from your data. I could only get correct longitude calculations to work using the 8 precision and 5 scale.