[GIS] Extracting all address locations from street centerline file

arcgis-desktopgeocodingmapinfo

I have a project in which I want to "extract" all of the geocoded addresses from a street centerline file (with my defined offset and whatever other user defined input is needed). If it is helpful here is a description of what type of format a street centerline file is in.

Currently the only way I know how to do this would be a brute force geocode (i.e. make all possible options and geocode the points that way). Which isn't an impossible option but I would like to see if a more efficient means exists.

I mostly use ESRI software (I have access to machines running 9.2 and 9.3) but if an easier solution exists in some freeware package I'm open to anything. I also have access to MapInfo through my university which I would be willing to use if a simple solution exists. Also all of the street centerline files are in .shp format so any other program will have to either use the shapefile format or have the ability for me to convert the shapefile to a useable format (I assume a trivial step).

I have a very little bit of programming experience in Python (and more extensive experience with statistical software but that won't really help out here). So if I had code options I suppose an ArcScript with a GUI or Python would be preferred.

Also my "universe" of locations I am creating will include street intersections, so if anyone can point to a solution to extract all of the street intersections in the street centerline file (and record both the intersecting streets) I would appreciate this as well. I assume this is more likely to already exist in some ArcScript.

Edit:
To address DavidF's concerns, I know what information is contained in a street centerline file, and I know they are not actual and verified street addresses. For this project I am not making my own geocoder, I am simply defining the universe of address locations according to the street centerline file itself (I also know this universe will be dependent on how far I decide to offset the address locations).

The immediate goal of this question is simply to define a coordinate for all potential addresses. I'm assuming all potential addresses are integers between the minimum and maximum values for the centerline. For this project it is ok that they are not real or verified.

For alittle bit more insight into the project and its goals, often with inferential spatial statistics distributions are created via creating random locations within your study boundary and calculating distributions from those random samples. Often times the universe of potential locations is dictated by residential addresses (i.e. a potential location has to be defined by whatever geocoder you are using, it can't be everywhere within the study boundary). Since these residential addresses are likely clustered in space, I want to see if functions like Ripley's K tend to be biased in using complete spatial randomness to define the sampling distribution as opposed to using the actual locations of addresses (defined apriori by whatever you are using to geocode those addresses).

I know that if I use a parcel based geocoder that pretty much all of these concerns are circumvented. I am still interested in extending the analysis to street centerline files though because as far as I'm aware street centerline files are still regularly used to geocode addresses, and so such a question is still pertinent. I plan on conducting the analysis with a parcel based geocoding system as well.

As far as DavidF's and Mark Ireland's concern about the "realness" of the addresses if I use every integer value, it is a legitmate concern given the nature of the project. If I use more addresses than really exist, it will induce artificial clustering of addresses. I would likely conduct post hoc specificity tests using Both David's suggestion (addresses only every 40 feet by a sampling strategy) and Mark's suggestion (verifying real addresses from some outside source). In either case I still want to be able to extract all the address points from the street centerline file. I am not verifying the accuracy of the geocoder from the onset, I am seeing if complete spatial randomness conforms to the distribution of points that a geocoding process could potentially give. I understand the errors associated with street centerline files, but I want to mimic how geocoding is actually conducted with a street centerline file.

Also if people can point to some literature that they think would be pertinent I would appreciate it. I know of a bit of work done on the accuracy of geocoding, but I have not seen any work addressing this particular issue.

Best Answer

Presumably you'd need to calculate the length of the road, divide by the number of addresses, and create an offset point for the addresses according to this result.

This might be useful for some tasks, but to my mind this doesn't really help address the issue of randomness vs reality, since what you're creating is not a reliable representation.

For example, there might be a single high-rise building with 100 units. You could end up spreading these along the road segment, when they are a single cluster. Also, if you have a segment of road that isn't straight, then this is going to throw things out (as well as make the task of calculating the offset more difficult).

Perhaps the best way is to get commercial geocoded data - like post office address data. I believe that there is such data for both the UK and Canada (not sure where you are) and as an academic study the access to this info should be free of charge (which is good because the prices to purchase are astronomical). I would think this would be more reliable than geocoding via an online service.

Related Solutions

[GIS] How to return individual address components (city, state, etc.) from GeoPy geocoder

Lubar, I saw your post at Stack Overflow but am going to post a similar answer here for consistency. It's a good question. I work in the address verification industry and have tackled your kind of problem before.

I linked to this Stack Overflow question in a comment; and it's important to know that there's really no guarantee about the format of complete freeform street addresses. As mentioned in the linked post, complete addresses can look like any of these:

1) 102 main street Anytown, state

2) 400n 600e #2, 52173

3) p.o. #104 60203

4) 1234 LKSDFJlkjsdflkjsdljf #asdf 12345

5) 205 1105 14 90210

(The reasons are explained in the linked post.) I realize that GeoPy returns addresses in a certain format -- depending on the geocoder used (which resulting format is out of GeoPy's control), but addresses can look all sorts of ways within a certain component (like having commas), and it's important to know that standardized addresses don't have commas (according to USPS Publication 28).

I helped work on an API just recently called US Street Address API from SmartyStreets; it was just upgraded to support geocoding and single-line address parsing.

GeoPy is designed to geocode, not parse into components (that task is actually really difficult for reasons I won't get into here). The US Street Address API will, however, componentize the address and return coordinates and other information about the address, and only if the addresses are real; no "guessed" results.

To parse a single-line address into components using Python, simply put the entire address into the "street" field:

import json
import pprint
import urllib

LOCATION = 'https://api.smartystreets.com/street-address/'
QUERY_STRING = urllib.urlencode({ # entire query sting must be URL-Encoded
    'auth-token': r'YOUR_API_KEY_HERE',
    'street': '1 infinite loop cupertino ca 95014'
})
URL = LOCATION + '?' + QUERY_STRING

response = urllib.urlopen(URL).read()
structure = json.loads(response)
pprint.pprint(structure)

The resulting JSON object will contain a components object which will look something like this:

"components": {
        "primary_number": "1",
        "street_name": "Infinite",
        "street_suffix": "Loop",
        "city_name": "Cupertino",
        "state_abbreviation": "CA",
        "zipcode": "95014",
        "plus4_code": "2083",
        "delivery_point": "01",
        "delivery_point_check_digit": "7"
}

The response will also include the combined first_line and delivery_line_2 so you don't have to manually concatenate those if you need them.

[GIS] Assigning centerline address ranges using ArcMap

Coming from someone who formerly worked with 9-1-1 data, the best answer would be: Use a combined geocoder. By using a combined geocoder, you can use your point data as a first pass, then fallback to street centerlines for a second pass if you got no hits with the point data.

Because you already have good point locations representing actual address locations, you should be able to geocode your tabular address "text" to the your point database. Of course, because new building/addresses are popping up all the time, it is nearly impossible for you to always have an up-to-date point layer.

Therefore, for your street centerlines, you are better off trying to capture the "extrapolated range" of the potential addresses on a block (ie- 100-199). This will capture all of the new addresses that have not yet been added to your point database.

You should also consider having attribute fields for the Left and Right side of your street. Typically though your ranges would look something like Left (100-198), Right (101-199). However, some jurisdictions, you might see that the extrapolated range for the left is 100-198, whereas the right is 101-149 (where an existing house is on the corner @149, and another across the street is @198). By having a left & right field, you are increasing the accuracy of the geocode for the right side.

You may want to take a look the the following NENA Standards documents for GIS data for additional standards/ideas/examples:

Best Answer

Related Solutions

[GIS] How to return individual address components (city, state, etc.) from GeoPy geocoder

[GIS] Assigning centerline address ranges using ArcMap

Related Question