[GIS] Convert multiple CSV files to shapefiles using Python

csvpythonshapefile

At my internship, I have been asked to learn Python on the job. It's been a rough road never having coded before so please forgive me if my problem is elementary.

The Lightning CSV files I am trying to convert have over 400,000 rows.
The headers go as follows: mon, day, year, hr, min, sec, lat, lon, ht, type, Ip, Id

I found the answers to this question as potential solutions. I struggle to modify this code in order to convert the CSVs into shapefiles.

The path for the CSVs is C:\Users\zherran\Desktop\shp\enwest2015*.csv

The path where I want the shapefile C:\Users\zherran\Desktop\shp\shapefiles

import shapefile as shp
import csv

out_file = 'enwest3days.shp'

#Set up blank lists for data
mon,day,year,hr,min,sec,lat,lon,ht,type,Ip,Id=[],[],[],[],[],[],[],[],[],[],[],[]

# sample row mon,day,year,hr,min,sec,lat,lon,ht,type,Ip,Id
#            6,2,2015,7,27,16.6,41.5,-101.4,13789.5,1,-7900,26abb8a


#read data from csv file and store in lists
with open('enwest*.csv', 'rb') as csvfile:
    r = csv.reader(csvfile, delimiter=';')
    for i,row in enumerate(r):
        if i > 0: #skip header
            mon.append(int(row[0]))
            day.append(int(row[1]))
            year.append(int(row[2]))
            hr.append(int(row[3]))
            min.append(int(row[4]))
            sec.append(float(row[5]))
            lat.append(float(row[6]))
            lon.append(float(row[7]))
            ht.append(float(row[8]))
            type.append(int(row[9]))
            Ip.append(float(row[10]))
            Id.append(complex(row[11]))

#Set up shapefile writer and create empty fields
w = shp.Writer(shp.POINT)
w.autoBalance = 1 #ensures gemoetry and attributes match
w.field('X','F',10,8)
w.field('Y','F',10,8)
w.field('Date','D')
w.field('Target','C',50)
w.field('ID','N')
w.field('ID','N')
w.field('ID','N')
w.field('ID','N')
w.field('ID','N')
w.field('ID','N')
w.field('ID','N')

#loop through the data and write the shapefile
for j,k in enumerate(x):
    w.point(k,y[j]) #write the geometry
    w.record(k,y[j],date[j], target[j], id_no[j]) #write the attributes

#Save shapefile
w.save(out_file)

Best Answer

a) With the solution you use (Pyshp (shapefile), you need

1) to extract the fields of the csv file
2) to define the fields of the shapefile (w.field('Target','C',50))
3) to construct the geometry from the lon et lat fields ( w.point(float(i['x']),float(i['y'])) (see CSV to Shapefile )

b) With Fiona, it is easier but you have the same problem of field definition.
c) osgeo/ogr complicates the things...
d) Therefore, be modern, to convert directly csv files to shapefiles, the best solution is now Pandas, GeoPandas(uses Fiona) and Shapely. You do not have to worry about the fields and the fields definitions (pandas.DataFrame.from_csv).

from pandas import DataFrame
from geopandas import GeoDataFrame
from shapely.geometry import Point
# convert the csv file to a DataFrame
data = DataFrame.from_csv('enwest2015_1.csv', index_col=False)
# extract the geometry from the DataFrame
points = [Point(row['lon'], row['lat']) for key, row in data.iterrows()]
#convert the DataFrame to a GeoDataFrame 
geo_df = GeoDataFrame(data,geometry=points)
# save the resulting shapefile
geo_df.to_file('enwest2015_1.shp', driver='ESRI Shapefile')

For the paths, it is a pure Python problem

Related Solutions

[GIS] From text data create a FeatureClass with table with python

In version 9.3 there is a toolbox called Create Features from Text File, which can do what you require. The help files are quite detailed and are found here.

The following code sample should be simple enough to adapt to reflect your dataset:

# Create geoprocessing dispatch object
import arcgisscripting
gp = arcgisscripting.create()

# Set up inputs to tool
inTxt = r"C:\temp\StreamPoints.txt"
inSep = "." #the symbol used to indicate a decimal point
output = r"C:\temp\Streams.shp"

# Run tool
gp.CreateFeaturesFromTextFile(inTxt, inSep, output, "#")

UPDATE:

If you require numerous attributes of a range of types to be written to a feature dataset from a text file or csv file, it my be easiest to use pyshp. I wrote up an answer to a similar question about pyshp a few days ago and will reproduce the relevant parts of this answer below.

The pyshp/xbase datatyping has always been tricky for me until I found this user guide for the xbase format and as a result of this question I have written a small note on my blog regarding the relevant pyshp datatypes, part of which I have pasted below:

C is ASCII characters
N is a double precision integer limited to around 18 characters in length
D is for dates in the YYYYMMDD format, with no spaces or hyphens between the sections.
F is for floating point numbers with the same length limits as N
L is for logical data which is stored in the shapefile's attribute table as a short integer as a 1 (true) or a 0 (false). The values it can receive are 1, 0, y, n, Y, N, T, F or the python builtins True and False

There is a DateTime type as well which should allow you to store correctly formatted time data as well using the same syntax as the above datatypes.

Here is an example python script showing the use of the various pyshp datatypes being read in from lists, which could be populated with your data using the csv module:

import shapefile as shp

#Input parameters
out_file = 'GPS_Pts.shp'

#Set up blank lists for data
x,y,id_no=[1.0,2.4,3.9],[4.3,5.5,6.1],[55,16,10]
date,notes = ['20121012','20121012','20121011'],['This','is a','string']
logic = [True,False,True]

#Set up shapefile writer and create empty fields
w = shp.Writer(shp.POINT)
w.autoBalance = 1 #ensures gemoetry and attributes match
w.field('X','F',10,5)
w.field('Y','F',10,5) #float - needed for coordinates
w.field('Date','D') #date
w.field('Notes','C',50) #string
w.field('ID_N','N',12) #double
w.field('Logic','L',1)#boolean

#loop through the data and write the shapefile
for j,k in enumerate(x):
    w.point(k,y[j]) #write the geometry
    w.record(k,y[j],date[j], notes[j], id_no[j], logic[j]) #write the attributes

#Save shapefile
w.save(out_file)

[GIS] Using PyShp to convert polygons in .csv to .shp files

Your approach is good, but you could make things clearer using dictionaries instead of list and the csv module allows it.
Moreover, your script use two loops while it is possible to simplify using only one (the second loop is redundant, one line of the csv file = one record of the shapefile).

1) With dictionaries:

Reading the csv file:

with open('your.csv', 'rb') as f:
    reader = csv.DictReader(f)
    for row in reader:
        print row
 {'xr': '22.5', 'yb': '31.353634', 'name': 'some Name', 'xl': '-25.3125', 'yt': '47.517193'}
 {'xr': '-0.703125', 'yb': '74.40216', 'name': 'another Name', 'xl': '-103.359375', 'yt': '80.87282'}

You can now use row['xr'] or row['name'] instead of row[n] (more explicit)

So your script becomes:

import csv
polyName, polyPart = [],[]
with open('your.csv', 'rb') as f:
   reader = csv.DictReader(f)
   for row in reader:
       bl  = [float(row['xl']),float(row['yb'])]
       tl  = [float(row['xl']),float(row['yt'])]
       br  = [float(row['xr']),float(row['yb'])]
       tr  = [float(row['xr']),float(row['yt'])]
       parr = [tl, tr, br, bl, tl]
       polyName.append(row['name'])
       polyPart.append(parr)

Writing the polygon shapefile

If you look at PyShpDocs you can see that:

A polygon is defined by:

w = shapefile.Writer(shapefile.POLYGON)
w.line(parts=[[[1,5],[5,5],[5,1],[3,3],[1,1]]])

and the script, as you wrote is:

 import shapefile
 # create the Polygon shapefile
 w = shapefile.Writer(shapefile.POLYGON)
 # the field
 w.field('name','C',maxStringLength)
 # write the polygons in the shapefile
 for part,name in zip(polyPart, polyName):
      w.poly(parts=[part])
      w.record(name) 
 #save the shapefile
 w.save('your.shp')

2) Final solution using only one loop

But the second loop is not necessary here: you can do it all with one loop (reading the csv file and writing the shapefile without using the polyName and polyPart lists).

w = shapefile.Writer(shapefile.POLYGON)
w.field('name','C',50)
with open('your.csv', 'rb') as f:
    reader = csv.DictReader(f)
    for row in reader:
        bl  = [float(row['xl']),float(row['yb'])]
        tl  = [float(row['xl']),float(row['yt'])]
        br  = [float(row['xr']),float(row['yb'])]
        tr  = [float(row['xr']),float(row['yt'])]
        parr = [tl, tr, br, bl, tl]
        w.poly(parts=[parr])
        w.record(row['name'])

w.save("your.shp')

Result in QGIS:

enter image description here