[GIS] Python – Convert csv with unknown number of columns to shapefile

csvlooppyshppythonshapefile

I have this script that converts a csv file to a shapefile. In this case I am specifying manually the name of columns header. Since now I have files with a big and always different number of columns, I'd like to automatize the script, i.e. making it reading the number of columns and their respective header. Any suggestions?

These are the first 3 line of the csv file:

id  longitude       latitude    D_20150403  D_20150521
1   -16.01746368    17.95987892 11.89926434 15.17788696
2   -16.01752663    17.95999527 16.99330902 15.84250259

    # initialize data lists
ID_n,lon,lat,D_20150403,D_20150521=[],[],[],[],[]

# read data from csv file and store in lists

with open(in_file, 'rt') as csvfile:
    r = csv.reader(csvfile, delimiter=',')
    for i,row in enumerate(r):
        if i > 0:
            ID_n.append(int(row[0]))
            lon.append(float(row[1]))
            lat.append(float(row[2]))
            D_20150403.append(float(row[3]))
            D_20150521.append(float(row[4]))


# set up shapefile writer and create empty fields
w = shp.Writer(shp.POINT)
w.autoBalance = 1
w.field('ID','C')
w.field('Lon','F',10,8)
w.field('Lat','F',10,8)
w.field('D_20150403','F',10,8)          
w.field('D_20150521','F',10,8)      


# loop through the data and write shapefile geometry and attributes
for i,j in enumerate(lon):
    w.point(j,lat[i])
    w.record(ID_n[i],j,lat[i],D_20150403[i],D_20150521[i])

# save output shapefile
w.save(out_shp)

# create output projection file
prj = open(out_prj, "w")
epsg = spatialreference = "GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119522E-09;0.001;0.001;IsHighPrecision"
    #getWKT_PRJ("4326")
prj.write(epsg)
prj.close()

I tried with this, which seems to replicate the data structure but something is wrong in the creation of the shapefile. Any idea?

# read data from csv file and work on the header

with open(in_file, 'rt') as csvfile:
    rh = csv.reader(csvfile, delimiter=',')
    field_names_list = next(rh)
dim_header = [[]]*len(field_names_list)
dic = dict(zip(field_names_list,dim_header))
print(field_names_list)

n_col = len(field_names_list)
print(n_col)

with open(in_file, 'rt') as csvfile:
    r = csv.reader(csvfile, delimiter=',')
    for i,row in enumerate(r):
        if i > 0:
            for j in range(0, n_col-1):
                dic[field_names_list[j]].append(row[j])

# set up shapefile writer and create empty fields
w = shp.Writer(shp.POINT)
w.autoBalance = 1
#write ID
w.field(dic[field_names_list[0]],'C')
#write all the other fields
for j in range(1, n_col-1):
    w.field(dic[field_names_list[j]],'F',10,8)

# loop through the data and write shapefile geometry and attributes
for i,j in enumerate(dic[field_names_list[0]]):
    w.point(j,dic[field_names_list[0]][i])
    for k in range(0, n_col-1):
        w.record(dic[field_names_list[k]][i])

# save output shapefile
w.save(out_shp)

# create output projection file
prj = open(out_prj, "w")
epsg = spatialreference = "GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]];-400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119522E-09;0.001;0.001;IsHighPrecision"
    #getWKT_PRJ("4326")
prj.write(epsg)
prj.close()

Best Answer

For that, the easiest solution is to use GeoPandas (many examples in GIS SE)

1 - read the csv file with Pandas as a DataFrame
2 - compute the geometry with Shapely (many examples in GIS SE)
3- convert the DataFrame to a GeoDataFrame
4 - save the GeoDataframe as a shapefile

import geopandas as gpd
import pandas as pd
# read the csv file
test = pd.read_csv("test.csv")
test.head()
id  longitude   latitude  D_20150403  D_20150521
0   1 -16.017464  17.959879   11.899264   15.177887
1   2 -16.017527  17.959995   16.993309   15.842503
# compute the geometry
from shapely.geometry import Point
test['geometry'] =test.apply(lambda row: Point(row['longitude'], row['latitude']), axis=1)
# convert to a GeoDataFrame
geo_test = gpd.GeoDataFrame(test,geometry=test.geometry)
# save the resulting shapefile
geo_test.to_file("result.shp")

Related Solutions

[GIS] Using pyshp to convert .csv file to .shp

The pyshp module is a bit tricky to get the hang of, but really useful once you get it going. I've written a script that reads in a csv of the example data and writes out a shapefile with the data stored as attributes of the correct datatypes. The pyshp/xbase datatyping has always been tricky for me until I found this user guide for the xbase format and as a result of this question I have written a small note on my blog regarding the relevant pyshp datatypes, part of which I have pasted below:

C is ASCII characters
N is a double precision integer limited to around 18 characters in length
D is for dates in the YYYYMMDD format, with no spaces or hyphens between the sections.
F is for floating point numbers with the same length limits as N
L is for logical data which is stored in the shapefile's attribute table as a short integer as a 1 (true) or a 0 (false). The values it can receive are 1, 0, y, n, Y, N, T, F or the python builtins True and False

The full listing is as follows:

import shapefile as shp
import csv

out_file = 'GPS_Pts.shp'

#Set up blank lists for data
x,y,id_no,date,target=[],[],[],[],[]

#read data from csv file and store in lists
with open('input.csv', 'rb') as csvfile:
    r = csv.reader(csvfile, delimiter=';')
    for i,row in enumerate(r):
        if i > 0: #skip header
            x.append(float(row[3]))
            y.append(float(row[4]))
            id_no.append(row[0])
            date.append(''.join(row[1].split('-')))#formats the date correctly
            target.append(row[2])

#Set up shapefile writer and create empty fields
w = shp.Writer(shp.POINT)
w.autoBalance = 1 #ensures gemoetry and attributes match
w.field('X','F',10,8)
w.field('Y','F',10,8)
w.field('Date','D')
w.field('Target','C',50)
w.field('ID','N')

#loop through the data and write the shapefile
for j,k in enumerate(x):
    w.point(k,y[j]) #write the geometry
    w.record(k,y[j],date[j], target[j], id_no[j]) #write the attributes

#Save shapefile
w.save(out_file)

I hope this helps.

[GIS] Using PyShp to convert polygons in .csv to .shp files

Your approach is good, but you could make things clearer using dictionaries instead of list and the csv module allows it.
Moreover, your script use two loops while it is possible to simplify using only one (the second loop is redundant, one line of the csv file = one record of the shapefile).

1) With dictionaries:

Reading the csv file:

with open('your.csv', 'rb') as f:
    reader = csv.DictReader(f)
    for row in reader:
        print row
 {'xr': '22.5', 'yb': '31.353634', 'name': 'some Name', 'xl': '-25.3125', 'yt': '47.517193'}
 {'xr': '-0.703125', 'yb': '74.40216', 'name': 'another Name', 'xl': '-103.359375', 'yt': '80.87282'}

You can now use row['xr'] or row['name'] instead of row[n] (more explicit)

So your script becomes:

import csv
polyName, polyPart = [],[]
with open('your.csv', 'rb') as f:
   reader = csv.DictReader(f)
   for row in reader:
       bl  = [float(row['xl']),float(row['yb'])]
       tl  = [float(row['xl']),float(row['yt'])]
       br  = [float(row['xr']),float(row['yb'])]
       tr  = [float(row['xr']),float(row['yt'])]
       parr = [tl, tr, br, bl, tl]
       polyName.append(row['name'])
       polyPart.append(parr)

Writing the polygon shapefile

If you look at PyShpDocs you can see that:

A polygon is defined by:

w = shapefile.Writer(shapefile.POLYGON)
w.line(parts=[[[1,5],[5,5],[5,1],[3,3],[1,1]]])

and the script, as you wrote is:

 import shapefile
 # create the Polygon shapefile
 w = shapefile.Writer(shapefile.POLYGON)
 # the field
 w.field('name','C',maxStringLength)
 # write the polygons in the shapefile
 for part,name in zip(polyPart, polyName):
      w.poly(parts=[part])
      w.record(name) 
 #save the shapefile
 w.save('your.shp')

2) Final solution using only one loop

But the second loop is not necessary here: you can do it all with one loop (reading the csv file and writing the shapefile without using the polyName and polyPart lists).

w = shapefile.Writer(shapefile.POLYGON)
w.field('name','C',50)
with open('your.csv', 'rb') as f:
    reader = csv.DictReader(f)
    for row in reader:
        bl  = [float(row['xl']),float(row['yb'])]
        tl  = [float(row['xl']),float(row['yt'])]
        br  = [float(row['xr']),float(row['yb'])]
        tr  = [float(row['xr']),float(row['yt'])]
        parr = [tl, tr, br, bl, tl]
        w.poly(parts=[parr])
        w.record(row['name'])

w.save("your.shp')

Result in QGIS:

enter image description here