[GIS] Python ogr2ogr language character encoding

encodingexportogr2ogrpython

In python script i'm exporting .shp files to .csv/.gpx/.kml

import ogr, ogr2ogr
...
ogr2ogr.main(["","-f", "CSV", csvfile, shpfile, "-lco", "GEOMETRY=AS_WKT", "-lco", "SEPARATOR=SEMICOLON"])
ogr2ogr.main(["","-f", "GPX", "-dsco", "GPX_USE_EXTENSIONS=YES", gpxfile, shpfile])
ogr2ogr.main(["","-f", "KML", kmlfile, shpfile])

The files are exported, but the language(Latvian) specific string field characters are incorrect. Shape files have string field "LABEL" with a value i.e. "marķieris".

In case of .kml file it is turned into "markieris", and in case of .csv and .gpx it is turned into "maríieris". All exported files have utf-8 (without BOM) encoding.

So far i had no luck finding if i have to pass some kind of additional parameter, or do something else to fix this language issue. Scrip is executed in Windows.

Best Answer

You should try with the following configuration option:

--config SHAPE_ENCODING="ISO-8859-4"

because ISO-8859-4 is the ISO alias of Latin 4.

Latin 4 introduces letters for Estonian, Latvian, and Lithuanian.

Related Solutions

QGIS – How to Fix Shapefile’s Character Encoding Using ogr2ogr on the Command Line

Quoting from the GDAL documentation for the Esri Shapefile driver:

An attempt is made to read the LDID/codepage setting from the .dbf file and use it to translate string fields to UTF-8 on read, and back when writing. LDID "87 / 0x57" is treated as ISO8859_1 which may not be appropriate. The SHAPE_ENCODING configuration option may be used to override the encoding interpretation of the shapefile with any encoding supported by CPLRecode or to "" to avoid any recoding. (Recoding support is new for GDAL/OGR 1.9.0)

Depending on which GDAL/OGR version you are using, ogr2ogr may be trying to translate your data to UTF-8 or not doing nothing at all.

So you would do either:

ogr2ogr --config SHAPE_ENCODING "UTF-8" -f "MySql" MySql:"basemap,host=127.0.0.1,user=myUser,password=myPass,port=3306" -lco engine=MYISAM "C:/path/to/data/tl_2012_us_county.shp" -nln county -nlt "geometry" -s_srs EPSG:4269 -t_srs EPSG:3857

ogr2ogr --config SHAPE_ENCODING "" -f "MySql" MySql:"basemap,host=127.0.0.1,user=myUser,password=myPass,port=3306" -lco engine=MYISAM "C:/path/to/data/tl_2012_us_county.shp" -nln county -nlt "geometry" -s_srs EPSG:4269 -t_srs EPSG:3857

Finally, check that your MySQL database is using UTF-8 and not Latin-1.

[GIS] Set encoding when using ogr2ogr to export from Oracle db to MapInfo table

The OGR driver for Oracle (OCI driver) accesses the database just like any other client, and so it's behavior regarding character set is the same as for any other client.

In particular, the character set in which the database will send the strings to the client (here ogr2ogr via its OCI plugin) is influenced by the setting of the NLS_LANG environment variable. On windows, that variable (like others) is in the registry. It may be that this variable is incorrectly set up in your environment, or maybe it is missing entirely if you use the light-weight Oracle "thin client".

Local characters like æøå is not intact in the output.

Do you mean they come out as "?" characters ? If so that probably means that you do not have any NLS_LANG set, and so the client library applies the default of US7ASCII (which is unable to encode those characters).

Confirm first that your database is correctly setup with the proper character set. What does the following show ?

SQL> select value from nls_database_parameters where parameter='NLS_CHARACTERSET';

If your database is on Windows and you do have it set up for ISOLatin1 then you should get something like WE8MSWIN1252. If it is on Unix/Linux (also ISOLatin1) then you should see WE8ISO8859P1.

Try the following: before running ogr2ogr set the following:

SET NLS_LANG=AMERICAN_AMERICA.WE8MSWIN1252

That will make sure the OCI plugin gets the strings in ISOlatin1. And you then can only hope that ogr proper does not play any tricks with those strings and passes them on to the Mapinfo plugin (hoping too that that also lets them through unadulterated).

PGCLIENTENCODING obviously has no effect: it is only for PostgreSQL

OGR_FORCE_ASCII is also best left out: according to the doc, it will replace all non-ASCII7 characters with "?" - precisely what you do not want. But then again it only applies to XML and derived formats - not your case anyway.

Best Answer

Related Solutions

QGIS – How to Fix Shapefile’s Character Encoding Using ogr2ogr on the Command Line

[GIS] Set encoding when using ogr2ogr to export from Oracle db to MapInfo table

Related Question