I've got an FME Workbench (version 2015.0) which searches for shapefiles in a folder tree. The shapefiles are contours which are split into tiles for a particular area. The workbench is fairly simple, it filters based on certain attributes then combines the output into a merged shapefile containing all the contours.
However, the workbench fails because the resulting shapefile is too big. I get the following error:
Error – Cannot write out a file larger than 4.295GB. Split your data
into two or more shape files
Of course, I didn't necessarily know beforehand that my output would be so large. Is there any way for FME to detect that the file will be too big and somehow automatically write the output to two or more shapefiles? Perhaps with the suffix "_1.shp", "_2.shp"?
Best Answer
No, there's no functionality that detects this and splits the file up for you. It's already registered as an enhancement request with our developers, and the reference number is PR#34128. I added a note linked to this question.
NB: Technically I believe the limit specified by the format is 2.14gb, so even 4.29 is stretching the format more than it should really be.
Of course, the question is whether a Shape dataset of that size is a good idea in the first place! Maybe you could use a different format?
If you are stuck on Shape, then I'd do as others have suggested. The ModuloCounter transformer is a good way to divide data (on a non-geographic basis) or you could split the data with either the Tiler transformer or the Clipper.