[GIS] Merging data using FME

fmefmeobjects

I am a newbie in FME and is trying to accomplish a similar implementation like this: Merging edited data back into ArcSDE master geodatabase?

But we are still in the proof of concept stage.So we are just starting of with 2 non-spatial databases (master and updated) being populated by FME using a single shapefile. Initially both database are equal. Also, we added a FeatureID column using the Counter transformer in FME.

We made a couple of updates to the updated database so that we can try sync up the updates to the master database. We are using this sample as reference: http://fmepedia.safe.com/articles/Samples_and_Demos/Creating-a-Unique-identifier-using-CRCCalculator.

During the population of the master database. We added a "CRC" column which we set as a varchar(100). Before writing to the master database, there is a CRCCalculator added so that we can populate the CRC column.

Syncing process:
– we have 2 readers which is set for the update db and master db.
– the update db will pass through a CRCCalculator first and will then be passed to a FeatureMerger as a supplier.
– the master db will pass through a AttributeRenamer for the CRC column and will then be passed to the same FeatureMerger as the requestor.

CRCCalculator parameters:
– Attributes = all of the columns
– Calculate CRC = Geometry and Attributes
– CRC output attribute = CRC

I have added an inspector in the "Complete" output port of the FeatureMerger and noticed that all of the CRC(from update db) and CRC_Master(from master db) are not equal, even though there are no changes to those rows. So FME is thinking that all of the data have changed.

What am i doing wrong in the syncing process?

Here is a screenshot of how the syncing process looks in FME:

enter image description here

UPDATE
– So i tried to test this again and added another CRCCalculator for the Master database. So now i can see 3 CRC columns in the FeatureMerger results. I noticed that the CRC value that was created in the translation matched and the CRC value that was saved in the Master database did not.

  • Now im wondering if the CRCCalculator changes something in every translation.

thanks.

Best Answer

I am going to try to answer your question, well, what I think is your question. What you have as a question is really a methodology for doing something, rather than stating what you're doing.

Here's what I think your situation is: You have a MASTER and an UPDATED dataset. You want to find out where there have been changes between MASTER and UPDATED and insert/update the MASTER dataset with the changes in the UPDATED dataset.

Here's how I'd implement that with FME:

  1. Get both datasets in (you have this, but note that there is something wrong with the MASTER on in your screenshot, since no features are being read
  2. Run them into a CHANGE DETECTOR transformer, with either all attributes selected, or just the ones that change
  3. Get the features from the Added and Deleted side and decide what to do with them! (Here you can use a featuremerger, or some SQL writers with appropriate settings.
Related Question