Assuming you are getting the "Invalid Topology" error you need to make sure that all geometry in this feature class is correct. Make sure the layer is in a File Geodatabase Feature Class format. Run the "Repair Geometry" then try DISSOLVE again.
Dissolve can create VERY large features from a large amount of small features and it can be a very memory intensive operation and thus take a very long time depending on the complexity of the input feature class and the dissolve parameters.
There are also other considerations when running dissolve explained in detail in the Dissolve Help and the How Dissolve Works.
When you feel that you have complied with all the usage requirements for the dissolve operation as outlined in the Help Usage article, you can submit a request for Support Assistance with ESRI. They have been very quick to respond lately. (within 24 hours) They will run the same steps on their end and provide you with solutions. You will need your ESRI customer number to request support.
Your problem is likely because of what you have included in the group by field.
ST_UNION is an aggregate function, meaning that it is dissolving based on what you specify as the GROUP parameter.
Here is what you entered:
SELECT c.fid, ST_Union(c.boundaryshape) FROM c Group by c.fid,c.boundaryshape;
According to this, you are grouping by your fid, which is probably a unique identifier, and also by boundaryshape, which is the geometry. The problem with this is that it is highly unlikely that the features sharing a unique ID, are going to have identical geometries. Therefore, the ST_Union is probably going to end up with something like the following:
Input:
fid boundaryshape
1 11
1 12
1 11
1 13
2 12
2 13
2 12
3 5
3 2
3 4
Output:
fid boundaryshape
1 (11 + 11)
1 12
1 13
2 (12 + 12)
2 13
3 2
3 4
3 5
In this case, you should only be dissolving based on your unique identifier, unless you actually have a number of duplicate geometries with the same fid, that you are trying to get rid of. Even in that situation though, you should only need to group based on the fid, and the union will take care of the geometry duplication.
I would try the following:
SELECT c.fid, ST_Union(c.boundaryshape) FROM c GROUP BY c.fid;
That should give you the desired result of dissolving all features based on the common attribute fid.
Here is the help document at the PostGIS site: ST_Union
See the first example listed at the bottom and notice that it does not include the geometry in the group by section.
Best Answer
Thanks to prompting by thehealingprocess, I did investigate a little more and the GP messages indicate the proper error is actually "Workspace or data source is read only"...which a bunch of others have encountered here as a bug at 10.1: http://forums.arcgis.com/threads/80394-Error-“Workspace-or-data-source-is-read-only.”-from-file-geodatabases. Now to see if I can implement one of those changes to fix it...
Didn't think to check the messages in the first instance since the GP ran just fine in ArcMap, just not in code.
UPDATE and FINAL ANSWER: I contacted ESRI support and the manifestation I was seeing - the code didn't work but executing manually in ArcMap did - prompted him to have me try ExecuteAsync instead of Execute to try to avoid conflicts with background processes. Apparently ExecuteAsync is effectively what ArcMap uses. When I did this, the tool completed successfully. So, there is a bug (he referenced NIM 094612), but the workaround is to use ExecuteAsync rather than Execute. Hope this helps someone else!