Automatic generalization algorithms are not yet a complete solution, they still often make decisions a careful cartographer would abhor. Natural Earth has a good background article
on how their approaches to generalizing features, a multi-step manual process. Axpand claims the Swiss Topography Office had an 80% success rate with automatic generalization when using their software with topographic maps.
You've mentioned in a few responses that you don't like the 'bony' geometries produced by many automated solutions, could you explain what aspects of the data you'd like to retain? Are you interested in positional accuracy of the results, or more in the aesthetics? That will help give us a sense of where a solution might lie.
The first part of your question doesn't really make sense. Topology is the study of adjacency, connectivity, and containment. With topology you examine the structure and (non-geometric) shape of things. You wouldn't, for example, use topology to "simplify" a torus into a sphere by filling in the hole.
A better question would be:
What algorithms exist for simplifying a triangulated 3D surface while preserving its topological structure?
You already found a cornucopia of references in a link from user @dtlarek in response to your question on http://math.stackexchange.com.
Additionally, you could look at the excellent GTS library. It provides two surface simplification algorithms:
It's not clear to me that these methods will preserve topological structure in all cases. You can see an example of what one method does in this video of the reverse of the surface simplification of a horse.
If Python is your style, then you can access GTS's functions through the PyGTS Python bindings.
EDIT:
Your picture (in the comments) does help to describe the problem, but that problem is not one of topology. It's true that the mesh structures in each of the heads in your image are not homeomorphic - the meshes are topologically different. However, you are asking about something different: how to simplify the mesh while preserving the appearance. You want the thing to "look" nearly the same, but have fewer edges and vertices. The image below gives another example.
The hands image is from the CGAL Triangulated Surface Mesh Simplification page. That page describes a well-developed and widely used approach to mesh simplification that is a "process of reducing the number of faces used in the surface while keeping the overall shape, volume and boundaries preserved as much as possible" (from the CGAL site).
Just remember, unless you are concerned with the topology of the mesh itself, this is not a question of topology.
Best Answer
The Java Topology Suite includes a TopologyPreservingSimplifier. The code does not include a reference for the implementation, beyond stating that it operates in a similar manner to Douglas-Peucker, with additional constraints on altering the topology.
This functionality has made it into the Java-to-C++ translation of JTS, libgeos, which is further exposed to python in the excellent Shapely python library.
I've had good luck with the topology preserving simplifier when operating on polygons with holes in tricky but non-diabolical inputs.
Of course there are nuances in the simplification requirements based on the application. For instance, in simplifying shorelines for a hydrodynamic model (my application), peninsulas are important to retain but small inlets can be eliminated. In this case, an approach which worked fairly well was along the lines of
which can also be accomplished using Shapely, or most any GIS program for that matter. I'm not suggesting this as a general (or efficient!) approach, but instead to say that not all methods are equally suited to all applications.