Ultimately, because all extensive features are given by curves and those curves are approximated by line segments (when projected) or geodesic segments (when not projected), any errors are due to the fact that a line segment joining two projected points likely deviates (at least a little) from the projection of a geodesic between those two points. The amount of deviation obviously depends on the projection. It also depends on how the curve is approximated by a sequence of segments (its "digitized," vector version). This makes it impossible to give a general answer or formula.
The problem and a process to assess it is illustrated in the question and answers at Why is the 'straight line' path across continent so curved? . The question shows two points and a piecewise approximation to the projected geodesic between them. Obviously this approximation differs from the line segment connecting the points (in the map). Indeed, the problem was found by obtaining a sequence of points along that geodesic and projecting them, then comparing those points (vertices of the broken curve) to the straight line between the projected points. The error in this example is extreme. Errors can become practically infinite when features come close to or cross singularities of the projection: a feature that crosses one of the poles in a Mercator projection is a good example.
My reply to that question maps the same feature with a different projection. In the new projection, there is no error even when the original geodesic is not broken. Although this is not shown, one demonstration that the new projection is free of error for this geodesic would be to break the geodesic into a sequence of points and separately project them: they would lie along the same projected line. (The new projection makes errors for other geodesics, of course: no projection can render all geodesics free of some kind of metric distortion.)
Thus, a general process for evaluating error consists of making two maps of the features: the original map and another one in which all sufficiently long segments of the digitized features are first subdivided into smaller segments. Potential errors occur where corresponding features in those two maps disagree substantially.
In general, when features are digitized at a scale appropriate for the application and a suitable projection is shown, calculations made in projected coordinates are adequate. When features extend over large parts of the globe, it often becomes necessary to change projections (on the fly) to accommodate specific analytic needs, such as preserving area or relative angles. If you know that all analyses will be based on geodetic coordinates (that is, using spherical geometry) and you exploit that fact when creating the digitization (such as when portions of features fall along long geodesics), you will likely need the "Geodetic Extender" option. When features extend over large parts of the globe, there may be few or no projections that adequately render all the features with sufficient accuracy for the intended analyses and again the "GE" option is worthwhile. Good examples are climatological models and oceanographic research covering multiple ocean basins. In all other cases--including almost every project at a continent level or smaller--this option is probably not necessary.
Best Answer
This summarizes my understanding of some of the basic ideas. Because it is hard to find all of them clearly described and summarized in one place, I could be wrong or misleading about some of them: comments and corrections are welcome.
"Geoids" are approximations to a surface of gravitational equipotential.
(NASA)
To get a sense of what is gained relative to a sphere or ellipsoid, note that
The difference in apparent elevations between a spherical model and a good ellipsoid is up to two dozen kilometers. This translates to maximum positioning discrepancies of about 22 kilometers. The relatively large amount of positioning discrepancy occurs because there is a systematic distortion of the sphere relative to the ellipsoid: it attains one extreme at the poles and another extreme at the Equator.
The difference in apparent elevations between a good ellipsoid and a geoid is typically less than 100 meters (about 0.1 kilometers). This is not a systematic difference: it varies a lot across relatively short sections of the earth (on the order of hundreds of kilometers). Consequently, the maximum horizontal positioning discrepancy resulting from any hypothetical geoid-based projection is likely on the order of meters or less (usually much less except perhaps over large, carefully chosen areas).
However, the deflection of the geoid (which is the amount by which true gravitational vertical direction varies) reaches up to about an arc-second, which makes it unsuitable for any kind of very high-accuracy mapping based on measuring latitude in terms of a local upward-pointing angle. An arc-second of deflection translates to almost 30 meters on the ground, and such deflections can vary from one extreme to the other over just a few hundred kilometers.
In return for squeezing out that last 0.5% of accuracy in describing how the geoid varies from the ellipsoid, you need hundreds to hundreds of thousands of parameters compared to two to describe an ellipsoid. Yes, it is mathematically possible to define a projection based on a geoid instead of an ellipsoid. [See "Coordinate charts" on pp 4-5 of this text, for instance. The modern mathematical definition of smooth curved surfaces, like a geoid, is based on a set of projections. The Implicit Function Theorem guarantees such projections exist for the geoid.] Computation would be, to say the least, inefficient (although it could be sped up by interpolation in precomputed tables). When necessary, the difference in vertical positioning can be computed after ellipsoid-based projection in terms of the geoid parameters or by interpolating in a precomputed grid of geoid values.
A serious potential problem with basing map projections on a geoid as reference surface is that the geoid is constantly changing worldwide. It will change with changes in sea level, for instance.
Because nowadays much geopositioning is done in geocentric coordinates, rather than by means of gravitational-based triangulating devices (such as levels), use of a geoid is practically irrelevant: an ellipsoid--however well it might or not be related to gravity, sea level, or the actual shape of the earth--serves as a reasonably stable reference surface relative to which everything else can be located and mapped. The geoid is then described relative to this reference. Its description is used in mapping primarily to permit GPS satellites to improve their positioning accuracy.