There's a distinct possibility that ArcScene is running up against the default Win32 process limit of 2GB. Originally, due to this limitation, ESRI was designing their 3D products to max out at around 20 million points.
Update: ESRI support note on the TIN size limitation (~15 million, dated 2008)
Approach
You either need to solve the system problem or perform a substitute or work-around comparison. Because you don't provide details about the error, I conclude you're looking for a work-around.
Solution
Consider converting both TINs to raster format, where a little map algebra does the rest. For example, the RMS measure of difference between the TINs is obtained by subtracting one grid from the other, squaring the result, and taking the square root of its average value. The answer depends on the cellsize (and grid origin) you use, but as the cellsize grows small, the answer will converge to the one you would obtain using the TINs directly.
Tools
There are many tools and libraries to perform such comparisons. They are the bread-and-butter operations of raster GIS packages like Spatial Analyst, IDRISI, and GRASS.
Accuracy
Note that no such procedure, whether applied to the TINs or some other representation, can estimate accuracy (which concerns agreement between the TIN and reality): it can only compare one data set to the other. Both could be hugely inaccurate, yet in close agreement.
Interpretation
There is nothing unusual about interpreting these results: one inspects both maps and the difference map, looks for outlying values in their differences, characterizes the statistical distribution of the differences, assesses the spatial correlation of the differences, identifies locations where clusters of high or low differences occur, etc., just as always.
Best Answer
Constrained triangulation or boolean operation code should work. Promising places to start looking include
Michael Leonov's guide to 2D boolean operations. This is exactly what you're looking for, but it's a little old. Leonov sells a .Net library, Polyboolean.
CGAL's 2D Triangulation
MathTools.net guide to computational geometry algorithms
Google's computational geometry software directory
"Computational Geometry Software Libraries"