I've written code for exactly this problem, but at the moment I cannot find it :-/.
[Used in a paper: "On reconstruction of polyhedra from slices," Int. J. Comput. Geom. & Appl., 6(1) 1996: 103-112.] However, it is easy enough to describe.
First, you need robust code to decide if a point is above, below, or on the plane
determined by one triangle.
See, e.g., the code described in Computational Geometry in C, for this low-level task
(which amounts to computing the signed volume of a tetrahedron),
and others following; or in many other equivalent sources.
If all three points of one triangle are to one side of the other,
the intersection is empty.
Let vertex $a$ of triangle $\triangle abc$
be above triangle $t'=\triangle a'b'c'$ and $b$ and $c$ below.
(I'll ignore the "on" cases for simplicity, but of course you must deal with them carefully.)
If $t$ and $t'$ intersect, then it must be that an edge of one intersects the other;
this is the key observation (also made by anon in another answer).
So either $ab \cap t'$ or $ac \cap t'$ is nonempty, or the analogous conditions with
the role of the triangles reversed.
Checking these conditions requires (again robust) code to determine if a segment
intersects a triangle. This can be computed by solving simultaneously a parametric equation for the segment and the plane containing $\triangle t'$, obtaining the
point $p$ of intersection, and determining if $p$ falls within $\triangle t'$.
The latter is a separate and easy task.
So the whole can be accomplished by a number of confined, controlled, elementary computations.
Create vectors pointing along each line by computing $(x_2,y_2,z_2)-(x_1,y_1,z_1)$ for both pairs. Make them into unit vectors by dividing them by their lengths. Call these two unit vectors $u$ and $v$.
Then you can use the inner product identity: $\langle u,v\rangle=\cos(\theta)$, where $\theta$ is the angle between the two vectors.
You want to create two small thresholds around 1 and -1. When the dot product is close to 1, this means that the vectors are very nearly pointing in the same direction, and when the dot product is nearly -1, they are very close to pointing in opposite directions. In both cases, they are "nearly parallel".
Best Answer
Find the intersection points of both rectangles in the plane of the other. You achieve that by plugging the coordinates of the vertices in the implicit equation of the plane and detecting a change of sign.
If you obtain four intersection points, which define two line segments, it suffices to test if these segments overlap (and where if you need the endpoints). Project the points on the direction of the common line (cross-product of the normal vectors), and you will get scalar values corresponding to abscissas along that line.
Checking for overlap in 1D is trivial.