MATLAB: How to rotate a vector by a certain amount along a specific plane

rotation

I have three points in inertial space which is enough to specify a plane. I have a vector that lies in this plane, and would like to rotate this vector so that it becomes perpendicular with its original self, through a rotation about the plane. For example:
my points:
a=[1 .3 .5];
b=[0 0 0];
c=[4 5 6];
%plane passes through points a, b, and c, a is the vector to rotate
EDIT
I mean to say a rotation on the plane (rotating with the pivot being the origin 0,0,0)

Best Answer

To find the vector that is in the same plane as the points c, a, and b, which is orthogonal to a we must:
1) first find the vector that is orthogonal to the plane CAB that passes through point b.
2) Rotate a about this vector 90 degrees.
To do this we can use the procedure described by @Bjorn Gustavsson(vector g is the plane that is orthogonal to plane c,a, and b)
Then, using the rodrigues equation (Rodrigues' rotation formula - Wikipedia), we plug in a for v, and k for g with theta being pi/2 and finally we obtain our correct vector.