A good reference is the recent book Calculus of Variations and Optimal Control Theory: A Concise Introduction by Daniel Liberzon http://press.princeton.edu/titles/9760.html
Optimal Control Theory: An Introduction is an older, somewhat more engineering-oriented book, reprinted by Dover Books. It is a good classic.
Start with 1 or 2 of these. Most other texts are heavier reading than both of these together.
The best ways of thinking about the effect of zeros are the Root Locus and the Bode diagram.
The zeros are the end point of the root locus. Thus a system with zeros whose real parts are not negative will become unstable if a feedback loop with sufficiently high gain is closed. Such zeros impose limits on the performance of a stabilizing controller.
Compared to a minimum-phase system whose Bode magnitude plot is identical, a nonminimum-phase system, that is, one with unstable zeros, will have extra phase of up to 180 degrees for each unstable zero. The extra phase corresponds to extra turns around the origin in the Nyquist diagram. That also explains how right half plane zeros make a system harder to control by feedback.
Additionally, zeros cannot be be moved by feedback, the way poles can be assigned. Zeros with nonnegative real part cannot be canceled, because the cancellation would be unstable. Thus the designed has to live with them, as a fundamental limitation on our ability to design a feedback controller.
(I should add that time-domain linear quadratic control is of limited use in understanding the role of zeros.)
Best Answer
First, consider the following first order transfer function:
$$ \frac{X(s)}{U(s)} = \frac{a}{s - a} $$
where $a \in \mathbb{C}$ is the system pole. If we observe the behavior of the system in time we have
$$ \dot{x}(t) = e^{a t} (u(t) - x(t))$$
Since $ a $ is complex we can write it as $ a = b + j c $ where $ b $ is the real part of $ a $ and $ c $ the imaginary part. Then the system becomes:
$$ \dot{x}(t) = e^{b t} e^{j c t} (u(t) - x(t))$$
Note that $ e^{j c t} $ will cause the system to oscillate, while $ e^{b t} $ will determine how (and if) the $x$ will converge to $u$.
If $ b < 0 $ the system will go to zero since $ e^{b t} \rightarrow 0 $ when $ t \rightarrow \infty $. Meanwhile, if $ b > 0 $ the system will diverge since $ e^{b t} \rightarrow \infty $ when $ t \rightarrow \infty $.
Note that $ c $ does not play a role here. So independently of the imaginary part, the real part of the pole needs to be negative for stability.
For the case $ b = 0 $ the system will neither converge nor diverge, however stability is defined by strict convergence, so $ b = 0 $ is not stable (attention to the choice of words, it may or may not be unstable, I recommend you to read more on marginal stability for this).