You can do this with elementary vector algebra. Call $D = A \times B$, and then $C = B \times D$. $C$ is automatically orthogonal to $B$.
Of course, it's a little difficult to know that this is indeed the vector most like $A$. I reasoned this out using geometric algebra: there is a unique plane denoted $iB$ that is orthogonal to $B$ (and thus contains all vectors orthogonal to $B$). The vector in $iB$ closest to $A$ is just the projection of $A$ onto this subspace. This projection is denoted $[A \cdot (iB)](iB)^{-1}$, and this is equivalent to the prescription I have given using the cross product above. Geometric algebra is ideally suited to formulating problems like these, as it naturally lets you work with orthogonal planes and relationships between vectors and planes.
Given two unit vectors $\hat{u}$ and $\hat{v}$, we can construct a vector perpendicular to both by their cross product:
$$\vec{n}=\hat{u}\times\hat{v}.$$
To obtain a perpendicular vector of unit length, just normalize $\vec{n}$:
$$\hat{n}=\frac{\vec{n}}{\|\vec{n}\|}=\frac{\hat{u}\times\hat{v}}{\|\hat{u}\times\hat{v}\|}.$$
Normalizing $\vec{n}$ requires the computation of $\|\hat{u}\times\hat{v}\|$. Since the norm of a vector is defined as the square root of the dot product of the vector with itself, it is impossible to normalize a vector without using square roots.
However, there is a way to look like you're avoiding square roots. If you can find the angle $\theta$ between the unit vectors $\hat{u}$ and $\hat{v}$ geometrically, you can employ the theorem that gives the norm of their cross product as:
$$\|\hat{u}\times\hat{v}\|=\sin{\theta}\\
\implies \hat{n}=\csc{\theta}\,(\hat{u}\times\hat{v}).$$
Of course, this method doesn't truly avoid square roots, since $\sin\theta$ is defined as a square root:
$$|\sin\theta| = \sqrt{1-\cos^2{\theta}}=\sqrt{1-(\hat{u}\cdot\hat{v})^2}.$$
Best Answer
given a non zero vector
$\vec{u}=(a,b)=a\vec{i}+b\vec{j}$
the vector $\vec{v}=(b,\color{red}{-}a)$ is perpendicular to $\vec{u}$.
the unit vector is then obtained by dividing by its norm
$||\vec{v}||=\sqrt{a^2+b^2}$.
so, the vector you seek is
$$\left(\frac{b}{||\vec{v}||},\frac{\color{red}{-}a}{||\vec{v}||}\right)$$.