First, there's something called the Riesz Representation Theorem. To understand it, start by fixing a vector $v \in H$. We can now use the dot product to define a continuous linear functional $L_v: H \rightarrow F$ by $$L_v(w) = \langle w,v \rangle.$$ What the Riesz Representation Theorem says is that every continuous linear functional $\phi:H \rightarrow F$ arises in this way! That is, given any element $\phi$ of the dual space $H^\ast$, there is some $v \in H$ so that $\phi(w) = \langle w, v \rangle$. So this is why we can sort of think as linear functionals as elements of the Hilbert space, and vice versa. In more mathematical terms, we say that the dual of $H$ is isomorphic to $H$.
Second, a note on dual spaces: I believe that typically the space $H^\ast$ is defined to be the set of continuous linear functionals. This distinction is important, as there are different types of duals. So far, I've been talking about the topological dual. However, there is an algebraic dual, which I have seen denoted $H^\star$, which is just all linear functionals $H \rightarrow F$, no continuity assumed. The Reisz Representation Theorem concerns only the topological dual. (The duals are actually the same for finite dimensional Hilbert spaces, but I don't believe the Hilbert spaces encountered in QM are.)
Third, adjoints: Given any Hilbert spaces $H, K$, and a continuous linear functional $A: H \rightarrow K$, there is a continuous linear map called the adjoint $A^\ast:K \rightarrow H$ (note that it goes the other way) that is defined by the equation $\langle Av, w \rangle = \langle v, A^\ast w \rangle$. You typically only see the case $K = H$. So no, the adjoint of an operator $A: H \rightarrow H$ is not an element of $H^\ast$, since the members of $H^\ast$ are continuous linear functionals from $H$ into $F$, and $H^\ast$ goes from $H$ into $H$.
Unfortunately, I don't know much about QM, so this last bit is just speculating on how I think the notation works. If you consider kets $|\phi \rangle \in H$ to be an element of the Hilbert space, then there is a continuous linear functional that I suppose you could call $\langle \phi |$ defined by $\langle \phi | v \rangle = \langle v, \phi \rangle.$ And conversely, given a bra $\langle \phi |$, by the Reisz Representation Theorem, there is a bra $| \phi \rangle$ so that $\langle \phi | v \rangle = \langle \phi , v \rangle.$
It's basically the same, it's mainly a matter of implicit identifications.
Namely, in a Hilbert space $H$ there is a natural isomorphism $H\to H^*$ (which is the bra-ket duality), so $H\otimes H\simeq H\otimes H^*$.
Now there is a map $H\otimes H^*\to L(H)$ which corresponds to what you call the outer product.
So for two kets $|\psi\rangle$ and $|\phi\rangle$ in $H$, you can either see $|\psi\rangle\otimes |\phi\rangle$ as an element of $H\otimes H$ and interpret that as a kind of two-particles state, or you can look at its canonical image in $H\otimes H^*$, which the physicists note $|\psi\rangle \langle\phi|$ (which makes sense since $\langle\phi|$ is the element of $H^*$ corresponding to $|\phi\rangle$), and then interpret it as an operator in $L(H)$.
Best Answer
The Dirac notation gets sometimes confusing and it may help to write everything using the standard Hilbert space notation (which is almost the same as Dirac's). Now:
Let $H$ be the Hilbert space and $\langle \cdot, \cdot \rangle$ the inner product. Fix $\psi, \varphi \in H$. Define the operator $A: H \to H$ by $$ Ax = \psi \langle \varphi, x \rangle $$ and the operator $A^\dagger: H \to H$ by $$ A^\dagger x = \varphi \langle \psi, x \rangle\,. $$ Now $$ \langle x, Ay \rangle = \langle x, \psi \rangle \langle \varphi, y \rangle $$ and $$ \langle A^\dagger x, y \rangle = \langle \psi, x \rangle^* \langle \varphi, y \rangle = \langle x, \psi \rangle \langle \varphi, y \rangle\,, $$ where I assumed that the inner product is linear in the second variable and conjugate-linear in the first variable. Thus $A^\dagger$ is the Hermitian adjoint of $A$.
Edit: The problem with your computation is that you misinterpreted $\langle A^\dagger x|$. Because $| A^\dagger x \rangle = | \phi \rangle \langle \psi | x \rangle $ we get $$ \langle A^\dagger x| = \langle x | \psi \rangle \langle \phi |\,. $$