Here's how you get from one to the other. Let me take the case of a particle moving in one dimension.
In this case, we assume that there exist vectors $|x\rangle$ which form a "dirac-normalized" basis (the position basis) for the Hilbert space in the sense that their inner products satisfy
$$
\langle x|x'\rangle =\delta(x-x')
$$
Note, as an aside, that these vectors are not normalizable in the standard sense, and therefore they do not strictly speaking belong to the Hilbert space.
Next, for each $|\psi\rangle$ in the Hilbert space, we define the position basis wavefunction $\psi$ corresponding to the state $|\psi\rangle$ as
$$
\psi(x) = \langle x|\psi\rangle
$$
So really, the value $\psi(x)$ of the position basis wavefunction $\psi$ at a point $x$ can simply be thought of as the basis component of $|\psi\rangle$ in the direction of $|x\rangle$ just as in the finite-dimensional case where one can find the component of a vector $|\psi\rangle$ along a basis vector $|e_i\rangle$ simply by taking the inner product $\langle e_i|\psi\rangle$.
A state is something that encodes our knowledge about the system.
And that's it.
There are many ways to encode a state in quantum mechanics. As a wavefunction ("Schrödinger representation"), as Fock momentum states ("Fock representation"), as a density matrix, as a ray in a Hilbert space, as a linear functional on the $C^*$-algebra of observables, as a point in a projective Hilbert space,...
Not all of these ways are always permissible. The "as a ray in a Hilbert space", "density matrix" and "functional on algebra of observables" are, to my knowledge, always possible, while e.g. the encoding as a wavefunction fails for quantum systems whose Hilbert space is finite-dimensional (e.g. qubits) because those don't have the usual position operator.
The representation as a density matrix generalizes to statistical quantum mechanics, that as a single ray in a Hilbert space or as a wavefunction does not. But whenever two of these descriptions are possible, they are, at the level of rigor of physicists at least, equivalent.
So the question "What really is a state in quantum mechanics?" doesn't have a single answer. Or any answer other than "it depends what you want to do with it".
But this should not surprise you, after all, it is the same in classical mechanics: You can have Newtonian or Lagrangian or Hamiltonian descriptions, and here a state is position and velocity, there a state is position and momentum, or even some generalized coordinates which have no direct physical meaning at all.
There is no truth to a state, be it classical or be it quantum, other than "it encodes all possible information about the physical system in a convenient way".
Best Answer
I think all other answers so far do not address the problem directly. The definition of the OP regarding kets is a quite common, though not the most used one, see e.g. references 1 and 2. So I don't think one should argue now which definition of "ket" is the right one (the question is ill-posed: it is just notation).
The OP further uses the word "state" to refer to an element of the Hilbert space, which a priori is fine, but it does not correspond to the physical notion of the word (but to be fair, I think many intro texts use this notion, although one might add a normalization constraint). Others have commented on that, although it is not the primary question and actually not relevant here.
The question asks about the difference between $\psi\in H$ and $|\psi\rangle\in L(\mathbb C,H)$, based on a confusion and mixing of different notations and concepts.
To start, let me recap the basics which lead to the notation introduced in the question: For any vector $v\in H$ of a complex Hilbert space $H$, we can define a map $|v\rangle: \mathbb C \to H$ by $|v\rangle: \lambda \mapsto \lambda v$. In fact, the thus defined map: $\mathcal I: H\to L(\mathbb C, H)$, with $\mathcal I: v\mapsto |v\rangle$ is a canonical isomorphism. This means, roughly speaking, that we can relate both objects $v$ and $|v\rangle$ in a natural way, without "loosing information". Further, with this definition, we can meaningfully speak of the adjoint of $|v\rangle$, which turns out to be the well-known "bra", denoted by $\langle v|$. As a last point, let me remark that it is exactly this notion of a "ket", which people refer to when saying things like "the bra is the adjoint of the ket" or so.
Let me now clear up the confusion in the question:
With the definition of "ket" given in the question, we have to make sure to understand all other symbols accordingly. Most importantly, if we write $|\psi\rangle\langle\psi|$, then we denote by this the composition of the "ket" and the "bra" map, i.e. $|\psi\rangle\langle\psi|:=|\psi\rangle \circ \langle \psi|: H \to H$, with $$|\psi\rangle\langle\psi| (v)= |\psi\rangle\left(\langle\psi,v\rangle_H\right)= \langle\psi,v\rangle_H \,\psi \quad . \tag 1$$
Hence, in this notation we have that $|\psi\rangle\langle\psi|$ acts as a hermitian one-dimensional projection on $\psi \in H$, which often is also written as $P_\psi$ (which makes no reference to any ket-notation). The spectral theorem now allows to decompose every hermitian operator $A$ as
$$ A = \sum\limits_{j=1}^{\dim H} a_j\, P_{\psi_j}\overset{(1)}{=}\sum\limits_{j=1}^{\dim H} a_j\, |\psi_j\rangle\langle\psi_j| \quad ,\tag 2$$
where the $\psi_j$ are (orthonormal) eigenvectors of $A$ with eigenvalues $a_j$. The first equality is the spectral theorem, the second a consequence of our notation and definitions.
We therefore obtain
$$A v\overset{(2)}{=}\sum\limits_{j=1}^{\dim H} a_j\,|\psi_j\rangle\langle\psi_j| (v)\overset{(1)}{=}\sum\limits_{j=1}^{\dim H} a_j\, \langle \psi_j,v\rangle_H\, \psi_j \tag 3 \quad ,$$
and everything makes sense, i.e. the linear operator $A$ maps vectors of the Hilbert space $H$ to vectors again.
Now, the reason many people seem to be confused here is, I suppose, that in most physics books a "ket" is defined to be a vector in the Hilbert space. Put differently, then it is assumed that $|\psi\rangle \in H$ and symbols like $\psi$ have no meaning; it is the whole object, the ket, which is the vector. Then the operator $P_\psi$ can also be written with this notation as $P_\psi=|\psi\rangle\langle \psi|$, where the symbol on the RHS is defined as the operator $|\psi\rangle\langle \psi|: H\to H $ with
$$ |\psi\rangle\langle \psi|: |v\rangle \mapsto \langle \psi|v\rangle |\psi\rangle \quad , \tag 4$$
just as before (if you identify both ways to write the inner product); but to emphasize: The symbols here have a different meaning than before. It is thus important to not mix both notations and concepts, and stick to one convention from the beginning. Else one might end up in contradictions and ill-defined expression as in the question.
You do not get or lose anything by using one or the other notation, a priori. For some purposes one might be more suitable than the other. On the other hand, many people (including myself) do not use any of these "bra-ket" notations at all.
References: