My question is partially inspired by the following discussion:
Topology on the Unitary Dual
Let me remind/explain how the Fell topology is defined (at least I recall the definition which I saw): let $G$ be a locally compact group. One first consider the set $Prim(G)$ of so called primitive ideals (those being a kernal of the irreducible representation): for $S \subset Prim(G)$ one defines $\overline{S}=\{I \in Prim(G): \bigcap S \subset I\}$. It is not too hard to show that this operation satisfies all properties of "being a closure" therefore it defines the topology on $Prim(G)$ where closed sets $S$ are those, for which $S=\overline{S}$. There is also natural mapping from the space $\hat{G}$ of all equivalence classes of irreducible representations $[\pi] \mapsto \ker \pi$ and one can define the topology on $\hat{G}$ as the weakest topology for which this map is continuous. This topology is called the Fell topology. According to the cited discussion, it should be true that convergence in this topology is equivalent to convergence of all matrix coefficients. However, since we deal with classes of representation, one should be careful how this convergence is defined: my guess would be the following. Let $([\pi_j])_j$ be a net in $\hat{G}$: we say that $[\pi_j] \to [\pi]$ where:
for each $\varepsilon >0$, compact $K \subset G$ and each $\xi_0,\eta_0 \in \mathcal{H}_{\pi}$ such that $\xi_0 \perp \eta_0$ there exists $j$ and $\xi_j,\eta_j \in \mathcal{H}_{\pi_j}$ such that for all $x \in K$:
$$|\langle \pi_j(x) \xi_j,\eta_j \rangle – \langle \pi(x) \xi_0,\eta_0 \rangle|<\varepsilon.$$
So my question is the following:
Is the convergence in the Fell topology equivalent to the convergence of matrix coefficients defined above?
EDIT: I should have been more precise: there is a one-to-one correspondence between nondegenarte representations of the so called full $C^*$-algebra of the group $C^*(G)$ and unitary group representations (this correspondence preserves the notion of irreducibility). All primitive ideal are therefore in this $C^*$-algebra. In particular my question also makes sense in the general context of $C^*$-algebras and I'm also interested in the answer for such a question.
EDIT 2: I corrected the obvious mistake pointed out in the answer.
Best Answer
In your inequality you need to evaluate the $\pi$'s somewhere! Unless I am mistaken unpacking Fell (1962), Theorem 2.2 and Remark following, $[\pi_j]\to[\pi]$ means that for every choice of an $\varepsilon>0$, a compact set $K\subset G$, an integer $n$, and vectors $u_1,\dots,u_n\in\mathscr H_\pi$, there exists a $j_0$ such that when $j\succcurlyeq j_0$, $\mathscr H_{\pi_j}$ contains $v_1,\dots,v_n$ such that $$ \left|\langle u_i,\pi(g)u_i\rangle-\langle v_i,\pi_j(g)v_i\rangle\right|<\varepsilon $$ for all $i\in\{1,\dots,n\}$ and all $g\in K$. In short, positive-definite functions associated with $\pi$ are uniform-on-compacta limits of positive-definite functions associated with the $\pi_j$. (See also Dixmier (1977), 3.4.10 for $\mathrm C^*$-algebras and 18.1.5 for groups.)