Finite subgroups are always contained in the maximal compact subgroup so if $ G $ has a finite maximal closed subgroup then $ G $ must be compact. Also a maximal subgroup always includes the center so if $ G $ has a finite simple subgroup which maximal then $ G $ must have trivial center in addition to being compact. Thus the only groups which have finite maximal closed subgroups are adjoint groups like $ SO(2n+1), PSO(2n),PU(n) $
Note that a finite simple group $
\Gamma $ is a maximal closed subgroup of $ PU_n $ if and only if the central extension $ n.G $ is a unitary $ 2 $ design as a subgroup of $ SU_n $. See Claim 3 of https://math.stackexchange.com/a/4477296/758507
Some examples of finite simple groups appearing as subgroups of $ PU_n $ are given here
https://mathoverflow.net/questions/414265/alternating-subgroups-of-mathrmsu-n
here
https://mathoverflow.net/questions/414315/finite-simple-groups-and-mathrmsu-n
the references in
https://mathoverflow.net/questions/17072/the-finite-subgroups-of-sun
and here
https://mathoverflow.net/questions/344218/on-the-finite-simple-groups-with-an-irreducible-complex-representation-of-a-give
But when it comes to the maximality of such finite simple subgroups of $ PU_n $ the most useful reference is
https://arxiv.org/abs/1810.02507
which, read correctly, supplies a full classification of maximal closed subgroups of $ PU_d $ that happen to be finite and simple.
The classification consists of a few infinite families of examples of maximal closed subgroups of $ PU_d $ which are finite and simple
$ PU_d $, $ d=\frac{3^k -1}{2} $ and $ d=\frac{3^k +1}{2} $ both have a maximal $ PSp_{2k}(3) $ for $ k \geq 2 $.
$ PU_d $, $ d=\frac{2^k-(-1)^k}{3} $ has a maximal $ PSU_k(2) $ for $ k \geq 4 $
In addition to these, there are a few dimensions $ d $ for which $ PU_d $ has more maximal closed finite simple subgroups than we would expect. These exceptional case are:
$ PU_2 $: $ A_5 $
$ PU_3 $: $ A_6,GL_3(2) $
$ PU_4 $: $ A_7,PSU_4(2) $
$ PU_6 $: $ A_7,PSL_3(4), PSU_4(3) $
$ PU_8 $: $ PSL_3(4) $
$ PU_{10} $: $ M_{11}, M_{12} $
$ PU_{12} $: $ Suz $
$ PU_{14} $: $ ^2 B_2(8) $
$PU_{18} $: $ J_3 $
$PU_{26} $: $ ^3 F_4(2)' $
$PU_{28} $: $ Ru $
$PU_{45} $: $ M_{23},M_{24} $
$PU_{342}$: $ O'N $
$PU_{1333}$: $ J_4 $
All these finite simple maximal closed subgroups of $ PU_n $ lift to finite quasisimple maximal closed subgroups. Some of these quasi simple lifts have simple "section" so to speak and thus correspond to a finite simple maximal closed subgroups of $ SU_n $. Some examples are
$ SU_3 $: $ GL_3(2) $
$ SU_6 $: $ A_7 $
The final answer is that $SO(3)$ (embedded into $SO(5)$ as an irrep) is maximal.
I'll first prove that $SO(3)$ is maximal among connected Lie groups, and then I'll worry about components after that.
So, let's assume $SO(3)\subseteq K\subseteq SO(5)$ with $K$ connected. We can assume $SO(3)\neq K\neq SO(5)$. Then counting dimensions and rank shows that $K$ is, up to cover, one of $SO(3)\times S^1, SO(3)\times SO(3)$, or $SU(3)$. You have already shown $SU(3)$ is not contained in $SO(4)$, so it must be one of $SO(3)\times S^1$ or $SO(3)\times SO(3)$. In any case, $K$ has a cover of the form $K' = SU(2)\times S^1$ or $SU(2)\times SU(2)$.
The irreps of $K'$ are tensor products of irreps of the factors. The factor $SU(2)$ has a unique irrep in each (complex) dimension, which alternates between real and quaternionic. The irreps of $S^1$ are all one dimensional, and classified by integers. They are complex, except the trivial rep is real.
Let's start with $K' = SU(2)\times S^1$. Since the rep a) be $5$-dimensional and b) contain $(\mathbf{5}\otimes \rho)$ as a sub rep (where $\mathbf{5}$ denotes the $5$-dim irrep of $SU(2)$ and $\rho$ denotes an arbitrary irrep of $S^1$), it follows that the rep of $K'$ given by $K'\rightarrow K\subseteq SO(5)$ is of the form $\mathbf{5}\otimes \rho$. If $\rho$ is trivial, then the covering $K'\rightarrow K$ has kernel containing $\{e\}\times S^1$, so is not finite, a contradiction. If $\rho$ is non-trivial, the rep is a tensor product of an orthogonal rep with a complex rep, so is complex (not orthgonal), again, giving a contradiction. Thus, this case cannot occur.
Let's next assume $K' = SU(2)\times SU(2)$. Unlike the previous case, we cannot simply assert that such a rep must contain $\mathbf{5}\otimes \rho$ because the $SO(3)$ we care about a priori could be diagonally embedded, or something like that.
So, let's just find all almost-faithful $5$-dim real reps of $K'$. If such a rep includes a sub rep of the form $\mathbf{1}\otimes \mathbf{n}$, of dimension $n\geq 2$, it must also include one of the form $\mathbf{m}\otimes \mathbf{n'}$ with $m\neq 1$ and $mn' = 5-n$. From here, it's easy to see that $n' = 1$. From here, it follows that precisely one of $n,m$ is odd, which then implies the rep is complex. (A real reap is a sum of real reps together with a sum of other irreps paired with their duals.)
So, the only reps we need to consider have the form $\mathbf{m}\otimes \mathbf{n}$ with both $m,n\geq 2$. Then it's clear the only option is $(\mathbf{2}\otimes \mathbf{2}) \oplus (\mathbf{1}\otimes\mathbf{1}).$ This rep (which corresponds to the usual $SU(2)\times SU(2)\rightarrow SO(4)\subseteq SO(5)$ is obviously reducible. This contradicts the fact that the $SO(3)\subseteq K$ action is irreducible. This concludes the case where $K$ is connected.
So, $SO(3)$ is maximal among connected groups, but what about components? Well, as in your previous questions, it follows now that $N:=N_{SO(5)}(SO(3))$ is a maximal subgroup of $SO(5)$ containing $SO(3)$. Let's see why $N = SO(3)$.
So, suppose $g\in N$. Then conjugation by $g$ induces an isomorphism of $SO(3)$. All isomorphisms of $SO(3)$ are inner, so there is an $h\in SO(3)$ with $hAh^{-1} = gAg^{-1}$ for all $A\in SO(3)$. Said another way, $h^{-1}g$ centralizes $SO(3)$.
Now, think of $SO(3)$ as acting on $\mathbb{C}^5$ as a complex rep, Schur's lemma tells us the the isomorphisms of the rep are $\mathbb{C}$-multiples of the identity matrix $I$. But anything which centralizes $SO(3)$ is an isomorphism of the rep. Thus, $h^{-1}g = \lambda I$ for some $\lambda \in \mathbb{C}$. As the only multiple of the identity in $SO(5)$ is the identity, we conclude $h^{-1}g$ is the identity, That is, $h = g$. This proves $N\subseteq SO(3)$, so $SO(3)$ is, in fact, a maximal subgroup of $SO(5)$.
Best Answer
Here is my attempt at an answer using the hint from Moishe Kohan.
Let $ H $ be an Ad-irreducible subgroup of a connected group $ G $. If $ H $ is not already an algebraic subgroup then take its Zariski closure $ \overline{H} $. Since $ H $ is Ad-irreducible so is $ \overline{H} $.
An algebraic subgroup of positive dimension and infinite index cannot be Ad-irreducible. Since $ \overline{H} $ is an Ad-irreducible algebraic subgroup then we must have that either $ {\rm dim}(\overline{H})=0 $ or the index $ [G:\overline{H}] $ is finite.
Since $ \overline{H} $ is an algebraic group it can only have finitely many connected components. But since $ \overline{H} $ is discrete every element is a connected component. Thus ${\rm dim}(\overline{H})=0 $ implies $ \overline{H} $ finite which implies $ H $ finite which implies ${\rm dim}(\overline{H})=0 $ (since finite sets are Zariski closed) thus all three statements are equivalent.
Now consider the condition $ [G:\overline{H}] $ is finite. Since we assumed $ G $ is connected that implies $ G/\overline{H} $ is a finite connected manifold thus trivial. So in fact $ G=\overline{H} $. Which is equivalent to saying that $ H $ is Zariski dense.
Thus we have that an ad irreducible subgroup $ H $ of a connected group $ G $ is either finite or Zariski dense.
Stated in a slightly more general way, this shows that an ad irreducible subgroup $ H $ of $ G $ is either finite or $ H $ is Zariski dense in every component of $ G $ that intersects $ H $.
EDIT:
It is interesting to note that for the special case that $ G $ is compact then corollary 3.5 of this paper
https://arxiv.org/pdf/1609.05780.pdf
states that a subgroup $ H $ of a connected compact simple group $ G $ is dense if and only if it is infinite and Ad-irreducible.
So if $ H $ is an ad irreducible subgroup of a connected compact group $ G $ then $ H $ ad irreducible implies $ G $ ad irreducible implies $ G $ simple. So now we have that $ G $ is simple compact and connected. So $ H $ is infinite if and only if it is dense.
In other words, either $ H $ is finite or it is dense and dense implies Zariski dense. Thus corollary 3.5 provides a proof of a stronger but more restricted result (if we strength the assumption to $ G $ compact then in the conclusion of the theorem we can replace Zariski dense with dense(in the manifold topology).