There are two extremities: on the "easy end" one has vector bundles which are classified by maps to the (more or less) well understood spaces like Grassmanians; on the "hard end" there are spherical fibrations/sphere bundles classified by maps to the classifying space of the monoid of self-homotopy equivalences of spheres.
There are ways to move "just slightly down" from the hard end, by considering self-homeomorphisms, self-PL-isomorphisms, self-diffeomorphisms, …
On the other hand I have not seen any ways to move "just slightly up" from vector bundles. What I mean here is this: one may switch from a vector bundle to an associated sphere bundle; the corresponding sphere bundle is one whose gluing maps are linear in one sense or another ($\mathbb R$-linear, or $\mathbb C$-linear, $\mathbb H$-linear, …). Thus there is a huge gap where on one end we have something linear and on the other something "as nonlinear as it gets".
My question is whether it is possible to squeeze in between some groups consisting of controllably nonlinear maps. Say, maps of degree $n$, with composites truncated back to degree $n$ or something similar.
Many years ago I was asking around about this, and Leonid Makar-Limanov suggested to look at maps of "finite codegree" – that is, something describable by series with vanishing low degree coefficients – since these readily form a group under composition; but this would still mean "moving down from the hard end" rather than "moving up from the easy end"…
…After some hesitation decided to add more speculative possibility that occurred when commenting to the answer below. Maybe reducing the structure group of a bundle from some linear group (say, $SU(n)$) to its increasingly highly connective covers could be interpreted as switching from linear gluing maps to ones which are "linear up to homotopy" in some sense, together with (coherent) choices of "linearizing homotopies". This brings up 2-vector bundles in the sense of Kapranov-Voevodsky or Baas-Dundas-Rognes, I wonder if this has been looked at from this angle?
Actually I am confused here by another thing, and maybe this needs separate question, but why not ask it right away? What I don't understand is this – reducing structure group to more and more highly connective covers sort of "moves towards a contractible "structure group"", while the ultimate "structure group" should be the aforementioned monoid of self-homotopy equivalences of the sphere rather than contractible. I seem to mix up different things here but how exactly?
Best Answer
This is more of a comment than an answer, but it's too long for a comment, so I'm putting it here.
It sounds as though you are asking what sorts of groups of diffeomorphisms there are acting transitively on spheres, so that for each such subgroup $\Gamma\subset \mathrm{Diff}(S^n)$, one can define a notion of $\Gamma$-bundles over manifolds, where a $\Gamma$-bundle over $M$ is a submersion $\pi:B\to M$ with fibers diffeomorphic to~$S^n$ together with an atlas of local trivializations $\tau_\alpha:B_{U_\alpha}\to U_\alpha\times S^n$ so that, on overlaps, we have $\tau_\beta\circ{\tau_{\alpha}}^{-1} (u,v)= \bigl(u,g_{\beta}^\alpha(u,v)\bigr)$ where, for each $u\in U_\alpha\cap U_\beta$, the map $v\mapsto g_\beta^\alpha(u,v)$ belongs to $\Gamma$.
Aside from the cases $\Gamma=\mathrm{SL}(n{+}1,\mathbb{R})$ (the vector bundle case) and $\Gamma = \mathrm{Diff}(S^n)$ (the general sphere bundle case), there are a number of different subgroups you might consider.
"Moving up from the easy end" might be interpreted as considering the case in which $\Gamma$ is a finite dimensional Lie subgroup of $\mathrm{Diff}(S^n)$, and this is, of course, a very classical subject. There are examples of such subgroups that are not conjugate to any subgroup of $\mathrm{SL}(n{+}1,\mathbb{R})$.
For example, when $n=1$, because $\mathbb{RP}^1$ is topologically a circle, one has the action of $\mathrm{PSL}(2,\mathbb{R})\simeq\mathrm{SO}(2,1)$ on $\mathbb{RP}^1\simeq S^1$, which is not conjugate to any subgroup of $\mathrm{SL}(2,\mathbb{R})$ acting in its usual way on $S^1$ (i.e., considered as the space of oriented lines through $0$ in $\mathbb{R}^2$). This generalizes to higher dimensions as the action of $\mathrm{SO}(n{+}1,1)$ on $S^n$, preserving the conformal structure (but not the linear structure). Still, all of these contain compact subgroups that act transitively on $S^n$, so I guess it's possible to reduce the structure group of such bundles to a compact group, which reduces it to the classical case of a vector bundle.
When $n=1$, it's a classical result (essentially due to Lie) that, up to conjugacy, the only proper Lie subgroups of $\mathrm{Diff}(S^1)$ that act transitively on $S^1$ are the circle itself and, for each $k\ge1$, the $k$-fold covering of the $\mathrm{SO}(2,1)$ action listed above. (Here, 'Lie subgroup' means a subgroup of $\mathrm{Diff}(S^n)$ that is defined as the set of diffeomorphisms satisfying some system of PDE.)
In the other direction, i.e., "moving down from the hard end", if you want $\Gamma$ to be a transitive Lie subgroup (in Lie's sense, i.e., one defined as the set of diffeomorphisms that satisfy some given system of PDE) of $\mathrm{Diff}(S^n)$, then there aren't any that have finite codimension in $\mathrm{Diff}(S^n)$ in the sense of $\mathrm{Diff}(S^n)/\Gamma$ being a finite-dimensional manifold in any sense.
In some sense, the 'largest' subgroup (when $n>1$) is the subgroup $\mathrm{SDiff}(S^n)$ consisting of volume-preserving diffeomorphisms of $S^n$. When $n$ is even, up to conjugacy, this is the only infinite dimensional transitive Lie subgroup. Whenn $n$ is odd, there are others, such as $\mathrm{Cont}(S^n,\omega)$, which consists of the diffeomorphisms of $S^n$ that preserve a given contact form $\omega$ up to a multiple, and its subgroup $\mathrm{SCont}(S^n,\omega)$ which consists of the diffeomorphisms of $S^n$ that preserve the contact form $\omega$ exactly. (Since not all contact forms on $S^n$ are globally equivalent, these actually give an infinite family of distinct Lie subgroups up to conjugacy .)
Whether there are examples of sphere bundles that are $\Gamma$-bundles for the above 'infinite dimensional' Lie subgroups but do not come from vector bundles, I don't know. It's an interesting question. I think you could always reduce any oriented sphere bundle's structure group from $\mathrm{Diff}(S^n)$ to $\mathrm{SDiff}(S^n)$, so maybe this answers the question for even $n$. I don't know about the contact cases, though. That might be interesting.