Some of these questions are addressed (in the derived setting) in my paper Integral Transforms and Drinfeld Centers in Derived Algebraic Geometry with John Francis and David Nadler --- for the underived setting you might also want to look at Hinich's Drinfeld double for orbifolds. I think the answer to your main question is no: if you take (quasi)coherent sheaves on any scheme as a symmetric monoidal category, its Drinfeld center is equivalent to itself. A more precise statement is that the derived Drinfeld center of QCoh of a stack is given by sheaves on the derived loop space of the stack. The underlying underived stack is the inertia --- which in the case of a scheme is just the underlying scheme. In this case though you still see something interesting on the derived level -- you get modules over the sheaf of Hochschild chains on your scheme, i.e. in char zero sheaves on the odd version of the tangent bundle [tangent complex if things aren't smooth]. The braided structure on this Drinfeld center is given by convolution over the base -- the free loop space has a homotopical group structure over the space
(with fibers the based loop spaces). For R-R bimodules, aka endofunctors of R-modules [for R a k-algebra], its center is k-mod -- in fact it's 2-Morita equivalent to k-modules. (We can think of it as a matrix algebra with entries given by Spec R, so we expect it to be Morita equivalent to scalars).
Interesting question ! As far as I know, there are at least two secretly equivalent answers.
You somehow already gave the first one: a modular tensor category is the same as a modular functor (though the precise statement is quite subtle, see the beautiful introduction to this paper : https://arxiv.org/abs/1509.06811). A modular functor is, roughly, a collection of compatible (projective) representations of mapping class groups, so in particular you need a representation of $SL_2(\mathbb{Z})$. It turns out this is also sufficient, in the sense that a premodular tensor category gives representations of mapping class groups in genus 0, and modularity is exactly what you need to extends this to higher genus.
A somehow more conceptual reason is related to the fact that those theories have an anomaly, i.e. you get only projective representations of MCG's, and invariant of 3-fold with some extra structure (I learned this point of view from Walker and Freed-Teleman). The origin of the story is the 3d Chern-Simons TFT introduced by Witten using Feynman integrals. It turns out that there is a 4d TFT around, a very simple topological version of Yang-Mills theory. This theory is simple because it involves integrating 4-forms on 4-folds, and those forms are actually exact. So if Feynman integrals really were integrals, by Stokes theorem this would be trivial on closed manifold, and for a 4-fold W with boundary we could define
$$Z_{CS}(\partial W):=Z_{YM}(W).$$
SInce any oriented 3-fold bound a 4-fold this would indeed be enough to define your theory.
It turns out $Z_{YM}$ is not trivial, but it is close to be: it is an invertible theory. It means in particular that it attaches 1-dimensional vector spaces to 3-fold, and that every 4-fold $W$ gives an isomorphism
$$Z_{YM}(\partial W) \longrightarrow Z_{YM}(\emptyset)=\mathbb{C}$$
which depends on the choice of $W$ only up to bordism. Hence you get a number defined up to this choice, and the bordism class of $W$ is precisely the extra structure occurring in this anomaly I was talking about.
How does it relates to your question ? Well, it is expected that every premodular category gives rise to a 4-dimensional TFT. Roughly speaking, to know whether this theory is invertible, it's enough to check what you get for the 2-sphere, which is a category, and it turns out this category is equivalent to the Müger center, i.e. the category of transparent objects. At the categorical level, invertibility means being equivalent to the category of vector spaces.
Therefore, the modularity condition is exactly what you need to go from a 4d TFT to a 3d TFT with anomaly.
Best Answer
To expand on my comment, this connection is indeed well-known and the key concept is that of ribbon category. A standard textbook reference is Turaev, Quantum Invariants of Knots and 3-Manifolds.
Braided and rigid is not enough to get links invariants, because RI will not hold in general (and in fact pretty much never). A nice exposition of that issue can be found in Selinger, A survey of graphical languages for monoidal categories (https://arxiv.org/abs/0908.3347) (in that reference autonomous means rigid and tortile means ribbon).
Any ribbon category provides an invariant of framed, oriented links for each object $X$. If $X$ is simple then the ribbon element $\theta_X$ acts as a multiple of the identity so that you can renormalize to get an invariant of oriented links. The choice of an isomorphism $X\cong X^*$, provided one exists, gets rid of the orientation.
If you want non trivial invariants of oriented links on the nose, you'd need that $\theta_X=id_X$ and $\theta_{X\otimes X} \neq id_{X\otimes X}$. While this isn't impossible (tautologically, the category of oriented tangles is a ribbon category which satisfies this for example) this is a pretty unnnatural condition.