Let $R$ be a (noncommutative) ring. (For me, the words "ring" and "algebra" are isomorphic, and all rings are associative with unit, and usually noncommutative.) Then I think I know what "linear algebra in characteristic $R$" should be: it should be the study of the category $R\text{-bimod}$ of $(R,R)$-bimodules. For example, an $R$-algebra on the one hand is a ring $S$ with a ring map $R \to S$. But this is the same as a ring object in the $R\text{-bimod}$. When $R$ is a field, we recover the usual linear algebra over $R$; in particular, when $R = \mathbb Z/p$, we recover linear algebra in characteristic $p$.
Suppose that $G$ is an algebraic group (or perhaps I mean "group scheme", and maybe I should say "over $\mathbb Z$"); then my understanding is that for any commutative ring $R$ we have a notion of $G(R)$, which is the group $G$ with coefficients in $R$. (Probably there are some subtleties and modifications to what I just said.)
My question: What is the right notion of an algebraic group "in characteristic $R$"?
It's certainly a bit funny. For example, it's reasonable to want $GL(1,R)$ to consist of all invertible elements in $R$. On the other hand, in $R\text{-bimod}$, the group $\text{Aut}(R,R)$ consists of invertible elements in the center $Z(R)$.
Incidentally, I'm much more interested in how the definitions must be modified to accommodate noncommutativity than in how they must be modified to accommodate non-invertibility. So I'm happy to set $R = \mathbb H$, the skew field of quaternions. Or $R = \mathbb K[[x,y]]$, where $\mathbb K$ is a field and $x,y$ are noncommuting formal variables.
Best Answer
It seems that you want some notions on noncommutative group scheme,right? In fact, A.Rosenberg has introduced noncommutative group scheme in his work with Kontsevich "noncommutative grassmannian and related constructions" (2008). Actually, this work gave a systematically treatment to the noncommutative grassmannian type space introduced in their early paper noncommutative smooth space and the work of Rosenberg himself on noncommutative spaces and schemes.
More comments: It seems that you want to know the linear algebra over noncommutative ring. I think you need to look at the paper by Gelfand and Retakh on Quasideterminants, I. And the main motivation for the "noncommutative grassmannian and related constructions" is to give a geometric explanation to the work of Gelfand and Retakh.
All of these work are based on functor of point of view.