A good reference for this sort of thing is Guckenheimer and Holmes, Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields, Springer-Verlag, 1983, if you're not already familiar with it (and even if you are, for that matter). Chapter 3 in particular is relevant to your question.
The simplest and earliest example I know regarding the
renormalization group idea is the following.
Suppose we
want to study some feature $\mathcal{Z}(\vec{V})$ of some object $\vec{V}$
which is in a set $\mathcal{E}$ of similar objects.
Suppose that unfortunately this question is too hard. What can one do?
The renormalization group philosophy is try to find a
"simplifying" transformation $RG:\mathcal{E}\rightarrow\mathcal{E}$, such that $\mathcal{Z}(RG(\vec{V}))=
\mathcal{Z}(\vec{V})$, and $\lim_{n\rightarrow \infty} RG^n(\vec{V})=\vec{V}_{\ast}$
with $\mathcal{Z}(\vec{V}_{\ast})$ easy to understand.
Example (Landen-Gauss, late 1700's):
Let $\vec{V}=(a,b)\in\mathcal{E}=(0,\infty)^2$ and consider
$$
\mathcal{Z}(\vec{V})=\int_{0}^{\frac{\pi}{2}}
\frac{d\theta}{\sqrt{a^2 \cos^2\theta+b^2\sin^2\theta}}\ .
$$
A good choice of renormalization transformation here is $RG(a,b)=\left(\frac{a+b}{2},\sqrt{ab}\right)$, as discovered by Gauss.
A recent example now.
Example (Kadanof-Wilson, late 1960's early 1970's):
Take $\mathcal{E}$ to be the set of Borel probability measures on $\mathbb{R}^{\mathbb{Z}^d}$. Let $\mathcal{Z}(\vec{V})$ be equal to $1$ if the two-point function decays exponentially and $0$ otherwise.
Then define $RG$ as the transformation which gives the law of the block-spinned/coarse-grained field as a function of the law of the original field.
Note that the feature $\mathcal{Z}$ that one would like to preserve can be defined a bit more loosely. One could, e.g., "define" it as the long-distance behavior of the random spin field with probability law $\vec{V}$ (or in physics jargon: the low energy effective theory).
Also in the dynamical systems context, it could be the chaotic behavior or not of a map $\vec{V}$. Then $RG$ could be a doubling transformation, i.e., composition of the map with itself together with some rescalings and reversals of orientation.
Best Answer
The renormalization approach to dynamical systems pioneered by Chen, Goldenfeld and Oono [1] applies the Gell-Mann and Low renormalization group from quantum physics [2] to extract the global behavior of a function known locally from perturbation theory.
For a more recent overview, see Renormalization Group as a Probe for Dynamical Systems.
[1] L.Y. Chen, N. Goldenfeld, and Y. Oono, Renormalization Group Theory for Global Asymptotic Analysis (1994).
[2] M. Gell-Mann and F.E. Low, Quantum Electrodynamics at Small Distances (1954).