Since $\mathfrak m$ is finitely generated, you can find a generating family $\frac {a_1}{b_1}, \ldots, \frac {a_n}{b_n} \in \mathfrak m$.
Now, let $c = b_1b_2 \ldots b_n$. Forall $i$, $c\frac {a_i}{b_i} \in R$, and so forall $m \in \mathfrak m, cm \in R$. Hence $c \in \mathfrak m^{-1}$
This is not really a problem about orders or fractional ideals, but about lattices. Let $V$ be a finite-dimensional ${\mathbf Q}$-vector space (such as a number field) and set $n = \dim_{\mathbf Q}(V)$. A lattice in $V$ is a finite free ${\mathbf Z}$-module in $V$ of rank $n$. If $V$ is a number field $K$, examples of lattices in $V$ include any order $R$ in $K$ and any $R$-fractional ideal.
When $L$ and $L'$ are lattices in $V$, check their sum $L+L' = \{x + y : x \in L, y \in L'\}$ is a lattice. If $L' \subset L$, the usual index $[L:L'] = |L/L'|$ is finite. We want to define an index $[L:L']$ even if $L'$ is not contained in $L$.
Here's how we can do it. For any two lattices $L$ and $L'$ in $V$, define the index $[L:L']$ to be the positive rational number
$$
\frac{[M:L']}{[M:L]},
$$
where $M$ is any lattice in $V$ containing $L$ and $L'$, and the numerator and denominator here are the usual notion of index (because $L$ and $L'$ are contained in $M$).
Exercises.
1) Check this is independent of the choice of $M$ and thus is well-defined. (Hint: use multiplicativity of the usual notion of index and the fact that any lattice containing $L$ and $L'$ must contain $L+L'$.)
2) Check this equals $|L/L'|$ if $L' \subset L$.
3) Check for any three lattices $L, L', L''$ in $V$ that $[L:L''] = [L:L'][L':L'']$.
4) For any lattices $L$ and $L'$ in $V$, and any ${\mathbf Q}$-linear automorphism $\varphi \colon V \rightarrow V$, check $[L:L'] = [\varphi(L):\varphi(L')]$.
5) For any lattice $L$ in $V$ and ${\mathbf Q}$-linear automorphism $\varphi \colon V \rightarrow V$, check $\varphi(L)$ is a lattice in $V$ and $[L:\varphi(L)] = |\det \varphi|$.
6) For any lattices $L$ and $L'$ in $V$, show there is a ${\mathbf Q}$-linear automorphism $\varphi \colon V \rightarrow V$ such that $\varphi(L) = L'$, and for any such $\varphi$ we have $[L:L'] = |\det \varphi|$. This provides a different way of defining the index $[L:L']$.
Using $V = K$ and considering the lattices $R$, $I$, and $J$, and using as $\varphi \colon K \rightarrow K$ suitable multiplication maps $\varphi(x) = \alpha{x}$, you can recover the properties you want. Define ${\rm N}(I) = [R:I]$, even if $I$ is not contained in $R$.
Best Answer
Question 1) Yes. The definition says $I$ is an $R$-submodule of $K$. And also, yes, the action is simply "scalar" multiplication. More generally, if $R$ is a subring of any ring $S$, then $S$ is an $R$-module via the action $r\cdot s=rs$ (for $r\in R$ and $s\in S$).
Question 2) $R$-submodules of $R$ are ideals so if $R$ is a field it won't have any interesting submodules of itself. But you are considering $R$-submodules of $K$. These no longer need to be ideals (of $K$) but simply subgroups under addition which are closed under "scalar" multiplication by $R$ (which is considerably weaker than being an ideal). Notice it is a fractional ideal of $R$ (not $K$). Fractional ideals do not have to be subsets of $R$ (which at first seems weird, because ideals are subsets + other conditions).
Question 3) Let $I$ be a fractional ideal of $\mathbb{Z}$. Then $I$ is a $\mathbb{Z}$-submodule of $\mathbb{Q}$ (i.e. a subgroup of $\mathbb{Q}$ under addition) and there exists some $r \in \mathbb{Z}$ such that $rI \subset \mathbb{Z}$. This implies $rI = m\mathbb{Z}$ for some $m \in \mathbb{Z}$. So $I = \frac{m}{r}\mathbb{Z}$ (multiples of the fraction $m/r$). Thus the name "fractional ideal". The ideals of $\mathbb{Z}$ are $J=x\mathbb{Z}$ for $x\in\mathbb{Z}$ whereas the fractional ideals allow $x$ to be a fraction.