First some responses to comments above:
One can (and should) define fractional ideals for any integral domain $R$. A fractional $R$-ideal is a nonzero $R$-submodule $I$ of the fraction field $K$ such that there exists $x \in K \setminus \{0\}$ such that $xI \subset R$. The product of two fractional ideals is again a fractional ideal, and the multiplication is associative and has $R$ itself as an identity: in other words, the set $\operatorname{Frac}(R)$ of fractional ideals of $R$ forms a commutative monoid.
We have the notion of a principal fractional ideal: this is a submodule of the form $xR$ for $x \in K^{\times}$. The set of principal fractional ideals forms a subgroup $\operatorname{Prin}(R)$ of $R$. One could take the quotient $\operatorname{Frac(R)}/\operatorname{Prin(R)}$, but this is a bit of a false step, as it is not a group in general.
In the case of Dedekind domains there is nothing to worry about:
Theorem: For an integral domain $R$, the following are equivalent:
(i) $R$ is Noetherian, integrally closed of dimension at most one (a Dedekind domain).
(ii) Every nonzero integral ideal of $R$ factors into a product of prime ideals.
(ii)' Every nonzero integral ideal of $R$ factors uniquely into a product of prime ideals.
(iii) The fractional ideals of $R$ form a group.
So for a Dedekind domain, certainly $\operatorname{Frac}(R)/\operatorname{Prin(R)}$ is a group, called the ideal class group of $R$.
In general, there is an easy way to remedy the problem that $\operatorname{Frac}(R)/\operatorname{Prin(R)}$ need not be a group. Namely, instead of taking the full monoid of fractional ideals, we restrict to the unit group $I(R)$, the invertible fractional ideals. For any domain $R$, we may define the Picard group
$\operatorname{Pic}(R) = I(R)/\operatorname{Prin}(R)$.
Because of the theorem above, for a non-Dedekind domain the Picard group isn't capturing any information about the prime ideals in particular. There is however a different construction -- coinciding with $\operatorname{Pic}(R)$ when $R$ is a Dedekind domain -- which does just this. For (a tiny bit of) motivation: even in the case of a Dedekind domain we don't take the free abelian group on all the prime ideals: we omit $(0)$.
Now let $R$ be any Noetherian domain. One can define the divisor class group $\operatorname{Cl}(R)$ as follows: let $\operatorname{Div}(R)$ be the free abelian group generated by the height one prime ideals $\mathfrak{p}$ (these are ideals so that there is no prime ideal $\mathfrak{q}$ properly in between $(0)$ and $\mathfrak{p}$). One can also define, for each $f \in K^{\times}$, a principal divisor $\operatorname{div}(f)$. (I don't want to give the exact recipe in the general case: it involves lengths of modules. If $R$ happens to be integrally closed, then the localization $R_{\mathfrak{p}}$ at a height one prime $\mathfrak{p}$ is a DVR, say with valuation $v_{\mathfrak{p}}$, and then one takes $\operatorname{div}(f) = \sum_{\mathfrak{p}} v_{\mathfrak{p}}(f) [\mathfrak{p}]$.) Again the principal divisors $\operatorname{Prin}(R)$ form a subgroup of $\operatorname{Div}(R)$ and the quotient $\operatorname{Div}(R)/\operatorname{Prin}(R)$ is the divisor class group.
There is a canonical homomorphism $\operatorname{Pic}(R) \rightarrow \operatorname{Cl}(R)$, which is in general neither injective nor surjective. However, the map is an isomorphism in the case that $R$ is a regular ring.
These constructions are the affine versions of more familiar constructions in classical algebraic geometry: they are, respectively, Cartier divisors and Weil divisors, which agree on a nonsingular variety but not in general.
Finally, one can also define analogues of these groups for certain non-Noetherian domains (the Noetherianity is used to ensure that $v_{\mathfrak{p}}(f) = 0$ except for finitely many primes $\mathfrak{p}$), e.g. Krull domains and Prufer domains. The latter is a domain in which each finitely generated nonzero ideal is invertible. Both are natural and interesting classes of rings.
For more details on this material, see e.g. the (rather rough and incomplete) notes
http://alpha.math.uga.edu/~pete/classgroup.pdf
[Addendum: also see Section 11 of factorization2010.pdf.]
For much more detail see the references cited therein, especially Larsen and McCarthy's Multiplicative Ideal Theory.
There is information on page 68 of Montgomery and Vaughan's book, and also on page 51 of "Introduction to analytic and probabilistic number theory" by Gérald Tenenbaum. Briefly, Montgomery has established that
$$
\limsup_{x \rightarrow +\infty}\frac{R(x)}{x\sqrt{\log\log(x)}} > 0
$$
and similarly with the limit inferior. So there is only modest room for improvement. Unfortunately I cannot find any reference to an upper bound conditional on RH. On page 40 Tenenbaum has a reference to page 144 of Walfisz' book on exponential sums. Walfisz uses Vinogradov's method to show that
$$
R(x) = O\left(x\log^{2/3}(x)(\log\log(x))^{4/3}\right).
$$
I don't own a copy of Walfisz' book, so I have no further details.
Best Answer
Yes, there is a formula for $\varphi(I)$ in the case of number fields. Let $R$ be the ring of integers of a number field. As mentioned in Greg's comment, it suffices to consider the case $I=\mathfrak{p}^n$ where $\mathfrak{p}$ is a maximal ideal of $R$. Then we have a surjective ring morphism
\begin{equation} \frac{R}{\mathfrak{p}^n} \to \frac{R}{\mathfrak{p}} \end{equation} such that the preimage of $(R/\mathfrak{p})^{\times}$ is exactly $(R/\mathfrak{p}^n)^{\times}$ (this is because $R/\mathfrak{p}^n$ is local). Thus $\varphi(\mathfrak{p}^n) = q^{n-1}(q-1)$ where $q=\operatorname{Card} (R/\mathfrak{p})$.
Note that there are Dedekind domains $R$ such that $R/I$ is never finite for $I \neq R$ : for example take $R=\mathbf{C}[T]$.
To define a function $\varphi$ for general rings, one would obviously need the hypothesis that $(R/I)^{\times}$ is finite, but then it is only clear that $\varphi$ is (weakly) multiplicative in the sense that $\varphi(I\cdot J) = \varphi(I) \varphi(J)$ if $I+J=R$.