$R^2=1-\frac{SSE}{SST}$, where $SSE$ is the sum of squared error (residuals or deviations from the regression line) and $SST$ is the sum of squared deviations from the dependent's $Y$ mean.
$MSE=\frac{SSE}{n-m}$, where $n$ is the sample size and $m$ is the number of parameters in the model (including intercept, if any).
$R^2$ is a standardized measure of degree of predictedness, or fit, in the sample. $MSE$ is the estimate of variance of residuals, or non-fit, in the population. The two measures are clearly related, as seen in the most usual formula for adjusted $R^2$ (the estimate of $R^2$ for population):
$R_{adj}^2=1-(1-R^2)\frac{n-1}{n-m}=1-\frac{SSE/(n-m)}{SST/(n-1)}=1-\frac{MSE}{\sigma_y^2}$.
The coefficient of determination can be calculated in various ways which coincide for linear regression. (If that's not true, then something in the comparison is not the coefficient of determination.) Away from that case, it gets messier. Various analogues or alternatives are often (but not always) labelled pseudo-. Watch out also for adjusted relatives penalising for using several predictors.
I have found the paper of Zheng and Agresti (2000) to be helpful in this territory.
Zheng and Agresti (2000) discussed the correlation between the response and the
fitted or predicted response as a general measure of predictive power for generalized linear models (GLMs). This measure has the advantages of referring to the original scale of measurement, of being applicable to all types of GLM and of being familiar to many users of statistics. Preferably, it should be used as a comparative measure for different models applied to the same data set, given that restrictions on values of the response may imply limitations on its value (see e.g. Cox and Wermuth, 1992).
For an arbitrary GLM, this correlation is invariant under a location-scale
transformation and it is the positive square root of the average proportion
of variance explained by the predictors. However, again for an arbitrary GLM,
it need not equal the positive square root of other definitions of R-square
(e.g. Hardin and Hilbe, 2001); and it need not be monotone increasing in the
complexity of the predictors, although in practice that is common. The
correlation is necessarily sensitive to outliers.
As the predicted is a function of the observed, the correlation calculated
from a sample may be expected to be biased upwards. A jackknifed correlation
is recommended as one alternative. Zheng and Agresti provide more discussion of
this point, including other estimators and a bootstrap approach to providing
confidence intervals for the correlation and to estimating the degree of overfitting.
Cox, D.R. and N. Wermuth. 1992. A comment on the coefficient of determination
for binary responses. American Statistician 46: 1-4.
Hardin, J. and J. Hilbe. 2001 (and later editions). Generalized Linear Models and Extensions. College Station, TX: Stata Press.
Zheng, B. and A. Agresti. 2000. Summarizing the predictive power of a
generalized linear model. Statistics in Medicine 19: 1771-1781.
Note The use of binary predictors in regression need not limit $R^2$ in linear regression. A simple example is the use of one continuous predictor and one binary predictor. As in principle the data could all lie on two straight lines, a value of 1 is achievable.
Best Answer
As Stephane hinted in the comment, it is the difference between model with or without intercept that matters.
For what it is worth, here is code from one of my packages:
Residuals are centered by design, leaves the fitted values with need to be centered in the intercept and not otherwise.