Unfortunately, ACF and PACF plots are only useful in determining ARIMA($p$,$d$,$q$) orders if either $p$ or $q$ are zero. See, e.g., the section on nonseasonal ARIMA in Forecasting: Principles and Practice by Hyndman and Athanasopoulos.
Thus, your best bet might be to investigate your ACF/PACF plots for regularities, add an AR or MA term as suggested by the plot, check ACF/PACF plots of residuals, and iterate until there is no more evidence of structure. This is essentially the box-jenkins approach.
Alternatively, fit multiple models, either exhaustively or greedily, and pick the one with the lowest AIC (or AICc or BIC). This is a more modern approach than Box-Jenkins, and it has arguably superseded the older one, not least because it is more easily automated. (Note that you can't decide on the order of differencing $d$ using information criteria.)
Well if you mean how to estimate the ACF and PACF, here is how it's done:
1. ACF:
In practice, a simple procedure is:
- Estimate the sample mean:
$$\bar{y} = \frac{\sum_{t=1}^{T} y_t}{T}$$
- Calculate the sample autocorrelation:
$$\hat{\rho_j} = \frac{\sum_{t=j+1}^{T}(y_t - \bar{y})(y_{t-j} - \bar{y})}{\sum_{t=1}^{T}(y_t - \bar{y})^2}$$
- Estimate the variance. In many softwares (including R if you use the acf() function), it is approximated by a the variance of a white noise: $T^{-1}$. This leads to confidence intervals that are asymptotically consistent, but the smaller than the actual confidence interval in many cases (leading to a larger probability of Type 1 Error), so interpret theese with caution!
2. PACF:
The PACF is a bit more complicated, because it tries to nullify the effects of other order correlations.
It is estimated via a set of OLS regressions:
$$y_{t,j} = \phi_{j,1} y_{t-1} + \phi_{j,2} y_{t-2} + ... + \phi_{j,j} y_{t-j} + \epsilon_t$$
And the coefficient you want is the $\phi_{j,j}$, estimated via OLS with the standard $\hat{\beta} = (X'X)^{-1}X'Y$ coefficients.
So, for example, if you would like the first order PACF:
$$y_{t,1} = \phi_{1,1} y_{t-1} + \epsilon_t$$
and the coefficient you want is the $\hat{\phi_{1,1}}$ given by OLS: $\hat{\phi_{1,1}}=\frac{Cov(y_{t-1},y_t)}{Var(y_t)}$ (assuming weak stationarity).
The second order PACF would be the $\phi_{2,2}$ coefficient of:
$$y_{t,2} = \phi_{2,1} y_{t-1} + \phi_{2,2} y_{t-2} + \epsilon_t$$
And so on.
Good references on this are Enders (2004) and Hamilton (1994).
Best Answer
The statement is related to the fact that the ACF of a stationary AR process of order p goes to zero at an exponential rate, while the PACF becomes zero after lag p. For an MA process of order q the theoretical ACF and PACF show the reverse behaviour, the ACF truncates after lag q and the PACF goes to zero at an exponential rate.
These properties can be used as a guide to choose the orders of an ARMA model. See for instance, Chapter 3 in Time Series: Theory and Methods by Peter J. Brockwell and Richard A. Davis and this.