Basically, can you do everything with the equivalent linear univariate
regression model that you could with the multivariate model?
I believe the answer is no.
If your goal is simply either to estimate the effects (parameters in $\mathbf{B}$) or to further make predictions based on the model, then yes it does not matter to adopt which model formulation between the two.
However, to make statistical inferences especially to perform the classical significance testing, the multivariate formulation seems practically irreplaceable. More specifically let me use the typical data analysis in psychology as an example. The data from $n$ subjects are expressed as
$$ \underset{n \times t}{\mathbf{Y}} = \underset{n \times k}{\mathbf{X}} \hspace{2mm}\underset{k \times t}{\mathbf{B}} + \underset{n \times t}{\mathbf{R}},
$$
where the $k-1$ between-subjects explanatory variables (factor or/and quantitative covariates) are coded as the columns in $\mathbf{X}$ while the $t$ repeated-measures (or within-subject) factor levels are represented as simultaneous variables or the columns in $\mathbf{Y}$.
With the above formulation, any general linear hypothesis can be easily expressed as
$$\mathbf{L} \mathbf{B} \mathbf{M} = \mathbf{C},$$
where $\mathbf{L}$ is composed of the weights among the between-subjects explanatory variables while $\mathbf{L}$ contains the weights among levels of the repeated-measures factors, and $\mathbf{C}$ is a constant matrix, usually $\mathbf{0}$.
The beauty of the multivariate system lies in its separation between the two types of variables, between- and within-subject. It is this separation that allows for the easy formulation for three types of significance testing under the multivariate framework: the classical multivariate testing, repeated-measures multivariate testing, and repeated-measures univariate testing. Furthermore, Mauchly testing for sphericity violation and the corresponding correction methods (Greenhouse-Geisser and Huynh-Feldt) also become natural for univariate testing in the multivariate system. This is exactly how the statistical packages implemented those tests such as car in R, GLM in IBM SPSS Statistics, and REPEATED statement in
PROC GLM of SAS.
I'm not so sure whether the formulation matters in Bayesian data analysis, but I doubt the above testing capability could be formulated and implemented under the univariate platform.
Multivariate regression means that one have multiple response variables $Y$'s. In matrix form this is
$$
Y = X B + E
$$ where $Y$ is $n\times m$ ($m$ responses, observations on $n$ units,) $X$ is $n\times p$, $B$ is $p\times m$ and finally $E$ is $n\times m$. This is formally very similar to $m$ multiple regressions. The model then must be completed by assumptions on the error term $E$. If the errors in the $m$ equations are independent, then the model is close to $m$ separate regressions. This is discussed in answers here: Explain the difference between multiple regression and multivariate regression, with minimal use of symbols/math
So why the tussle?
Dependence between error terms in separate equations, leads to SUR seemingly unrelated regressions.
Some coefficients in separate equations might be shared, or some other restrictions on the $B$ coefficient matrix. This can lead to more efficient estimation, or in the case of restricted rank regression better predictions.
Null hypothesis for testing might involve multiple equations simultaneously. MANOVA might be an example.
(these include among them both your examples)
Best Answer
As for Question 1, you are correct with what you said.
As for Question 2, multivariate stands for an analysis involving more than one response variables. To my knowledge there is no differentiation in terminology with respect to the predictor variables. To be consistent one could maybe say, but I am not sure, "simple multivariate regression" when multiple responses and one predictor variable are present.
As for Question 3, I'd say you are right again.
As for Question 4, the term bivariate refers to a situation when there are two continuous variables in total, i.e. an analysis that can be visualized in a 2d scatter plot (simple linear regression and correlation for example).
So now what does univariate refers too? I think (and I might be wrong) that is the case when you have one response and one or more categorical predictor(s). So, for example you measure the heights of trees coming from the same parent tree, or the weight of chicken fed with the different feeds. This type of analyses would be analyzed as a t-test or Analysis of Variance.
The difference between univariate and bivariate can be seen when you visualize the data. If you plot something as a bar graph, (or dot plot) it is univariate, if you plot something on a 2d scatter plot, it is bivariate. I might be wrong here but I am sure if that's the case someone will comment!