I will assume that E in the question is energy density. First we have to distinguish between two sizes:
size of the observable universe = current proper distance to particle horizon = current proper distance to infinite redshift, and
size of the whole universe.
Assuming trivial global topology, the size of the entire universe is infinite if space is flat or open (hyperbolic), and finite if space is closed (spherical). To note, if the magnitude of the current density parameter for curvature $\Omega_k(to)$ is smaller than a certain threshold, it is impossible to know whether the universe is flat and infinite or closed and finite. (Vardanyan et al (2009) How flat can you get?) Clearly the total size of an infinite universe was infinite at t=0, while the total size of a finite universe was zero.
Now, the alternative view of matter travelling into pre-existing empty space (Minkowski spacetime) is precisely the model proposed by Edward Milne in 1932. For that precise model to be valid we have to assume:
Now, if you stay within GR, then matter distribution determines spacetime, so that a non-neglibible matter density implies a spacetime different from Minkowski (so that if the matter moves, the spacetime changes over time). Having cleared that, as I said in this answer, any spacetime described by the FLRW metric, where objects are static in comoving coordinates and redshift is due to expansion of space, can also be described, via an appropriate change of coordinates, by a spherically-symmetric (SS) metric, where objects move along radial timelike geodesics and redshift is due to positional (gtt) and Doppler factors. Since both metrics describe the same physical system, they are observationally equivalent in all respects.
Two notes on the equivalent SS metric:
usually (but not in the Milne case) has a cosmological horizon (gtt & grr -> $\infty$), located at the Hubble distance in flat FLRW models,
is static (gtt & grr do not depend on time), only in the empty (Milne) and lambda-vacuum (de Sitter) cases,
even in the next simplest case, the flat matter-only FLRW model known as Einstein-de Sitter model, gtt & grr cannot be expressed algebraically in terms of non-comoving time (t' below), as I mentioned in this answer.
I expanded this below to practice a bit of LaTex:
Any homogeneous and isotropic spacetime described by the FLRW metric in terms of a set of coordinates $(t, r, \theta, \phi)$:
$$\begin{equation}
ds^2 = - c^2 dt^2 + \frac {a(t)^2 dr^2} {(1 - k r^2)} + a(t)^2 r^2 d\Omega^2
\end{equation}$$
where:
t = comoving time = proper time for all observers at constant $(r, \theta, \phi)$
r = comoving radial coordinate = radial coordinate enclosing constant proper mass
$d\Omega^2 = d\theta^2 + sin^2(\theta) d\phi^2$
can be described by a spherically symmetric (SS) metric in terms of a different set of coordinates $(t', r', \theta, \phi)$, where the ' does not mean a derivative:
$$\begin{equation}
ds^2 = - c^2 gtt(t', r') dt'^2 + grr(t', r') dr'^2 + r'^2 d\Omega^2
\end{equation}$$
where:
$\Omega$, $\theta$ and $\phi$ are the same as in the FLRW metric, and obviously
$r' = a(t) r$
Expressing $t' = f(t, r)$ and its partial derivatives as $ft$ and $fr$:
$$\begin{equation}
grr = \frac {1} {[1 - k r^2 - \left( \frac {r} {c} \frac {da(t)} {dt} \right)^2]}
\end{equation}$$
$$\begin{equation}
gtt = \frac {grr} {ft^2} (1 - k r^2)
\end{equation}$$
The problem lies in expressing gtt & grr in terms of $(t', r')$, which can be done only in the Milne and de Sitter cases.
The question of the center of the universe is a question of whether the universe is the same at all points. The easiest way to see that the universe now does not have a center is to use the Newtonian big bang. In such a description, everything is flying away from everything else with a velocity vector proportional to the position vector, where we are at the origin:
$$ v= a r $$
Suppose you are on one of the objects at position r. Then, from your point of view, everything is shifted in $r$, because of your new center $r\rightarrow r-r_0$, but everything is also shifted in $v$, because your velocity is not zero relative to us, but you will describe yourself as stationary. So $v\rightarrow v-ar_0$. The result is that you describe the objects as flying away from you with a speed proportional to their position vector.
The Newtonian big-bang is homogenous--- everyone feels that they are at the center. It is exactly analogous to the relativistic big-bang, which is also homogenous. But the Newtonian big-bang is infinite, while the relativistic big-bang is finite, in that there is no horizon in Newton.
The horizon in relativity occurs where the objects fly away at the speed of light, or equivalently, where the light-rays that reach you emerge straight from the big-bang (since looking further out is looking back in time). The horizon makes the space bounded, but it does not pick out a center, because every point has a horizon symmetric around itself.
Best Answer
You are right that a finite universe, if flat, would necessarily have a center.
However, an infinite universe has no center. An infinite muffin in 3D has a divergent volume at any point in its history, and so any point you choose will have equal (equally infinite, speaking loosely) amounts of stuff in every direction from it. All we can say is that the universe has expanded by some factor between time $t_1$ and time $t_2$.
Alternatively, a finite universe can also have no center if it is curved. In particular, if it has positive curvature1 everywhere, it loops back on itself. The analogy here is of the surface of a balloon. This is a finite 2D surface, and it has a perfectly well defined area. Still, all points are equal and none can be said to be the center. Even if the balloon is expanding, the "center" is not a part of the "universe" in this model.
The balloon analogy is used a lot in explaining cosmology, but in fact the infinite muffin describes our actual universe better. As best we can tell, our actual universe is flat and infinite.
1 Uniform positive curvature is characteristic of a (hyper)sphere. Uniform negative curvature in 2D yields a saddle.