Basically, I think the idea that the universe is infinite comes from considerations of the large-scale curvature of spacetime. In particular, the FLRW cosmological model predicts a certain critical density of matter and energy which would make spacetime "flat" (in the sense that it would have the Minkowski metric on large scales). If the actual density is greater than that density, then spacetime is "positively curved," which implies that it is also bounded - that is, that there is a certain maximum distance between any two spacetime points. (I don't know the details of how you get from positive curvature to being bounded, but as suggested by a commenter, look into Myers's theorem if you're curious.) However, if the actual density is not greater than that critical density, there is no bound, which means that for any distance $d$, you could find two points in the universe that are at least that far away. I think that's what it means to be infinite.
Overall, the observations done to date, paired with current theoretical models, are inconclusive as to whether the actual density of matter and energy in the universe is greater than or less than (or exactly equal to) the critical density.
Now, if the universe is in fact infinite in this sense, it still could have had a big bang. The FLRW metric includes a scale factor $a(\tau)$ which characterizes the relative scale of the universe at different times. Specifically, the distance between two objects (due only to the change in scale, i.e. ignore all interactions between the objects) at different times $t_1$ and $t_2$ satisfies
$$\frac{d(t_1)}{a(t_1)} = \frac{d(t_2)}{a(t_2)}$$
Right now, it seems that the universe is expanding, so $a(\tau)$ is getting larger. But if you imagine running that expansion in reverse, eventually you'd get back to a "time" where $a(\tau) = 0$, and at that time all objects would be in the same position, no matter whether space was infinite or not. That's what we call the Big Bang.
It is true that FRW metric, if redirected backwards in time, predicts a singularity.
However, when the universe's size is comparable to the Planck scale, no one really knows what truly happens.
To date there is no successful and consistent theory of quantum gravity, although a lot of partially successful ones exist.
String theory, for instance has become a leading competitor in the race for QG.
As for the expansion rate, that is the subject of Inflationary Cosmology.
The FRW metric, along with Einstein's Field Equations, yield equations of motion for the cosmos.
$$
H^2=\frac{8\pi G \rho(t)}{3}
$$
These yield different solutions depending on different mixtures of energy density constituents, if the universe is matter-dominated, an expansion rate is given, and if the universe is radiation-dominated a different one is yielded.
A truly exponential growth factor is yielded if the universe is "Dark Energy" dominated, meaning an energy density which is constant in time, regardless of the expansion of the universe.
$$
H^2=\frac{8\pi G \rho_0}{3}\Rightarrow \frac{\dot{a}}{a}=\sqrt{\frac{8\pi G \rho_0}{3}}
$$
Where $\rho_0$ is a constant.
One solution for DE was vacuum energy, but that description has it's own set of problems.
(say about 120 orders of magnitude of discrepancy...)
By the way, evidence suggests that the universe is right now expanding exponentially albeit very gradually.
Having said that, the universe might or might not be infinite per-say,
it is enough that it spans a space that is larger than our event horizon.
There is also a finite amount of energy at every instant of time in the universe,
even though, the existence of dark energy suggests that energy is added to the universe all the time.
A really nice book to "set you straight" is S. Dodelson's Introduction to Cosmology.
You only need to read the first chapter, and MAYBE chapter 8 (I think) that deals with inflation.
Best Answer
have a read through Did the Big Bang happen at a point? as this provides important background.
If, as you say, you are considering only a simply connected universe, so it isn't finite due to its topology, then the assumption we make when solving Einstein's equations is that the universe is the same everywhere - the technical terms are isotropic and homogeneous. So the universe has been infinite for as long as it has existed. The expansion doesn't mean it is expanding into anything, it just means the distance between things in the universe is increasing with time.