Probably the simplest approach is, as Andy W suggested, to use a seasonal univariate time series model. If you use R, try either auto.arima() or ets() from the forecast package.
Either should work ok, but a general time series method does not use all the information provided. In particular, it seems that you know the shape of the curve in each year, so it might be better to use that information by modelling each year's data accordingly. What follows is a suggestion that tries to incorporate this information.
It sounds like some kind of sigmoidal curve will do the trick. e.g., a shifted logistic:
\begin{equation}
f_{t,j} = \frac{r_te^{a_t(j-b_t)}}{1+e^{a_t(j-b_t)}}
\end{equation}
for year $t$ and week $j$ where $a_t$, $b_t$ and $r_t$ are parameters to be estimated. $r_t$ is the asymptotic maximum, $a_t$ controls the rate of increase and $b_t$ is the mid-point when $f_{t,j}=r_t/2$. (Another parameter will be needed to allow the asymmetry you describe whereby the rate of increase up to time $b_t$ is faster than that after $b_t$. The simplest way to do this is to allow $a_t$ to take different values before and after time $b_t$.)
The parameters can be estimated using least squares for each year. The parameters each form time series: ${a_1,\dots,a_n}$, ${b_1,\dots,b_n}$ and ${r_1,\dots,r_n}$. These can be forecast using standard time series methods, although with $n=5$ you probably can't do much apart from using the mean of each series for producing forecasts. Then, for year 6, an estimate of the value at week $j$ is simply $\hat{f}(6,j)$ where the forecasts of $a_6$, $b_6$ and $r_6$ are used.
Once data start to be observed for year 6 you will want to update this estimate. As each new observation is obtained, estimate the sigmoidal curve to the data from year 6 (you will need at least three observations to start with as there are three parameters). Then take a weighted average of the forecasts obtained using the data up to year 5 and the forecast obtained using only the data from year 6, where the weights are equal to $(40-t)/36$ and $(t-4)/36$ respectively. That is very ad hoc, and I'm sure it can be made more objective by placing it in the context of a larger stochastic model. Nevertheless, it will probably work ok for your purposes.
Best Answer
Plain and simple: Include the time information when plotting your data and calculating your slope.
Right now, you use the following data for this:
What you should use is however:
Note how the first time value in each column corresponds to the average of the times corresponding to the values which you used to calculate the respective mean.