Following the proof given in Milnor's Topology From A Differentiable Viewpoint:
$\require{AMScd}$
$\begin{CD}
x \in U \subseteq M @>F>> c \in V \subseteq N \\
@V\phi VV @VV\psi V \\
\phi\left(U\right) \subseteq H^{m} @>>\psi F \phi^{-1}> \psi\left(c\right) \in \psi\left(V\right)
\end{CD}$
Because $c \in N$ is a regular value of $F$, for every $x \in F^{-1}\left(c\right) \subseteq M$, there are charts $\left(U,\phi\right)$ at $x$ in $M$ and $\left(V,\psi\right)$ at $c$ in $N$ such that $\psi F \phi^{-1}: \phi\left(U\right) \subseteq H^{m} \to \psi\left(V\right) \subseteq \mathbb{R}^{n}$ is smooth, and has a regular value at $\psi\left(c\right)$.
$\begin{CD}
\phi\left(U\right) \subset W \subseteq \mathbb{R}^{m} @>G>> \psi\left(c\right) \in \mathbb{R}^{n}
\end{CD}$
Let $W$ be an open subset of $\mathbb{R}^{m}$ such that $W \cap H^{m} = \phi\left(U\right)$; and let $G:W\to\mathbb{R}^{n}$ be the smooth extension of $\psi F \phi^{-1}$ over $W$. Now, we can always choose $W$ small enough so that $G^{-1}\left(\psi\left(c\right)\right)$ does not contain any critical points (Sard's lemma; $\mathbb{R}^{m}$ is regular). Thus, $\psi\left(c\right)$ is a regular value of $G$; and by preimage theorem (for smooth manifolds), $Z := G^{-1}\left(\psi\left(c\right)\right)$ is a (m-n)-dimensional submanifold of $\mathbb{R}^{m}$.
Furthermore, since $G$ is constant over $Z$, $T_{a}Z \subseteq ker\left\{DG\left(a\right)\right\}$ for every $a \in Z$. But $DG\left(a\right):\mathbb{R}^{m}\to\mathbb{R}^{n}$ is surjective, and the rank-nullity theorem implies $dim \left(ker\left\{DG\left(a\right)\right\}\right) =$ (m-n). Thus, $T_{a}Z = ker\left\{DG\left(a\right)\right\}$.
Now, define $\pi: Z \subseteq \mathbb{R}^{m} \to \mathbb{R}$ as $\left(x_{1},\ldots,x_{m}\right) \mapsto x_{m}$.
To show that $0 \in \mathbb{R}$ is a regular value of $\pi$:
Suppose otherwise. That is, suppose $\exists$ $a \in Z \cap \partial H^{m}$ such that $d\pi_{a}:T_{a}Z \to \mathbb{R}$ is not surjective. Then, $ker \left\{d\pi_{a}\right\}$ $=$ $T_{a}Z$ $=$ $ker \left\{DG(a)\right\}$. But, we know that $ker \left\{d\pi_{a}\right\} \subseteq \mathbb{R}^{m-1} \times \left\{0\right\} = \partial H^{m}$. Thus, $ker \left\{DG(a)\right\} \subseteq \partial H^{m}$ if $0$ is not a regular value for $\pi$.
Now, since $c \in N$ is a regular value of $F|_{\partial M}$ as well, arguing as before, we can show that $\bar{G} := G|_{W\cap\partial H^{m}}$ has a regular value at $\psi\left(c\right)$. That is, for every $a \in Z\cap\partial H^{m}$, $D\bar{G}\left(a\right): \mathbb{R}^{m-1} \to \mathbb{R}^{n}$ is surjective; and by rank-nullity theorem, $dim \left(ker\left\{D\bar{G}\left(a\right)\right\}\right) = $ (m-n-1)
Finally, $ker \left\{DG(a)\right\} \subseteq \partial H^{m}$ implies $ker \left\{DG(a)\right\} = ker \left\{D\bar{G}(a)\right\}$, which is clearly false (dimension mismatch). Hence, $0$ must be a regular value for $\pi$.
Since $0 \in \mathbb{R}$ is a regular value for $\pi$, $\left\{z \in Z | \pi\left(z\right) \geq 0\right\}$ $=$ $\phi\left(U \cap F^{-1}\left(c\right)\right)$ is a manifold with boundary $\left\{z \in Z | \pi\left(z\right) = 0\right\}$ $=$ $\phi\left(U \cap F^{-1}\left(c\right) \cap \partial M \right)$.
$\phi$ being a diffeomorphism, $U \cap F^{-1}\left(c\right)$ is a manifold with boundary $U \cap F^{-1}\left(c\right) \cap \partial M$. Observing that this is true for every $x \in F^{-}\left(c\right)$ completes the proof.
I've been strugling with this exercise for a while too. Here is my solution. I agree with your solution to show that $\pi : E \rightarrow M$ is a submersion. To show that $E_p = \pi^{-1}(p)$ is regular submanifold diffeomorphic to $F$, we can use Theorem 3.5 in Jeffrey Lee's book. That is we just need to show that there is a smooth immersion homeomorphic to the fibre $E_p$.
$\textbf{Proof that $E_p$ is a regular submanifold diffeomorphic to $F$}:$
Consider the diffeomorphism $\phi : \pi^{-1}(U) \rightarrow U \times F$. Note that $\{p\} \times F$ is a regular submanifold of $U\times F$. Because of this, the restriction of the smooth map $\phi^{-1}$ to $\{p\}\times F$ is smooth. That is we have a smooth map
$$
\phi^{-1}|_{\{p\}\times F} : \{p\}\times F \rightarrow \pi^{-1}(U)
$$
where the image of the domain is $\pi^{-1}(p)=E_p$. Also because $\phi^{-1}$ is a diffeomorphism, then $d\phi^{-1}$ is an isomorphism for any point in $U\times F$. Therefore the differential of the map $\phi^{-1}|_{\{p\}\times F}$ is injective at each point. So the map is an immersion. Therefore $E_p$ is a regular submanifold diffeomorphic to $F$.
(Note :The diffeomorphic part of this conclusion does not explicitly state in the theorem 3.5 but we can prove that. I prefer the same theorem in other book such as John Lee's smooth manifold Proposition 5.2 which is include this.)
$\textbf{Proof that if $F$ and $M$ is connected, then so is $E$}$ :
To proof this i need this following theorem from topology (e.g from Willard's book) : If a topological space $X$ is connected and $\mathscr{U}$ is an open cover for $X$, then any two points can be connected by a simple chain consisting of elements of $\mathscr{U}$.
By local trivialization for each $p \in M$ we have an open subset $U \subset M$ containing $p$ and a diffeomorphism $\phi : \pi^{-1}(U) \rightarrow U \times F$. Let $\{\pi^{-1}(U)\}$ be the open cover for $E$. By above theorem we can have a simple chain connecting any two points $v,w \in E$ if all the elements of the open cover $\{\pi^{-1}(U)\}$ connected. Because $F$ and $M$ connected, then $U\subset M$ connected, $U \times F$ connected, $\phi^{-1}(U\times F) = \pi^{-1}(U)$ connected. So we have a simple chain where each of its elements is connected (implies path-connectedness). By this we can easily make a path connecting $v,w \in E$ by joining the paths from each chain.
I think many ways to prove this but this is the one that i find it quite convincing. Let me know if you have another solution or found error in my proof.
$\textbf{EDIT}$:
To see more clear that $E_p$ and $F$ is diffeomorphic, we can just restrict the map $\phi : \pi^{-1}(U) \rightarrow U \times F$ to the domain and codomain (which is both regular submanifold) $E_p$ and $\{p\} \times F$ respectively to obtain the map
$$
\phi|_{E_p}: E_p \rightarrow \{p\} \times F
$$
Because the restriction of a smooth map to domain (or codomain) which is a regular submanifold is smooth, then the map above is a diffeomorphism. However, the alternative for the second proof (about connectedness of $E$) you can look at here Show that the total space $E$ of a fibre bundle $\pi : E \rightarrow M$ is connected.
Remark about restriction of smooth map to regular submanifold :
$\bullet$ Restriction to Domain
Let $F : M \rightarrow N$ be a smooth map and $S \subset M$ is a regular submanifold. Let $\iota : S \hookrightarrow M$ is the inclusion map. Then $F|_S = F \circ \iota : S \rightarrow N$ is smooth.
I know this result first from John Lee's book smooth manifold (p.112), but i didnt found it (explicitly) in Jeff Lee's book (it doesnt mean that its not there, because i'm not reading Jeff Lee's book thorough), maybe its because this result can easily proved. Here is the proof from John Lee's :
Notice that $S \subset M$ is a embedded submanifold (which is a regular submanifold in Jeff Lee's terminology) so by definition $\iota : S \hookrightarrow M$ is a smooth map. Therefor $F|_S = F \circ \iota : S \rightarrow N$ is a smooth map because its the composition of smooth maps.
Here the argument "$S\subset M \quad \text{reg. submanifold} \implies \iota : S \hookrightarrow M \quad \text{smooth}$ " follows from definition because John Lee define embedded submanifold (regular submanifold) very carefully from the beginning (look its definition on page 98). If you want to refer to Jeff Lee's book, he is mention it in p.132 (paragraph above Corollary 1.35). It states as
"... to say that inclusion $S \hookrightarrow M$ is an embedding (which is a smooth map) is easily seen to be the same that $S$ is a regular submanifold...".
But i dont think its really that easy for beginner to see. So i think the better way to study submanifold carefully is by John Lee's book. However i just found its direct proof in L.Tu's book Theorem 11.14
$\bullet$ Restriction to Codomain
The result for this available in John Lee's (p.113 Corollary 5.30) and Jeff Lee's books (p.132 Corollary 3.15). However, as usual, i prefer John Lee's. Here is what he says
Let $M$ be a smooth manifold and $S\subset M$ be an embedded submanifold (regular submanifold in Jeff Lee's). The every smooth map whose image contain in $S$ is also smooth as a map from $N$ to $S$.
By combining these two results, we can safely says that
$$
\phi|_{E_p}: E_p \rightarrow \{p\} \times F
$$
is a diffeomorphism.
[Look Walter Poor's Differential Geometric Structure for the same proof (basically) except he use the Regular Level Set Theorem to show that $E_p = \pi^{-1}(p)$ is regular submanifold of $E$.]
Best Answer
The inclusion map is a submersion, since the derivative is an isomorphism at each point, hence a surjection. (Presumably you've proved in class or as an exercise that submersions are always open maps. In this case, you could apply the inverse function theorem directly.)