A possible strategy is the following (I did not work on details, e.g. might work for even/odd $n$ only, and the expressions may not be accurate, but the general idea is hopefully clear) -
Starting from a graph with no edge, we add colored edges in iterations, such that in iteration $i$ each node has degree $2i$ (or $i$ for even $n$?). Let $p_i$ be the probability of success until the end of iteration $i$, and $q_i$ the probability of failing before the end of iteration $i$. Then considering we add $n$ new edges in iteration $i$, we have:
$$ p_{i-1} \cdot \left(1-\frac{4i}{m}\right)^n \le p_i \le p_{i-1} \cdot \left(1-\frac{2i}{m}\right)^n$$
which makes it easy to estimate the desired probability $p_{n/2}$. Also, the failure probability can be estimated combining the above with the following:
$$q_i = q_{i-1} + (1-q_{i-1})\cdot (1-p_{i})$$
Though I'm not sure if this is a sufficiently delicate solution!
This proof considers the even-sized subsets of vertices of a pentagon. There are 16 such subsets: The empty set, the five sides of $P$, the five diagonals of $P$, and the five sets of size 4.
Each subset corresponds to a vertex in the graph we want to color.
The color of the edge between two vertices is found by considering the symmetric difference of their two subsets.
The symmetric difference operation $\triangle$ takes any two such subsets to a third.
If the subsets are different, the result cannot be empty, and so must be one of the other types: a side, a diagonal, or size 4.
The coloring strategy is simply to use one color for each of these three types.
Any triangle $\{a,b,c\}$ in the graph we are coloring consists of three edges, colored according to the type of $f(a) \triangle f(b)$, the type of $f(b) \triangle f(c)$, and the type of $f(a) \triangle f(c)$.
This is easy to reason about, because the identity
$$
\underbrace{(f(a) \triangle f(b))}_{\mbox{subset 1}} \; \triangle \;
\underbrace{(f(b) \triangle f(c))}_{\mbox{subset 2}} =
\underbrace{f(a) \triangle f(c)}_{\mbox{subset 3}}
$$
tells us that if there is a monochromatic triangle, then there must be two subsets of the same type whose symmetric difference is also of the same type.
But the symmetric difference of two sides is either a diagonal or of size 4, depending on whether the sides shared a vertex or not.
Similarly, the symmetric difference of two diagonals is either a side or of size 4, depending on whether the diagonals shared a vertex or not.
And finally, the symmetric difference of two distinct subsets of size 4 (so each subset is only missing a single vertex) has to be a subset of size 2 (namely the two missing vertices), and is thus either a side or a diagonal.
$\scriptsize\mbox{(You might wonder whether the final coloring of the graph is symmetric in all three colors, since it is clearly symmetric}$
$\scriptsize\mbox{in the first two (by rearranging the pentagon to make the diagonals be the sides), and surprisingly, the answer is yes!)}$
Best Answer
The second generalization (all the vertices on a hyperedge must have different colors) is just a graph coloring problem in disguise. Define a graph by putting an (ordinary, $2$-vertex) edge between any two vertices that are on the same hyperedge. Then we are just trying to find a proper coloring of this graph.
The first generalization, where we only require that the vertices on a hyperedge are not all the same color, is a genuinely new problem. Even if it's a bit harder to think of applications for it, this is the only worthwhile definition, because it's the definition of a problem we couldn't talk about before.
(I'm exaggerating a bit; the "rainbow chromatic number" of a hypergraph, where every vertex on a hyperedge has a different color, does see some use. That's because, even though it reduces to a graph coloring problem, it reduces to a graph coloring problem for a special kind of graph that's a union of cliques, and we might still be able to say something about it that doesn't hold in general.)