Given two functions $f, g : X \rightarrow X$, their composition $f \circ g$ is a new function $X \rightarrow X$. When $f \circ g$ is given an argument $x \in X$, it first applies $g$ to produce an element $g(x)$, and then applies $f$ to that. The result is $f(g(x))$, and this element is defined to be the value of $f \circ g$ in $x$.
Summing up, we get what Srivatsan wrote in a comment:
$$
(f \circ g)(x) = f(g(x)).
$$
Since we defined an operation that takes two functions $f$ and $g$ and produces a new function $f \circ g$, it makes sense to check if the operation makes the set $X^X$ of functions $X \rightarrow X$ into a monoid.
To do so, we need to prove that composition is associative, i.e.:
$$
(f \circ g) \circ h = f \circ (g \circ h)
$$
for any choice of $f, g, h$, and also that there exists a special function $i : X \rightarrow X$, such that, for all $f$:
$$
f \circ i = i \circ f = f.
$$
All of this is extremely easy to verify, just keep in mind that to check if two functions are equal is it sufficient to check that they have the same value on every possible argument.
First:
I know how to show that if $f,g,h$ are in $F$ then $F$ is associative
That sentence does not make sense. The conclusion, "$F$ is associative" is unconnected to the premise, "if $f,g,h$ are in $F$." The fact that the premise introduces notation for three things that are never again mentioned ($f$, $g$, and $h$) should tell you that there is something very wrong with that sentence.
Moreover, "associative" is a property of binary operations, not of sets. So $F$, which is not an operation, cannot "be associative."
What you need to show is that function composition is associative; that is, you need to show that if $f$, $g$, and $h$ are functions that can be composed, then $(f\circ g)\circ h = f\circ (g\circ h)$ as functions.
Then you need to show that $F$ is closed under function composition; that is, that if $f$ and $g$ are elements of $F$, then we can compute $f\circ g$ and $f\circ g$ will also be in $F$.
Those two things will show that $F$ is semigroup under function composition.
If your definition of "semigroup" requires your set to be nonempty, then you may also want to show that $F$ is nonempty by explicitly exhibiting an element of $F$.
Second:
I can show that $F$ has an identity element with $g=x$ such that $f(x)(g)=(g)((f(x))=f(x)$.
Again, this is rather badly written to the point of unintelligibility (though one can guess what you meant to write, what you actually wrote is nonsense). $g=x$ does not describe a function. If you want to describe a function, you'll want to say "the function $g\colon \mathbb{R}\to\mathbb{R}$ defined by the rule $g(x)=x$ for all $x\in \mathbb{R}$" or words to that effect. Also, "$f(x)(g)$" is nonsense as written, as is $(g)(f(x))=f(x)$. Instead, what you need to show is that $f\circ g = f$ and $g\circ f = f$ for all $f\in F$. That amounts to showing that for every $x\in\mathbb{R}$ $(f\circ g)(x) = f(x)$ and $(g\circ f)(x) = f(x)$.
Thirdly: assuming you have successfully done that, then think about what it would mean for a function $f$ to have an inverse. Inverse of what kind? Inverse with respect to composition. So a function $f\colon\mathbb{R}\to\mathbb{R}$ has an inverse in $(F,\circ)$ if and only if there exists $h\colon\mathbb{R}\to\mathbb{R}$ such that
$$h\circ f = g = \mathrm{id}_{\mathbb{R}}\quad\text{and}\quad f\circ h = g = \mathrm{id}_{\mathbb{R}}.$$
Now, do you know any properties that a function must satisfy in order to have an inverse in the sense of composition? Can you produce a function from $\mathbb{R}$ to itself that does not have those properties, and therefore is not invertible? If so, what do you conclude?
Best Answer
Inverses aren't relevant for monoids, but since you wrote about inverses, consider what I initially wrote below.
There are only $2^3 = 8$ functions from $A$ to $B$, so you could just check them all and show none of them is an identity element.