As others have mentioned, you can consider left/right derivatives, or even Dini derivatives of functions.
Generally, when we are looking at a real-valued function $G$ that is only defined on an interval $[a,b]$, we think of the derivatives at $a$ and $b$ as simply being the right-hand and left-hand derivatives at those points (respectively). That is, when we look at the derivative $G'(a) = \lim_{x \rightarrow a} \frac{G(x)-G(a)}{x-a}$, the limit should be taken using only values of $x$ that lie inside $[a,b]$, since that is the domain we are working in. This is equivalent to taking the limit only from the right of $a$.
To show that the right derivative of $F$ at $a$ is $f(a)$, we can just use the same argument from one of the standard proofs of the FTC.
For $x \in [a,b]$, we have that $F(x) = \int_a^x f(t)dt$.
Then $$F(x) - F(a) = \int_a^x f(t) dt - \int_a^a f(t) dt = \int_a^x f(t)dt,$$
as $\int_a^a f(t)dt = 0$.
By the Mean Value Theorem for Integrals, for each $x \in (a, b)$, there exists $c_x \in [a, x]$ such that $$\int_a^x f(t)dt = f(c_x)(x-a).$$ Rearranging, we obtain that
$$\frac{F(x)-F(a)}{x-a} = f(c_x),$$
so that the right derivative of $F$ at $a$ is
$$\lim_{x \rightarrow a^+} \frac{F(x)-F(a)}{x-a} = \lim_{x \rightarrow a^+} f(c_x)$$
Since each $c_x$ lies in the interval $[a, x], c_x$ must converge to $a$ as $x$ approaches $a$. Since $f$ is continuous on $[a, b]$, we have that $\lim_{x \rightarrow a^+} f(c_x) = f(a)$, so that the right derivative of $F$ at $a$ is $f(a)$, as desired.
The argument for the left derivative of $F$ at $b$ is analogous.
Best Answer
Try $f(x)=x(1-x)\,\sin\frac{1}{x(1-x)}$ on $(0,1)$ and $f(0)=f(1)=0.$ It's obviously continuous, and the difference quotient at the endpoints takes all values between -1 and 1.