Consider the function
$$
f(x)=\left\{\begin{array}{rll} 1+x^2 & \text{if} & x \,\,\text{rational} \\ -x^2 & \text{if} & x \,\,\text{irrational}\end{array}\right.
$$
Then, for $x=0$, the limit $\lim_{h\to 0}\dfrac{f(h)-f(-h)}{2h}$ exists, although $f$ nowhere continuous.
Consider now the function
$$
f(x)=\left\{\begin{array}{rll} 1 & \text{if} & x=0 \\ 0 & \text{if} & x\ne 0\end{array}\right.
$$
Then the limit $\lim_{h\to 0}\dfrac{f(x+h)-f(x-h)}{2h}$ exists, for every $x$, although $f$ is not continuous at $x=0$. This example can be generalised, and obtain an $f$ which is discontinuous in countably many points (for example all the rationals), while the central difference converges.
Suppose now that limit $\lim_{h\to 0}\dfrac{f(x+h)-f(x-h)}{2h}$ exists for every $x$ is some open interval. Does this imply that $f$ is not differentiable in at most countably many points?