We first start with the usual definition of limit.
Let $c$ be a real number / point. A neighborhood of $c$ is an open interval $I$ containing $c$. If $I$ is a neighborhood of $c$ then the set $I\setminus \{c\} $ is said to be a deleted neighborhood of $c$.
Let $A$ be a non empty subset of $\mathbb {R} $ and let $f:A\to\mathbb {R} $ be a function. Further let $c$ be a point such that there exists some deleted neighborhood of $c$ contained in $A$.
A number $L$ is said to be the limit of $f$ at $c$ (written symbolically as $L=\lim\limits_{x\to c} f(x) $) if the following condition holds: for every $\epsilon>0$ there exist a $\delta>0$ corresponding to it such that $|f(x) - L|<\epsilon$ whenever $x\in A$ and $0<|x-c|<\delta$.
Such a number $L$ may or may not exist, but if it exists then the above definition ensures that it must be unique (see proof later).
The above definition can be generalized a bit to include domains which are not necessarily neighborhoods. And the question here provides such a general definition using concept of limit points.
The same generalization however does not apply to one sided limits because of lack of corresponding concepts like left / right limit points. The idea of one sided limits is restricted to introductory courses of calculus where typical domains are intervals/neighborhoods. In that context here is a definition of left hand limit.
Let $A$ be a non-empty subset of $\mathbb {R} $ and let $f:A\to\mathbb {R} $ be a function. Further let $c$ be a point such that there exists a number $h>0$ with $(c-h, c) \subseteq A$.
A number $L$ is said to be the left hand limit of $f$ at $c$ (written symbolically as $L=\lim\limits_{x\to c^{-}} f(x) $) if the following condition holds: for every $\epsilon>0$ there exists a $\delta>0$ corresponding to it such that $|f(x) - L|<\epsilon $ whenever $x\in A$ and $0<c-x<\delta$.
The key prerequisite in the notion of limit is that if $c$ is the point under consideration then the function must be defined at points arbitrarily near to $c$. For one sided limits the function must be defined at points arbitrarily near to and on the corresponding side of $c$.
The uniqueness of a limit is based on the fact that values of the function are near the limit $L$. If $L_1,L_2$ are distinct then you can not ensure values $f(x) $ to be arbitrarily near to both $L_1,L_2$ at the same time. If you go too close to $L_1$ then you have to necessarily move away from $L_2$.
This is more technically expressed as follows:
Theorem (with easy proof) : Let $a, b$ be two real numbers with $a\neq b$. Then there exists a neighborhood $I$ of $a$ and a neighborhood $J$ of $b$ such that $I\cap J=\emptyset$.
This key fact is the basis of the general notion of a Hausdorff space.
Best Answer
If $c$ is not a limit point of $A$, then for all small enough $\delta > 0$, the set $\{ x \in A : 0 < \lvert x-c\rvert < \delta\}$ is empty, and the condition
$$(x\in A \land 0 < \lvert x-c\rvert < \delta) \implies \lvert f(x) - L\rvert < \varepsilon$$
is vacuously satisfied for every $L$. Thus the limit would not be unique if we didn't require $c$ to be a limit point of $A$.