Definition A explicitly talks about subsets of real numbers, but in fact is perfectly applicable to any partially ordered set. By contrast, your Definition B does not explicitly say it is about real numbers, but in fact is only about subsets of the entire real numbers with the usual ordering.
In fact, we don't just talk about "least upper bound" when dealing with partially ordered sets, we talk about "least upper bound in $X$".
Your argument for equivalence assumes in fact that you are dealing with all of $\mathbb{R}$: to see how it can fail, consider the totally ordered subset of the real numbers given by $X=[0,1)\cup\{2\}$. Let $S=[0,1)$. Then $\{2\}$ is a least upper for $S$ in $X$, but your argument breaks down, since for $\epsilon=\frac{1}{2}$ we have that every $s\in S$ satisfies $s\leq 2-\frac{1}{2}$; yet $2$ is the least upper bound of the set $S$ in $X$, even though it does not satisfy Definition 2. The reason is that although, within the set of real numbers, $2-\epsilon$ "works" as an upper bound for $S$, this element is not in $X$; not being a member of the club of elements of $X$, it does not qualify to be a least upper bound in $X$, let alone the least upper bound.
So, definition A is valid in arbitrary partially ordered sets. Definition B is only equivalent to Definition A in some contexts, e.g., when dealing with the set of all real numbers in their usual ordering.
As far as what your lecturer said, it's hard to go by second-hand reports. It is possible that he said that in a partially ordered set $X$, a subset $S$ may have more than one minimal upper bound: an element $b\in X$ is a minimal upper bound for $S$ if and only if $s\leq b$ for all $s\in S$, and if $a$ is an upper bound for $S$ and $a\leq b$, then $a=b$.
As to your example, it does not really work because "$a-\epsilon$" has no inherent meaning; it can only mean something if you have an operation defined that allows you to subtract $\epsilon$ from $a$; since your $a$ and $b$ are not actually a real numbers (at least, not under the usual ordering) you cannot talk about $a-\epsilon$ directly; you have to explain what that means. And I would say that $a$ and $b$ are minimal upper bounds rather than "least upper bounds", since they both fail Definition A (which is the one that is applicable in generality).
I also want to note that your argument that Definition A and B are equivalent is badly phrased (possibly because you are talking about (1) and (2) and don't specify if you are talking about definition A or definition B). You should assume that $b$ satisfies (A) and prove that it satisfies (B) (you essentially do this, but you are assuming that your set is all of $\mathbb{R}$, or at least dense), and then assume that $b$ satisfies (B) and prove it satisfies (A).
$\sup S$ is shorthand for the least upper bound of $S$ in some set
$T\supset S$ with respect to an order $\leq$ on $T$. However, we
often omit $T$ and $\leq$ when they are obvious from the context.
Consider the case when $T=\mathbb{Q}$ and $\leq$ is the usual order.
Since $\sup\left\{ x\in\mathbb{Q}:x^{2}<2\right\} $ has an upper
bound but no least upper bound (in $T=\mathbb{Q}$), this supremum does not
exist. Hence, $\mathbb{Q}$ does not satisfy the least upper bound
property (a.k.a. Dedekind completeness).
However, if instead we had $T=\mathbb{R}$, the above supremum is $\sqrt{2}$.
Another example is given by @Hagen von Eitzen, where we look for $\sup \emptyset$. Regardless of whether $T=\mathbb{Q}$ or $T=\mathbb{R}$, in both of these spaces, $0$ is an upper bound of $\emptyset$. Furthermore, if $x$ is an upper bound of the empty set, so too is $x-1$. Therefore, there exists no least upper bound. We can write this as $\sup\emptyset = -\infty$ (in fact, if $T=\overline{\mathbb{R}}$, the extended reals with the obvious order, this is a precise statement).
Note that this does not contradict Dedekind completeness of the reals because Dedekind completeness only requires bounded nonempty sets to have supremums.
Best Answer
It need not be inside or outside as you have illustrated in your example.
In the last paragraph, notice a few keywords of which I will highlight.A non-empty finite subset has a greatest element, in this case, they must be inside the set.
If a set has a greatest element, then it must be inside the set.