$\def\d{{\,\prime}}$
$\def\nR{{\mathbb{R}}}$
$\def\l{\left}\def\r{\right}$
$\def\vf{{\vec{f}}}$
$\def\vc{{\vec{c}}}$
$\def\vr{{\vec{r}}}$
$\def\ve{{\vec{e}}}$
$\def\vF{{\vec{F}}}$
$\def\rmF{{\mathrm{F}}}$
$\def\rmC{{\mathrm{C}}}$
$\def\rmN{{\mathrm{N}}}$
$\def\rmT{{\mathrm{T}}}$
$\def\nN{{\mathbb{N}}}$
What your teacher tried to explain is most easily described with the help of the principle of virtual work.
The sphere poses a constraint to the chain. It defines the shape of the chain. The chain itself implicitly represents also a constraint (it does not lengthen under pull forces). The sphere does not move therefore it does not exert any work on the chain (we neglect friction here).
Let the curve $\vf:\nR\rightarrow\nR^3$ describe the shape of the constraint in dependence on the curve length. We have extended the domain of this function to full $\nR$ to avoid problems when the chain moves.
For our example we can use
$$
\vf(s) = \begin{cases}
(R,s) & @ s < 0\\
\l(R\cos\l(\frac sR\r),R\sin\l(\frac sR\r)\r) &@ s\in\l[0,\frac\pi2R\r]\\
\l(\frac{\pi}2R-s,R\r) &@ s > \frac\pi2R
\end{cases}
$$
Note, that we do not really need the actual representation. It is just added that we know about what we speak. The curve $\vf$ is represented with red color in the following picture together with its parameterization through $s$.
For simplicity we do not care about the bending of the
chain if it slides off the sphere on the thread side just as if
it were also supported by some shelf there.
If $s$ is the positioning parameter of the chain in space and $\xi\in [0,l]$ is the actual position of the chain element on the chain the chain can be described by
$$
\vr(s,\xi) = \vf(s-\xi).
$$
For $\xi=0$ the point $\vr(s,0)=\vf(s)$ is the beginning of the chain. For $\xi=l$ the point $\vr(s,l)=\vf(s-l)$ is the end of the chain. Values of $\xi$ in between of $0$ and $l$ address other points $\vr(s,\xi)$ on the chain. If we enlarge $s$ the chain glides up along the curve $\vf$ if we reduce $s$ the chain glides down along the curve $\vf$. To see this you can follow the chain beginning at $\vr(s,0)=\vf(s)$.
The placement of the chain from your picture results for $s=\frac\pi2 R$.
Excursion into mechanics with constraints (Principle of Work):
At first consider a single particle constraint to some curve. In
reality the particle could be a very small and heavy locomotive on a
curved rail track in the mountains (up and down).
The curve is given by a smooth function $\vf:\nR\rightarrow\nR^3$ in
dependence of some parameter $s$ (e.g. the path length in one
direction starting from some predefined point as for our above
example).
The particle is affected by some free force $\vF_\rmF$ which would
also exists if the constraint wouldn't be there. Furthermore, the
particle is affected by the constraint force $\vF_\rmC$ which keeps
the particle on the curve.
The constraint force counteracts the component $\vF_\rmN$ of the free force the
normal direction of the curve.
The resulting force is the tangent component $\vF_\rmT$ of the free force.
The particle is in equilibrium if the resulting tangent force $\vF_\rmT=\vF_\rmF+\vF_\rmC$ is
zero.
The nice thing of the principle of virtual work is that we do not need
to calculate explicitly the normal force but we can restrict the
equations to the tangent direction.
The tangent direction of the curve at some location $s$ is the
derivative $\vf\,'(s)$ of $\vf(s)$. We can scale the tangent direction
by some parameter $\delta s$. In this way we get a straight line
parameterized by $\delta s$ which is tangent to the curve at the point
$s$. Note, that close to $s$ the curve and the tangent line look very similar
(inclusively the scale division for $s$ and $\delta s$).
For each given value $\delta s$ the vector $\delta\vr:=\vf\,'(s)\cdot
\delta s$ is a tangent vector on the curve at $s$. The tangent
quantities such as $\delta \vr$ and $\delta s$ are often denoted as
virtual in physics. And if physicans talk about small deviations
$\delta s$ they actually mean such tangent quantities.
The force balance in the equilibrium point is
$$
\vec0 = \vF_\rmF + \vF_\rmC
$$
If we multiply this equation by a tangent vector at the location $s$ of the particle we get
$$
0 = \delta\vr\cdot(\vF_\rmF + \vF_\rmC) = \delta\vr\cdot\vF_\rmF + \underbrace{\delta\vr\cdot\vF_\rmC}_{=0}
$$
whereby the second term $\delta\vr\cdot\vF_\rmC$ is zero because the
tangent vector $\delta\vr$ and the constraint force $\vF_\rmC$ are
perpendicular to each other.
So our resulting equilibrium equation is
$$
0=\delta\vr\cdot \vF_\rmF = \delta s \vf\,'(s)\cdot \vF_\rmF.
$$
We are only considering here the case of one degree of freedom $s$
where we can just set $\delta s=1$. (Note by the way, that if we had
more parameters, e.g., $ s=( s_1, s_2)$, we would have to test for all
possible tangent directions, e.g. $\delta s=(1,0)$ and $\delta
s=(0,1)$.)
For $\delta s=1$ our equilibrium equation reads as
$$
0= \vf\,'(s)\cdot \vF_\rmF.
$$
which is one scalar equation which can be solved for the scalar
curve parameter $s$.
As I already mentioned above. The nice thing about our first
application of the principle of virtual displacement is that we have
eliminated the unknown constraint forces in the equation to be
solved.
Now we consider a system of $n$ coupled particles with one degree of
freedom $s$. All particles are constraint to $n$ curves $\vr_i =
\vf_i(s)$ with $i=1,\ldots,n$.
Now, there must be additional internal constraint forces $\vF_{ij}$
between the particles else they would run on the curves $\vf_i$
independently and their positions could not be parameterized by the
same parameter $s$.
Thereby, $\vF_{ij}$ is the force exerted by particle $j$ on particle $i$.
Because of actio=reactio we have $\vF_{ij} = -\vF_{ji}$. Therefore we
only need the forces $\vF_{ij}$ with $i>j$ in our formulas.
Inclusively the internal constraint forces the equilibrium force
balance for the $i$-th particle reads as
$$
0=\sum_{j=1}^{i-1} \vF_{ij} - \sum_{j=i+1}^n \vF_{ji} + \vF_{\rmC i} + \vF_{\rmF i}.
$$
The $\vF_{\rmC i}$ are the constraint forces from the environment and
$\vF_{\rmF i}$ are the free forces on the $i$-th particle as for our
one-particle problem.
What are examples of internal constraint and how do the internal
constraints work? A simple internal constraint would be a light staff
with a heavy particle attached to each end. To show that this is not
the only kind of internal constraint we can extend this example. We do
not attach the particles directly but put them on two light wheels
which are attached rotary to the ends of the staff such that they work
together like a mechanism with one degree of freedom such that the
motion of the particles is coupled and can be expressed by one parameter $s$.
In a difference coordinate system $\vr_j-\vr_i$ the point $\vr_j$ is
constrained to the curve $\vf_j(s)-\vf_i(s)$ and we have the same
situation as for the constrained single particle. The constraint force
$\vF_{ji}$ is perpendicular to the tangent of the difference
curve
$$
(\delta\vr_j-\delta\vr_i)\perp \vF_{ji}\quad\Rightarrow\quad
(\delta\vr_j-\delta\vr_i)\cdot \vF_{ji} = 0.
$$
From the single-particle example we have some hope to eliminate the
constraint forces by exploiting this orthogonality.
But, in the balance formula $0=\sum_{j=1}^{i-1} \vF_{ij} -
\sum_{j=i+1}^n \vF_{ji} + \vF_{\rmC i} + \vF_{\rmF i}$ for the force
on the $i$-th particle all internal constraint forces are mixed
in. Furthermore, since we only have one parameter $s$ to determine the
position of all particles we only need one formula.
For that reason let us try the sum over all the scalar products:
$$
0 = \sum_{i=1}^n \delta \vr_i \cdot \l(\sum_{j=1}^{i-1} \vF_{ij}-
\sum_{j=i+1}^n \vF_{ji} + \vF_{\rmC i} + \vF_{\rmF i}\r)
$$
$$
0= \sum_{i=1}^n\sum_{j=1}^{i-1} \delta \vr_i \cdot\vF_{ij}-
\sum_{i=1}^n\sum_{j=i+1}^n \delta \vr_i \cdot\vF_{ji} +\sum_{i=1}^n \delta \vr_i \cdot \vF_{\rmC i} + \sum_{i=1}^n \delta \vr_i \cdot\vF_{\rmF i}
$$
From the single-particle problem we already know $\delta\vr_i\cdot \vF_{\rmC i}=0$ and there remains only the balance equation
$$
0= \sum_{i=1}^n\sum_{j=1}^{i-1} \delta \vr_i \cdot\vF_{ij}-
\sum_{i=1}^n\sum_{j=i+1}^n \delta \vr_i \cdot\vF_{ji} + \sum_{i=1}^n \delta \vr_i \cdot\vF_{\rmF i}
$$
Let us have a closer look onto the first two sums with the internal constraint forces $\vF_{ij}$
$$
\sum_{i=1}^n\sum_{j=1}^{i-1} \delta \vr_i \cdot\vF_{ij}- \sum_{i=1}^n\sum_{j=i+1}^n \delta \vr_i \cdot\vF_{ji}
$$
For easier comparison of the terms in the two sums we exchange the names of the indexes in the second one:
$$
\sum_{i=1}^n\sum_{j=1}^{i-1} \delta \vr_i \cdot\vF_{ij}- \sum_{j=1}^n \sum_{i=j+1}^n \delta \vr_j \cdot\vF_{ij}
$$
That is just re-naming and does not change the value of the sum. Let
us have a look onto the set of index pairs over which we have to sum
in the first term and in the second one. We can do this graphically in
the following diagram. There mark the locations $(i,j)$ of the index
pairs which are included in the sum with points. I just draw some of
these points and put a gray triangle onto the region where we would
have to mark all grid points.
If we do the same for the second term we see that we obtain exactly
the same triangle and therefore also the same set of index pairs.
If you prefer we can also check this in set-notation. The set of index
pairs for the first term is:
$$
\{(i,j)\in\nN^2 \mid 1\leq i\wedge i\leq n \wedge 1 \leq j\wedge j \leq i-1\}
$$
Just transforming the last inequality:
$$
=\{(i,j)\in\nN^2 \mid 1\leq i\wedge i\leq n \wedge 1 \leq j\wedge j+1 \leq i\}
$$
Just reordering of the inequalities:
$$
=\{(i,j)\in\nN^2 \mid 1 \leq j \wedge 1\leq i \wedge j+1 \leq i\wedge i\leq n\}
$$
The inequalities $1\leq j$ and $j+1\leq i$ already imply $1\leq i$
therefore we can drop this inequality. Furthermore, $j+1\leq i$
implies the inequality $j\leq n$ which we can add without danger. This gives the set
$$
=\{(i,j)\in\nN^2 \mid 1 \leq j \wedge j\leq n \wedge j+1 \leq i\wedge i\leq n\}
$$
of index pairs for the second term.
Since the sets of index pairs for the sums are the same we can combine the summands under the same sum sign
$$
\sum_{i=1}^n\sum_{j=1}^{i-1} \delta \vr_i \cdot\vF_{ij}- \sum_{j=1}^n \sum_{i=j+1}^n \delta \vr_j \cdot\vF_{ij}
=\sum_{i=1}^n\sum_{j=1}^{i-1} \l( \delta \vr_i \cdot\vF_{ij}-\delta \vr_j \cdot\vF_{ij}\r)
$$
and factor out the internal constraint forces
$$
=\sum_{i=1}^n\sum_{j=1}^{i-1} \underbrace{\l( \delta \vr_i -\delta \vr_j \r)\cdot\vF_{ij}}_{=0} = 0
$$
Eventually the sum becomes zero with the orthogonality condition for
the internal constraint forces $\l( \delta \vr_i -\delta \vr_j
\r)\cdot\vF_{ij}=0$ which we discussed further above.
Exploiting this our balance equation
$$
0= \sum_{i=1}^n\sum_{j=1}^{i-1} \delta \vr_i \cdot\vF_{ij}-
\sum_{i=1}^n\sum_{j=i+1}^n \delta \vr_i \cdot\vF_{ji} + \sum_{i=1}^n \delta \vr_i \cdot\vF_{\rmF i}
$$
becomes just
$$
0 = \sum_{i=1}^n \delta \vr_i \cdot\vF_{\rmF i}.
$$
The sum over the products of the virtual displacements $\delta\vr_i$
with the free forces $\vF_{\rmF i}$ is zero.
This formula is called the Principle of Virtual Work.
Just to get a formula usable for calculation we substitute $\delta\vr_i=\vf_i\,'(s)\cdot\delta s$
$$
0 = \sum_{i=1}^n \delta s \vf_i\,'(s) \cdot\vF_{\rmF i}
$$
and set $\delta s=1$ which gives
$$
0 = \sum_{i=1}^n \vf_i\,'(s) \cdot\vF_{\rmF i}.
$$
This is one scalar formula for our one degree of freedom $s$ in the
system.
If our system of particles is actually a continuum instead of discrete
particles the index $i\in \{1,\ldots,n\}$ for $\vr_i(s)$ is replaced by a continuous
parameter $\xi\in [0,l]$ with some length $l>0$ and we write $\vr(s,\xi)$ instead of $\vr_i(s)$.
Beside discrete forces $\vF_{\rmF i}$ which operate at certain
places $\xi_{\rmF i}$ $i=1,\ldots,n_{\rmF}$ of the constrained
continuum there can also act a distributed force per length
$\vF_{\rmF}^\d(\xi,\vr)$ on the continuum.
To get a transition from the principle of virtual work for discrete
particle systems to continuous particle systems one can start with a
with a discretization of the continuum.
We dissect the continuum into $k$ sections $j=1,\ldots,k$ with length
$\frac lk$. The parameter value for the $j$-th section runs from
$\xi=\xi_{j-1}:=(j-1)\frac lk$ to $\xi_j:=j\frac{l}{k}$. Furthermore,
we address each of the sections by some intermediate parameter
$\xi^*_j$ with $\xi_{j-1}<\xi^*_j<\xi_j$. For an instance $\vr(s,\xi_j^*)$
can be the center of mass of the $j$-th section.
If the pieces are small enough, i.e., the number of pieces $k$ is high
enough, a discrete particle system will be a good approximation of the
system of pieces. We consider the points $\vr(s,\xi_j^*)$ at the
intermediate parameter values $\xi_j^*$ as location of the $j$-th particle.
The free force caused by the distributed force per length on the $j$-th piece will be:
$$
\int_{\xi_{j-1}}^{\xi_j} \vF_\rmF^\d(\vr(s,\xi))d\xi\approx \vF_\rmF^\d(\vr(s,\xi^*_j))\cdot(\xi_j-\xi_{j-1})
$$
Now we apply the principle of virtual work to this approximative particle system and obtain
$$
0 = \sum_{j=1}^{k}\delta \vr(s,\xi_j^*)\cdot \vF_\rmF^\d(\vr(s,\xi^*_j))\cdot(\xi_j-\xi_{j-1})
+ \sum_{i=1}^{n_{\rmF}} \delta \vr(s,\xi_{\rmF i}) \cdot \vF_{\rmF i}
$$
we expect the approximation to get better with higher with larger $k$.
For $k=1,2,\ldots$ we obtain a sequence of Riemann sums for the first
sum which is converging to the integral $\int_0^l
\delta\vr(s,\xi)\cdot\vF_{\rmF}^\d(s,\vr(s,\xi)) d\xi$ for $k\rightarrow\infty$.
This leads us to the principle of virtual work for a 1d-continuum with one degree of freedom $s$:
$$
0 = \int_0^l\delta\vr(s,\xi)\cdot\vF_{\rmF}^\d(s,\vr(s,\xi)) d\xi
+ \sum_{i=1}^{n_{\rmF}} \delta \vr(s,\xi_{\rmF i}) \cdot \vF_{\rmF i}
$$
Thereby, the integral accounts for the distributed free force acting
on the continuum and the sum accounts for the discrete free forces
acting on it.
The application of the principle of virtual work delivers
$$
F_s\delta s + \int_{\xi=0}^l \delta\vr(s,\xi)\cdot \vF_{\rm G}^\d d \xi = 0
$$
where $F_s$ is the supporting force of the chain, i.e., the thread force, $\vF_{\rm G}^\d = -\lambda g\ve_z$ is the weight force per length and $\delta\vr(\xi,s) = \frac{\partial \vr}{\partial s}(s,\xi)\delta s$ is the virtual vectorial displacement of the string.
$$
\l(F_s + \int_{\xi=0}^l \frac{\partial \vr}{\partial s}(s,\xi)\cdot \vF^\d_{\rm G} d \xi\r)\delta s = 0
$$
and if this should hold for virtual chain displacements $\delta s\neq 0$ we get the condition
$$
F_s = -\int_{\xi=0}^l \frac{\partial\vr}{\partial s}(s,\xi)\cdot \vF^\d_{\rm G} d \xi
$$
Note, that $\frac{\partial \vr}{\partial s}$ is just the unit vector in tangential direction of the chain. So, because of the constraint we are just integrating the tangential component $\frac{\partial \vr}{\partial s}(s,\xi)\cdot \vF^\d_{\rm G}$ of the weight force.
You could say that the constraint force of the sphere deviates the pull force in the chain.
The problem with your approach is that you are missing the deviation force of the chain in the calculation of the normal force $dN$. If you cut out a (finite) piece of chain then the ends of this piece will not have the same tangent direction because of the curvature of the sphere. If some force (the weight of the remainder of the chain) acts tangentially on the lower end of the piece it cannot be fully compensated by the tangent force at the higher end (the tangent vectors are linearly independent). The force difference must be compensated by a normal force on the shpere.
We can also just use another approach to check the result.
Assumption: no friction, no damping.
Potential energy of chain depending on the start angle $\theta$ on the sphere
$$
E_{\rm pot}(\theta) = g\lambda\underbrace{(l-R\theta)}_{\small\mbox{length}}\underbrace{\left(-\frac{l-R\theta}2\right)}_{\small\mbox{height}}+\int_0^\theta g \underbrace{R\sin(\bar\theta)}_{\mbox{height}}\underbrace{\lambda R d\bar\theta}_{d m} =
g\lambda\left(-\frac{(l-R\theta)^2}2 +R^2(1-\cos(\theta))\right)
$$
The cut force in the thread on the chain-side is
$$
F=\frac1{R}\frac{d E_{\rm pot}}{d\theta} = \frac{g\lambda}{R}\left(-R(R\theta-l) + R^2\sin(\theta)\right)
$$
which gives at $\theta=\frac{\pi}2$
$$
F=g\lambda\left(l-R\frac{\pi}2+R\right)
$$
Hopefully, without any mistakes. But, you should check. So, in the end this calculation delivers also solution A.
Note, a very good book on classical mechanics (with constraints) is Arnold's "Mathematical Methods of Classical Mechanics". This is a graduate text but the first chapters should already be readable for advanced students from high-school.
Best Answer
Reducing a two body problem into a effective one body problem is just an mathematical manipulation to simplify the coupled differential equations, that you have to solve, and convert them to an easier problem (two uncoupled differential equations). There is no approximation involved. Actually, it is just a transformation of variables and after you have solved the problem in these variables you can transform back and you have the exact solution of your original problem. The concept of a reduced mass is only introduced, because after the transformation the differential equation of the relative coordinate looks like one of a fictitious particle with this particular mass.