Yes, it is the case that $\mu_f' = \mu_v$.
We will use the notation introduced in section "Attempt #2" of the original post. Additionally, we make the following two definitions that will be used throughout the proof:
$$
\begin{align}
Z &:= (a, b] \\
\mathcal{J} &:= \left\{(c, d] :\mid c, d\in [a, b] \right\}
\end{align}
$$
The proof breaks down into four steps:
1) We show that it suffices to show that $\mu_f'(E) = \mu_v(E)$ for all $E \in \mathcal{B}(Z)$ in order to conclude that $\mu_f' = \mu_v$.
2) We show that $\mu_f^\pm(J) \geq \mu_v^\pm(J)$, respectively, for all $J \in \mathcal{J}$.
3) We deduce that $\mu_f^\pm(E) = \mu_v^\pm(E)$, respectively, for all $E \in \mathcal{B}(Z)$.
4) We conclude that $\mu_f'(E) = \mu_v(E)$ for all $E \in \mathcal{B}(Z)$.
The proof hinges on two characterizations of the Jordan decomposition of signed measures given in proposition 6.21(i & ii) in Dshalalow's "Foundations of Abstract Algebra", 2nd edition and in the version of the Jordan decomposition theorem given in Doob's "Measure Theory".
===
We show that it suffices to show that $\mu_f'(E) = \mu_v(E)$ for all $E \in \mathcal{B}(Z)$ in order to conclude that $\mu_f' = \mu_v$, as follows. (1.1) We show that $\mu_v(Z^c) = 0$. (1.2) We show that $\mu_f'(Z^c) = 0$. (1.3) We conclude that it suffices to show that $\mu_f'(E) = \mu_v(E)$ for all $E \in \mathcal{B}(Z)$ in order to conclude that $\mu_f' = \mu_v$.
1.1 We will show that $\mu_v(Z^c) = 0$. Since $Z^c = (-\infty, a] \cup (b, \infty)$, it suffices to show that $\mu_v\left((-\infty, a]\right), \mu_v\left((b, \infty)\right) = 0$.
$$
\begin{align}
\mu_v\left((-\infty, a]\right) & = \mu_v\left(\bigcup_{n = 0}^\infty \left(a - (n + 1), a - n\right]\right) \\
& = \sum_{n = 0}^\infty \mu_v\left(\left(a - (n + 1), b - n\right]\right) \\
& = \sum_{n = 0}^\infty \left(v_f(a - n) - v_f(b - (n + 1))\right) \\
& = \sum_{n = 0}^\infty (0 - 0) \\
& = 0 \\
\mu_v\left((b, \infty)\right) & = \mu_v\left(\bigcup_{n = 0}^\infty \left(b + n, b + (n+1)\right]\right) \\
& = \sum_{n = 0}^\infty \mu_v\left(\left(b + n, b + (n+1)\right]\right) \\
& = \sum_{n = 0}^\infty \left(v_f(b + (n + 1)) - v_f(b + n)\right) \\
& \overset{(*)}{=} \sum_{n = 0}^\infty V_{b + n}^{b + (n + 1)}(f) \\
& \overset{(**)}{=} 0
\end{align}
$$
where equality $(*)$ is a fundamental result from the theory of functions of bounded variation (see e.g. lemma 12.15(b) in Yeh's "Real Analysis", 2nd edition), and equality $(**)$ is due to the fact that, for $s, t \in [b, \infty)$ with $s \leq t$, we have
$$
\begin{align}
V_s^t(f) &= \sup \left\{\sum_{k = 1}^n \left|f(t_k) - f(t_{k - 1}) \right| :\mid s = t_0 \leq t_1 \leq \cdots \leq t_n = t, n \in \{1, 2, \dots\}\right\} \\
& = \sup \left\{\sum_{k = 1}^n \left|f(b) - f(b) \right| :\mid s = t_0 \leq t_1 \leq \cdots \leq t_n = t, n \in \{1, 2, \dots\}\right\} \\
& = 0
\end{align}
$$
1.2 We show that $\mu_f'(Z^c) = 0$, as follows. (1.2.1) We construct finite measures $\mu^+$ and $\mu^-$ on $(\mathbb{R}, \mathcal{B})$ such that $\mu_f = \mu^+ - \mu^-$ and for which $\mu^\pm(Z^c) = 0$. (1.2.2) We apply a certain minimality condition on the Jordan decomposition of signed measures to show that $\mu_f^\pm = \mu^\pm$, respectively. (1.2.3) We conclude that $\mu_f'(Z^c) = 0$.
1.2.1 We will construct finite measures $\mu^+$ and $\mu^-$ on $(\mathbb{R}, \mathcal{B})$ such that $\mu_f = \mu^+ - \mu^-$ and for which $\mu^\pm(Z^c) = 0$. Define the set functions $\mu^\pm : \mathcal{B} \rightarrow [0, \infty]$ as follows:
$$
\mu^\pm(E) := \mu_f^\pm(E \cap Z)
$$
respectively.
The fact that $\mu^\pm$ are measures follows from the fact that $\mu_f^\pm$ are measures. The fact that $\mu^\pm$ are finite follows from the observation that $\mu^\pm \leq \mu_f^\pm$, respectively, and from the observation that $\mu_f^\pm$ are finite, since $\mu_f = \mu_g - \mu_h$ is. As for $\mu^\pm(Z^c)$, let's calculate: $\mu^\pm(Z^c) = \mu_f^\pm(Z^c \cap Z) = \mu_f^\pm(\emptyset) = 0$.
Finally, to show that $\mu_f = \mu^+ - \mu^-$ it suffices to show that $\mu_f(Z^c) = 0$, for suppose we have shown that $\mu_f(Z^c) = 0$, then for $E \in \mathcal{B}$,
$$
\begin{align}
\mu_f(E) &= \mu_f(E \cap Z) + \mu_f(E \cap Z^c) \\
&= \mu_f(E \cap Z) \\
&= \mu_f^+(E \cap Z) - \mu_f^-(E \cap Z) \\
&= \mu^+(E) - \mu^-(E)
\end{align}
$$
We will now show that $\mu_f(Z^c) = 0$. Since $Z^c = (-\infty, a] \cup (b, \infty)$, it suffices to show that $\mu_f\left((-\infty, a]\right), \mu_f\left((b, \infty)\right) = 0$.
$$
\begin{align}
\mu_f\left((-\infty, a]\right) & = \mu_f\left(\bigcup_{n = 0}^\infty \left(a - (n + 1), a - n\right]\right) \\
& = \sum_{n = 0}^\infty \mu_f\left(\left(a - (n + 1), a - n\right]\right) \\
& = \sum_{n = 0}^\infty \left(\mu_g\left(\left(a - (n + 1), a - n\right]\right) - \mu_h\left(\left(a - (n + 1), a - n\right]\right) \right) \\
& = \sum_{n = 0}^\infty \left(\left(g(a - n) - g(a - (n + 1))\right) - \left(h(a - n) - h(a - (n + 1))\right)\right) \\
& = \sum_{n = 0}^\infty \left(\left(g(a) - g(a)\right) - \left(h(a) - h(a)\right)\right) \\
& = 0 \\
\mu_f\left((b, \infty)\right) & = \mu_f\left(\bigcup_{n = 0}^\infty \left(b + n, b + (n+1)\right]\right) \\
& = \sum_{n = 0}^\infty \mu_f\left(\left(b + n, b + (n+1)\right]\right) \\
& = \sum_{n = 0}^\infty \left(\mu_g\left(\left(b + n, b + (n+1)\right]\right) - \mu_h\left(\left(b + n, b + (n+1)\right]\right) \right) \\
& = \sum_{n = 0}^\infty \left(\left(g(b + (n + 1)) - g(b + n)\right) - \left(h(b + (n + 1)) - h(b + n)\right)\right) \\
& = \sum_{n = 0}^\infty \left(\left(g(b) - g(b)\right) - \left(h(b) - h(b)\right)\right) \\
& = 0
\end{align}
$$
1.2.2 We will show that $\mu_f^\pm = \mu^\pm$, respectively. By definition $\mu^\pm \leq \mu_f^\pm$, respectively, so we are left to show that $\mu^\pm \geq \mu_f^\pm$, respectively. According to the version of the Jordan decomposition theorem given in Doob's "Measure Theory", if $(\Omega, \mathcal{A}, \lambda)$ is a signed measure space, if $\lambda = \lambda^+ - \lambda^-$ is the unique Jordan decomposition of $\lambda$, and if $\lambda = \lambda_1 - \lambda_2$ is any representation of $\lambda$ as the difference between two measures, then $\lambda^+ \leq \lambda_1$ and $\lambda^- \leq \lambda_2$. Therefore, since by (1.2.1) $\mu_f = \mu^+ - \mu^-$, we obtain $\mu^\pm \geq \mu_f^\pm$, respectively, as desired.
1.2.3 We will show that $\mu_f'(Z^c) = 0$. Indeed,
$$
\mu_f'(Z^c) = \mu_f^+(Z^c) +\mu_f^-(Z^c) = \mu^+(Z^c) + \mu^-(Z^c) = 0
$$
1.3 We will now show that it suffices to show that $\mu_f'(E) = \mu_v(E)$ for all $E \in \mathcal{B}(Z)$ in order to conclude that $\mu_f' = \mu_v$. Indeed, suppose we have shown that, for every $E \in \mathcal{B}(Z)$, $\mu_f'(E) = \mu_v(E)$. Then, using (1.1) and (1.2) we see that for $E \in \mathcal{B} = \mathcal{B}(\mathbb{R})$ we have
$$
\begin{align}
\mu_f'(E) & = \mu_f'\left((E \cap Z) \cup (E \cap Z^c)\right) \\
& = \mu_f'(E \cap Z) + \mu_f'(E \cap Z^c) \\
& = \mu_f'(E \cap Z) \\
& = \mu_v(E \cap Z) \\
& = \mu_v(E \cap Z) + \mu_v(E \cap Z^c) \\
& = \mu_v\left((E \cap Z) \cup (E \cap Z^c)\right) \\
& = \mu_v(E)
\end{align}
$$
We show that $\mu_f^\pm(J) \geq \mu_v^\pm(J)$, respectively, for all $J \in \mathcal{J}$, as follows. (2.1) We first show that for every $J \in \mathcal{J}$, there is some collection of subsets $S_J \subseteq \mathcal{B}(J)$, such that $\mu_v^+(J) = \sup_{E \in S_J} \mu_f(E)$. (2.2) We deduce, by citing a suitable property of the Jordan decomposition of signed measures, that for every $J \in \mathcal{J}$, $\mu_f^+(J) = \sup_{E \in \mathcal{B}(J)} \mu_f(E) \geq \mu_v^+(J)$. (2.3) Using a similar argument we show that for every $J \in \mathcal{J}$, $\mu_f^-(J) \geq \mu_v^-(J)$.
2.1 We will show that, for every $J \in \mathcal{J}$, there is some collection of subsets $S_J \subseteq \mathcal{B}(J)$, such that $\mu_v^+(J) = \sup_{E \in S_J} \mu_f(E)$. Let $J \in \mathcal{J}$. If $J = \emptyset$, we can take $S_J$ to be $\{\emptyset\}$. Otherwise $J = (c, d]$ for some $c, d \in [a, b]$ such that $c < d$. Define $\mathcal{P}_J$ to be the set of strict partitions of $[c, d]$, that is to say $\mathcal{P}_J$ is the collection of all finite sets of points $\{c = t_0 < t_1 < \cdots < t_n = d\}$.
For every $Q = \{c = t_0 < t_1 < \cdots < t_n = d\} \in \mathcal{P}_J$ define
$$
\begin{align}
\alpha(Q) & := \bigcup_{k \in \{j \in \{1, 2, \dots, n\} \mid: f(t_j) > f(t_{j - 1})\}} (t_{k - 1}, t_k]\\
\beta(Q) & := \sum_{k = 1}^n \max\left(f(t_k) - f(t_{k - 1}), 0\right)
\end{align}
$$
and note that $\mu_f\left(\alpha(Q)\right) = \beta(Q)$, since for any $s, t \in \mathbb{R}$ such that $s < t$,
$$
\begin{align}
\mu_f\left((s, t]\right) &= \mu_g\left((s, t]\right) - \mu_h\left((s, t]\right) \\
&= \left(g(t) - g(s)\right) - \left(h(t) - h(s)\right) \\
&= \left(g(t) - h(t)\right) - \left(g(s) - h(s)\right) \\
&= f(t) - f(s)
\end{align}
$$
Now define
$$
S_J := \{\alpha(Q) :\mid Q \in \mathcal{P}_J\}
$$
Then $S_J \subseteq \mathcal{B}(J)$ and furthermore
$$
\mu_v^+(J) = v_f^+(d) - v_f^+(c) \overset{(***)}{=} {V^+}_c^d(f) = \sup_{Q \in \mathcal{P}_J} \beta(Q) = \sup_{E \in S_J} \mu_f(E)
$$
Equality $(***)$ can be proved similarly to equality $(*)$ above (in section 1.1).
2.2 We will show that for every $J \in \mathcal{J}$, $\mu_f^+(J) \geq \mu_v^+(J)$. According to proposition 6.21(i) in Dshalalow's "Foundations of Abstract Algebra", 2nd edition, for every $D \in \mathcal{B} = \mathcal{B}(\mathbb{R})$, $\mu_f^+(D) = \sup_{E \in \mathcal{B}(D)} \mu_f(E)$. Therefore, using the result of the previous step,
$$
\mu_f^+(J) = \sup_{E \in \mathcal{B}(J)} \mu_f(E) \geq \sup_{E \in S_J} \mu_f(E) = \mu_v^+(J)
$$
2.3 We can show that $\mu_f^-(J) \geq \mu_v^-(J)$ for all $J \in \mathcal{J}$ using arguments analogous to those presented in (2.1) and (2.2) (this time invoking Dshalalow's proposition 6.21(ii)).
We show that $\mu_f^\pm(E) = \mu_v^\pm(E)$, respectively, for all $E \in \mathcal{B}(Z)$, as follows. (3.1) Firstly we show that $\mu_f^\pm(J) = \mu_v^\pm(J)$, respectively, for all $J \in \mathcal{J}$. We do so by applying a certain minimality property of the Jordan decomposition of signed measures to the results obtained in step (2) above. (3.2) We generalize (3.1) using Dynkin's $\pi$-$\lambda$ theorem to show that $\mu_f^\pm(E) = \mu_v^\pm(E)$, respectively, for all $E \in \mathcal{B}(Z)$.
3.1 We will show that $\mu_f^\pm(J) = \mu_v^\pm(J)$, respectively, for all $J \in \mathcal{J}$. Recall that in step (2) above we showed that $\mu_f^\pm(J) \geq \mu_v^\pm(J)$, respectively, for all $J \in \mathcal{J}$. Therefore, it is left to show that, for all $J \in \mathcal{J}$, $\mu_f^\pm(J) \leq \mu_v^\pm(J)$. Indeed, according to the version of the Jordan decomposition theorem given in Doob's "Measure Theory", if $(\Omega, \mathcal{A}, \lambda)$ is a signed measure space, if $\lambda = \lambda^+ - \lambda^-$ is the unique Jordan decomposition of $\lambda$, and if $\lambda = \lambda_1 - \lambda_2$ is any representation of $\lambda$ as the difference between two measures, then $\lambda^+ \leq \lambda_1$ and $\lambda^- \leq \lambda_2$. Therefore, recalling from section "Attempt #2" of the original post that $\mu_f = \mu_v^+ - \mu_v^-$, we obtain that $\mu_f^\pm \leq \mu_v^\pm$, respectively.
3.2 We will show that $\mu_f^\pm(E) = \mu_v^\pm(E)$, respectively, for all $E \in \mathcal{B}(Z)$. $\mu_f^+$ and $\mu_v^+$ are positive measures that coincide on $\mathcal{J}$. Since $\mathcal{J}$ is a $\pi$-system that generates $Z$ and that includes $Z$, we conclude (for instance, by using Dynkin's $\pi$-$\lambda$ theorem) that $\mu_f^+(E) = \mu_v^+(E)$ for all $E \in \mathcal{B}(Z)$. By the same reasoning, $\mu_f^-(E) = \mu_v^-(E)$ for all $E \in \mathcal{B}(Z)$.
We will show that or all $E \in \mathcal{B}(Z)$, $\mu_f'(E) = \mu_v(E)$. Recall from section "Attempt #2" of the original post that $\mu_v = \mu_v^+ + \mu_v^-$. Therefore, using (3.2), for all $E \in \mathcal{B}(Z)$,
$$
\mu_f'(E) = \mu_f^+(E) + \mu_f^-(E) = \mu_v^+(E) + \mu_v^-(E) = \mu_v(E)
$$
Q.E.D.
Acknowledgements
The proof idea was suggested to me by the last sentence of section X.6, "Functions of bounded variation vs. signed measures", on p. 164 of Doob's "Measure Theory", Springer 1993.
You cannot enumerate the set of partitions of $[a,b]$. In other words, there is no list $\{P_1,P_2,\ldots\}$ which consists of all partitions of $[a,b]$. Other than that, you are correct:
- You consider the set $\mathcal P$ of all partitions of $[a,b]$.
- For each $P\in\mathcal P$, if $P=\{x_0,x_1,\ldots,x_n\}$, you compute the sum $\displaystyle J_P=\sum_{i=0}^{n-1}\bigl|f(x_i)-f(x_{i+1})\bigr|$.
- Then the total variation of $f$ is $\sup\{J_P\mid P\in\mathcal P\}$.
Concerning your question but whether you could take the sum with respect to a “single general partition”, you will have to explain first what that is. Note that, in general, if $P,P^\star\in\mathcal P$, then $J_P\ne J_{P^\star}$. And, (again, in general) there is no partition $P$ such that the total variation is equal to $J_P$; in other words, that supremum is not, in general, a maximum.
Best Answer
To see that the limit and supremum in question might produce difference results define $f:[0,1]\to[0,1]$ as $$ f(x) = \begin{cases} 0, &\text{ if } \ x \neq 1/2, \\ 1, &\text{ if } \ x = 1/2. \end{cases} $$ Clearly $\mathrm{Var}(f) = 2$, but any partition of $[0,1]$, with diameter however small, will result in $0$ variance-sum unless it includes the point $1/2$ in the partition of $[0,1]$.