The answer is yes, in a very general way.
What I claim, first, is that the Lévy-Montague reflection theorem holds in ZF for any definable continuous cumulative hierarchical representation of the set-theoretic universe $V$. That is, if you have defined sets $U_\alpha$ for every ordinal $\alpha$, such that
- the sequence $\alpha\mapsto U_\alpha$ is definable, without parameters
- the sequence is monotone $\alpha\leq\beta\implies U_\alpha\subseteq U_\beta$
- the sequence is continuous, $U_\lambda=\bigcup_{\alpha<\lambda}U_\alpha$ for limit ordinals $\lambda$
- the sequence accumulates to the entire universe, $V=\bigcup_\alpha U_\alpha$
Then for every formula $\varphi(x)$ in the first-order language of set theory, there is some ordinal $\alpha$ such that $\varphi$ is absolute between $U_\alpha$ and $V$. Indeed, there is a closed unbounded class of such ordinals.
The proof is nearly identical to the usual proof of the reflection theorem — these hypotheses are what is used about the $V_\alpha$ hierarchy.
Once one has reflection, then one can define HOD using the hierarchy, as you suggest, by saying a set $a$ is ordinal definable, if there is some $\alpha$ and some ordinal parameters below $\alpha$, such that $a$ is definable in $U_\alpha$ from those parameters. A set is *hereditarily ordinal definable, if $a$ and every hereditary member of $a$ is ordinal definable.
In any model $V$ of ZF, if a set is ordinal definable, then by reflection it is ordinal definable in that sense. And conversely, if a set is ordinal definable in that sense, in some $U_\alpha$ using some formula $\varphi$ with ordinal parameters $\vec \beta$, then in $V$ we may define the set using parameters $\langle\alpha, \vec\beta,\varphi\rangle$ as parameters. (And note the very subtle point that we must sometimes use the formula $\varphi$ itself as a parameter, meaning its Gödel code, since in an $\omega$-nonstandard model the formula $\varphi$ used to define $a$ in $U_\alpha$ might be nonstandad. I view it as a kind of lucky miracle that ordinal-definability doesn't stumble on this problem.)
So this version aligns with the usual version of HOD.
Conclusion. It doesn't matter which definable continuous hierarchy you use when defining HOD. They all give the same class.
Another way to see this is to observe the following:
Theorem. For any such hierarchy $U_\alpha$ as above, there is a closed unbounded class of ordinals $\theta$ for which $$U_\theta=V_\theta.$$ In particular, any two such hierarchies agree on a class club.
Proof. This is the typical club argument. First, by continuity, the class of such ordinals $\theta$ is closed. And it is unbounded, since we can start in any $U_{\alpha_0}$, find a $V_{\alpha_1}$ containing all those elements, and then a $U_{\alpha_2}$ containing all those elements, and so on, building an alternating chain
$$U_{\alpha_0}\subseteq V_{\alpha_1}\subseteq U_{\alpha_2}\subseteq\cdots$$
If $\theta=\sup_n\alpha_n$ is the supremum, then $U_\theta=V_\theta$ by continuity. So this is a closed unbounded class. $\Box$
Since reflection also works on a class club, we see in this way that every formula $\varphi$ reflects to a ordinal on which the two hierarchies agree.
Best Answer
The proposition holds for the limit ordinal $\lambda$ iff $V_\lambda$ is wellorderable.
For if $V_\lambda$ is wellorderable where $\lambda$ is a limit, then it proposition easily follows at $\lambda$. Conversely, suppose the proposition holds at limit ordinal $\lambda$. (Without using AC) we can specify a certain ordinal $\eta$ and an injection $\pi:\eta\to V_\lambda$, such that $\eta$ cannot be injected into any $V_\alpha$ with $\alpha<\lambda$, but then applying the hypothesis with $S=\eta$, it follows that $|V_\lambda|=\eta$. To define $\eta,\pi$, first for each $\alpha<\lambda$, define $\eta_\alpha$ and an injection $\pi_\alpha:\eta_\alpha\to V_{\alpha+1}$, as follows. For ordinals $\beta$, let $W^\alpha_\beta$ be the set of all binary relations $R\in V_\alpha$ which are wellorders in ordertype $\beta$, and let $\eta_\alpha$ be the least $\beta$ such that $W^\alpha_\beta=\emptyset$. Then define $\pi_\alpha:\eta_\alpha\to V_{\alpha+1}$ by $\pi_\alpha(\beta)=(\alpha,W^\alpha_\beta)$. Now let $D=\{(\alpha,\beta)\bigm|\alpha<\lambda\wedge\beta<\eta_\alpha\}$, and define $\pi':D\to V_\lambda$ as $\pi'(\alpha,\beta)=\pi_\alpha(\beta)$. Note that $D$ is wellorderable, and let $\eta$ be an ordinal and $\sigma:\eta\to D$ a bijection. Then define $\pi:\eta\to V_\lambda$ by $\pi(\xi)=\pi'(\sigma(\xi))$. Then $\pi$ is an injection, so $|\eta|\leq|V_\lambda|$. I claim that $|\eta|=|V_\lambda|$. For otherwise we have $|\eta|<|V_\lambda|$. So applying the proposition at $\lambda$ with $S=\eta$, we get that $|\eta|<|V_\alpha|$ for some $\alpha<\lambda$. But this easily contradicts the construction.