The problem here is that you are using the Ito formula for Lévy processes – your process is not a Lévy process, but a semimartingale that is driven by a Lévy process (read the 8.3.4 – "Ito Formula for Seminartingales" in the Tankov book). For example, at a point $t$ where there is a jump $j_t$, your process doesn't satisfy
$$
S_t = S_{t-} + j_t
$$
but rather
$$
S_t = j_tS_{t-}
$$
More precisely, you process is a Dolean-Dades exponential of a Lévy process.
So you have to use Ito for semimartingales (proposition 8.19 in Tankov), i.e.
$$
\begin{align}
f(t, X_t) - f(0, X_0) &= \int^t_0 \frac{\partial f}{\partial s}(s, X_s) ds + \int_0^t\frac{\partial f}{\partial x}(s, X_s) dX_s \\
&+ \frac{1}{2} \int^t_0 \frac{\partial^2 f}{\partial x^2}(s, X_s) d[X, X]_s^c \\
&+\sum_{s \leq t, \Delta X_s \neq 0} [f(s, X_s) - f(s, X_{s-}) - \Delta X_s\frac{\partial f}{\partial x}(s, X_s)]
\end{align}
$$
where $[X, X]_s^c$ is the quadratic variation of the continuous part of $X$.
This formula is less straight-forward to apply, because for example if you insert $S_t$ as $X_t$ and use $\log$ as $f$, then the second term on RHS is
$$
\int_0^t\frac{\partial f}{\partial x}(s, S_s) dS_s = \int_0^t\frac{1}{S_t}S_t(\mu ds + \sigma dW_s + dj) = \mu t + \sigma W_t + \int^t_0dj
$$
How do you interpret the last integral in this situation? It is actually an integral against the jump measure of $S_t$, $J_S(x, t)$. This is pure hand-waving, but for now I just argue that this is the sum of jumps up to time $t$:
$$
\int^t_0dj = \int_0^t \int_\mathbb R xJ_S(ds, dx) = \sum_{s\leq t, \Delta S_s \neq 0} j_s
$$
This part will cancel against the term $- \Delta X_s\frac{\partial f}{\partial x}(s, X_s)$ inside the sum in the Ito for semimartingales so that essentially, you will be left with the usual Ito for Lévy processes, except that the jump part will look like
$$
f(X_t) - f(X_{t-})
$$
instead of
$$f(X_{t-} + \Delta X_t) - f(X_{t-})$$
Thus, at a point where there is a jump, you get
$$
f(S_t) - f(S_{t-}) = f(j_t S_{t-}) - f(S_{t-}) = \ln(j_t) + \ln(S_{t-}) - \ln(S_{t-})
$$
As for your solution attempt I'm not sure what you tried to do there – it seems you are trying to derive an Ito formula of your own. I suggest using the original formula with the said modification for the jumps.
Your process corresponds to
$$
S_t = S_0 + \int^t_0 S_s \mu ds + \int^t_0 \sigma S_s dW_s + \sum S_s j_s
$$
from which you can read off $a_t$, $b_t$, and then plug into the Ito formula. For example,
$$
a_t \frac{\partial f(S_t, t)}{\partial x} dt = \mu S_t \frac{1}{S_t} dt = \mu dt
$$
After going on a literature hunt I found an answer in this script on page 30.
The integral is decomposed into its off-diagonal terms (first row) and its diagonal/quadratic terms (second row).
The scaling factor of 2 occurs since we're integrating twice over the same domain.
Now to the quadratic term:
We are working with a compensated Poisson process which means that the process is centered around zero.
$$
\mathbb{E}[Y_t] = \lambda \mathbb{E}[Z] t
$$
Discretizing the integral gives us:
$$
\sum (\Delta \phi_{t-})^2 \Delta[Y_t - \lambda \mathbb{E}[Z]]\ \Delta[Y_t - \lambda \mathbb{E}[Z]]
$$
Substituting in the mean $\mathbb{E}[Y_t] = \mathbb{E}[Y_t]$ yields:
$$
\sum_t (\Delta \phi_{t-})^2 \Delta[Y_t - \mathbb{E}[Y_t]]^2
$$
where $ \Delta[Y_t - \mathbb{E}[Y_t]]^2$ is the diagonal term of the covariance for an infinitessimal small timestep $\Delta$.
We further know that the variance for a Poisson process $N_t$ of length $t$ is
$$
\mathbb{V}[N_t] = \lambda t
$$
and that the variance of a scaled random variable $X$ is
$$
\mathbb{V}[aX] = a^2 \mathbb{V}[X].
$$
Since the jumps of the compensated Poisson process are modulated of sorts by the random variable $Z$ we obtain:
$$
\sum_t (\Delta \phi_{t-})^2 \Delta[Y_t - \mathbb{E}[Y_t]]^2 \\
= \sum_t (\Delta \phi_{t-})^2 \mathbb{V}[Z \Delta N_t] \\
= \sum_t (\Delta \phi_{t-})^2 Z^2 \mathbb{V}[\Delta N_t] \\
= \sum_t (\Delta \phi_{t-})^2 Z^2 \lambda \Delta t
$$
the continuous version of which is
$$
\int \phi_{t-}^2 Z^2 \lambda dt
$$
Best Answer
You can deduce some of the properties directly from the statement about $f(X_t)$.
Firstly, $f$ has to be twice differentiable (otherwise $\frac{1}{2}\int_0^t f''(X_s)|u_s|^2ds$ would be undefined).
Secondly, $u_s$ has to be square-integrable (again, otherwise the integral above would be undefined).
Thirdly, $v_s$ has to be integrable (otherwise $\int_0^tv_sf'(X_s)ds$ would be undefined).
In addition, $u_t$ has to be measurable with respect to the filtration generated by $B_t$ (otherwise the integral $\int_0^tu_sdB_s$ would be undefined), and $\eta_t$ has to be measurable with respect to the filtration generated by $Y_t$ (otherwise the integral $\int_0^t\eta_sdY_s$ would be undefined).