First of all: What is $y$? I assume, just some suitable fixed function?
Your approach 1) is certainly wrong since it ignores the dependence of $g$ on $f$.
Now your second approach is almost correct. However neither $\theta(x)$ nor $\theta(\tau)$ is correct. Instead you have to take $\theta(x-\tau)$, i.e. exactly the same argument that you insert into $f$. Now you can split the integral into to terms:
$$\frac{d}{d\varepsilon}\int f(x)\log\left(\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau\right)dx\quad\\\quad+\frac d{d\varepsilon}\int\varepsilon\theta(x)\log\left(\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau\right)dx$$
Now we first treat the first term which is equal to
$$\int\frac{f(x)\cdot(y*\theta)(x)}{\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau}dx$$
and setting $\varepsilon=0$ yields
$$\int\frac{f(x)\cdot(y*\theta)(x)}{(y*f)(x)}dx=\int\frac{f(x)\cdot(y*\theta)(x)}{g(x)}dx.$$
Now the second term is equal to
$$\int\theta(x)\log\left(\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau\right)dx+\int\frac{\varepsilon\theta(x)\cdot(y*\theta)(x)}{\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau}dx$$
and setting $\varepsilon=0$ yields
$$\int\theta(x)\log(g(x))\,dx.$$
Hence the final result should be $$\int\left(\frac{f(x)\cdot(y*\theta)(x)}{g(x)}+\theta(x)\log(g(x))\right)dx.$$
One additional remark: I have just seen that you write $g(x)=f(x)*y(x)$. You should avoid that and write $g(x)=(f*y)(x)$ instead.
To answer your question in the comment: yes you can express it in this form. This works as follows. First, we only consider the first term of the final result.
$$\int\frac{f(x)\cdot(y*\theta)(x)}{g(x)}dx=\int\frac{f(x)}{g(x)}\int y(\tau-x)\theta(\tau)\,d\tau\,dx.$$
Now we change the order of integration (of course you should justify this step rigorously but that needs more information about $y$ and the domain of $F$). Thus we obtain
$$\int\theta(\tau)\int \frac{f(x)}{g(x)}y(\tau-x)\,dx\,d\tau.$$
Of course we can relabel the integration variables and thus, by combining it with the second term of the result (which is already in the desired form) we obtain
$$\int\left(\int \frac{f(\tau)}{g(\tau)}y(x-\tau)\,d\tau+\log(g(x))\right)\theta(x)\,dx$$
Perhaps there is some neat way to rewrite the inner integral but I fail to see it.
Best Answer
The problem you found in the application of the formula for functional derivatives above is due the fact that it is not correct, since it is valid only for particular classes of functionals: the general definition of functional derivative is exactly the right side of the one you tried to use, i.e. $$ \frac{\delta F}{\delta p} [p](\phi)\triangleq \left[\frac{\mathrm{d}}{\mathrm{d}\epsilon}F(p +\epsilon\phi)\right]_{\epsilon=0} = \lim_{\epsilon \to 0} \frac{F(p +\epsilon\phi) -F(p)}{\epsilon}\label{1}\tag{1} $$ Said that, applying the definition to the quadratic functional under analysis we get $$ \frac{\delta F}{\delta p} [p](\phi) = 2\left[\int\cos(x)p(x)\mathrm{d}x - \int\cos(y)q(y)\mathrm{d}y\right] \int\cos(x)\phi(x)\mathrm{d}x $$ Edit following the comment of the Asker.
In order to clarify some of the doubts pointed out in the comment, I expanded a bit definition \eqref{1} above to analyze in a clearer way its meaning: it states that, in order to calculate the functional derivative of a functional $F$ at a "point" $p$, you should calculate the ordinary first order derivative of the functional applied to the variation $p+\epsilon \phi$, considered as a function of the real variable $\epsilon$. This implies that we should proceed in the calculations by using all the ordinary rules for differentiation including the chain rules: thus, when the functional $F$ is given by the composition of a ($C^\infty$-real valued) function of a real variable and a (real valued) functional $H$, i.e. $$ F(p)=g\circ H(P)= g\big(H(p)\big), $$ we have that $$ \begin{split} \frac{\delta F}{\delta p} [p](\phi) &= \left[\frac{\mathrm{d}}{\mathrm{d}\epsilon}g\big( H(p +\epsilon\phi)\big)\right]_{\epsilon=0} = \lim_{\epsilon \to 0} \frac{g\big( H(p +\epsilon\phi)\big) -g\big( H(p) \big)}{\epsilon}\\ & = \lim_{\epsilon \to 0} \frac{g\big( H(p +\epsilon\phi)\big) -g\big( H(p)\big)}{ {H(p +\epsilon\phi) -H(p)}}\cdot\frac{H(p +\epsilon\phi) - H(p)}{\epsilon}\\ &=\left. \frac{\mathrm{d}g(x)}{\mathrm{d} x}\right|_{x=H(p)}\cdot \left[\frac{\mathrm{d}}{\mathrm{d}\epsilon}H(p +\epsilon\phi)\right]_{\epsilon=0}\\ &=\left. \frac{\mathrm{d}g(x)}{\mathrm{d} x}\right|_{x=H(p)}\cdot \frac{\delta H}{\delta p} [p](\phi) \end{split} $$ which is basically an application (and the basic proof in disguise) of the chain rule formula.
Notes
An analysis of the general definition of functional derivatives is offered in this Q&A, where you can have a look in order to have more information and some references on the concept.
Regarding references: almost any book in the calculus of variation includes several examples of calculations of functional derivatives. However, they mostly (if not exclusively) deal with functional of integral type, which are not of the same kind of the functional studied here. Therefore my advice is to have a look at this, this, this and this Q&A which deal with functionals of non integral type. And, apart from consult the references given in the cited Q&A if needed, I advice to look at the first volume [1] of the monograph by Mariano Giaquinta and Stefan Hildebrandt, in my opinion a wonderful reference.
[1] Giaquinta, Mariano; Hildebrandt, Stefan (1996), Calculus of Variations I. The Lagrangian Formalism, Grundlehren der Mathematischen Wissenschaften, 310 (1st ed.), Berlin: Springer–Verlag, pp. xxix+475, ISBN 3-540-50625-X, MR 1368401, Zbl 0853.49001.