I think you could not arrive at your desired Feynman rules because your second equation is wrong. You see, given your definition of the graviton field as a small perturbation to the Minkowski metric, we can rewrite
\begin{align*}
g^{\mu\nu} g^{\rho\sigma} F_{\mu\rho} F_{\nu\sigma} &\approx (\eta^{\mu\nu} \eta^{\rho\sigma} - \kappa h^{\mu\nu} \eta^{\rho\sigma} - \kappa h^{\rho\sigma} \eta^{\mu\nu}) F_{\mu\rho} F_{\nu\sigma} \\
&= F^2 + 2 \kappa {h^\mu}_\nu\ \partial_{[\mu}A_{\rho]} \partial^{[\rho}A^{\nu]}\,. \\
\end{align*}
Moreover, you made a mistake in your expansion of the invariant measure.
$$ \sqrt{-g} \approx 1 + \frac{\kappa}{2} \eta_{\alpha \beta}h^{\alpha \beta}$$
These considerations give your Interaction Lagrangian the following form:
$$ \boxed{\mathcal L_I = -\frac{\kappa}{4} \eta_{\alpha \beta}h^{\alpha \beta} \partial_{\mu}A_{\nu} \partial^{[\mu}A^{\nu]} - \frac{\kappa}2 {h^\mu}_\nu\ \partial_{[\mu}A_{\pi]} \partial^{[\pi}A^{\nu]}}\,. \tag4$$
You do not need to write your Lagrangian in the quadratic form (e.g. $A_\sigma \hat{N}^{\sigma\lambda}_{\mu\nu} A_\lambda$) unless, of course, you want to find the free propagator of your theory (which is $\hat{N}^{-1}$, if you have that free term in your Lagrangian, which you do not).
Note that there are six terms in the above Lagrangian which I will list down below.
- $-\frac{\kappa}{4} \eta_{\alpha \beta}\ h^{\alpha \beta}\ \partial_{\mu}A_{\nu}\ \partial^{\mu}A^{\nu}$
- $+\frac{\kappa}{4} \eta_{\alpha \beta}\ h^{\alpha \beta}\ \partial_{\mu}A_{\nu}\ \partial^{\nu}A^{\mu}$
- $- \frac{\kappa}2 h^{\mu\nu}\ \partial_{\mu}A_{\pi}\ \partial^{\pi}A_{\nu}$
- $+ \frac{\kappa}2 h^{\mu\nu}\ \partial_{\pi}A_{\mu}\ \partial^{\pi}A_{\nu}$
- $+ \frac{\kappa}2 h^{\mu\nu}\ \partial_{\mu}A_{\pi}\ \partial_{\nu}A^{\pi}$
- $- \frac{\kappa}2 h^{\mu\nu}\ \partial_{\pi}A_{\mu}\ \partial_{\nu}A^{\pi}$
For each of these terms, you can try to write down the momentum-space contribution to the vertex interaction,
as follows. Consider, without loss of generality, the first term from the above list: $-\frac{\kappa}{4} \eta_{\alpha \beta}\ h^{\alpha \beta}\ \partial_{\mu}A_{\nu}\ \partial^{\mu}A^{\nu}$.
- Step 1: Start with the constant coefficients of the interaction term (such as $-\frac{\kappa}{4} \eta_{\alpha \beta}$). Write them down with an extra factor of $i$, the imaginary number.
- Step 2: Assign to each field in the interaction term (such as $h^{\alpha \beta}$) a corresponding schematic in the Feynman diagram (such as $h^{\rho \sigma}$) and write down the Lorentz indices of the pairing in terms of Minkowski matrices (such as $\eta^{\alpha \rho}\eta^{\beta \sigma}$).
- Step 3: Write down the derivatives that act on a certain field (such as $\partial_{\mu}A_{\nu}$) as the momentum of the corresponding field in the Feynman diagram (such as $\eta_{\nu\alpha}{k_1}_\mu$ for choice of $A_\alpha$ as the representative schematic).
- Step 4: Add all possible contributions due to different choices of schematic assignments (such as $\eta_{\nu\beta}{k_2}_\mu$ for choice of $A_\beta$ as the representative schematic for $\partial_\mu A_\nu$).
The result should look like this.
$$ -\frac{i\kappa}{4} \eta_{\alpha \beta} \eta^{\alpha \rho}\eta^{\beta \sigma} (\eta_{\nu\alpha}{k_1}_\mu {\eta^\nu}_\beta {k_2}^\mu + \eta_{\nu\beta}{k_2}_\mu {\eta^\nu}_\alpha {k_1}^\mu) = -\frac{i\kappa}{2} (k_1 \cdot k_2) \ \eta^{\rho \sigma} \eta_{\alpha \beta} \tag{5.1}$$
You can consider the above steps as Feynman meta-rules to find the Feynman rules for any "well-behaved" theory. Try working out the other terms (#2 through #6) by yourself and see if you can get the following results from each of these terms.
$$ + \frac{i\kappa}{2} \eta^{\rho \sigma} {k_1}_\beta {k_2}_\alpha \tag{5.2}$$
$$ - \frac{i\kappa}{2} ({\eta_\alpha}^\sigma {k_2}^\rho {k_1}_\beta + {\eta_\beta}^\sigma {k_1}^\rho {k_2}_\alpha) \tag{5.3}$$
$$ + \frac{i\kappa}{2} (k_1 \cdot k_2)\ {\eta^\rho}_{(\alpha} {\eta_{\beta)}}^\sigma \tag{5.4}$$
$$ + \frac{i\kappa}{2} \eta_{\alpha\beta} {k_1}^{(\rho} {k_2}^{\sigma)} \tag{5.5}$$
$$ - \frac{i\kappa}{2} ({\eta_\alpha}^\rho {k_2}^\sigma {k_1}_\beta + {\eta_\beta}^\rho {k_1}^\sigma {k_2}_\alpha) \tag{5.6}$$
These are the ten terms that you wanted to have (eq. 3). I hope that helps. Please comment below if you do not understand something and need further clarification.
NOTE:
To understand why the meta-rules work the way they do, please read a standard book on Quantum Field Theory. I would recommend A. Zee: Quantum Field Theory in a Nutshell, Ch. 1.7. You will also find this resource quite useful.
Best Answer
I don't get how to apply these rules, when the interaction contains derivatives, either. If you are unsure you can go the long way. If you do not know how, here is how one could do it:
Try to evaluate (e.g. for the $g\phi \partial_\mu \phi \partial^{\mu} \phi$ interaction) the $\mathcal{O}(g)$ greens functions $G(x_1,x_2,x_3)$ of 3 external scalar particles, e.g. by using wicks theorem, and see what the vertex rule is. In position space you get 6 terms of the form: $-ig\int d^4x D(x_1-x) (\partial_\mu D)(x_2-x) (\partial^\mu D)(x_3-x)$. Then consider the momentum space greensfunction $\tilde{G}(k_1,k_2,k_3) = \int d^4x_1 \int d^4x_2 \int d^4x_3 G(x_1,x_2,x_3) e^{-ik_1x_1}e^{-ik_2x_2}e^{-ik_3x_3}$ (usual convention is all momenta ingoing, i.e. $e^{-ikx}$). This gives you the $g k_2 k_3$ term. When you do this for all the other contractions you get $3$ non equal contributions and $2$ each of the total $6$ are equal (therefore the factor $2$).
When you have different scalar fields it works almost the same, you just only consider contractions between same types of scalar fields.
The QCD case is also the same, however all $6$ are non equal, because they you have an additional space time and colour index. Therefore you get the $6$ terms.