For factoring simple quadratic equations, it's simply a matter of remembering simple forms and tricks. For example, one of the easiest quadratic factorings is the difference of squares
$a^2 -b^2 = (a+b)(a-b)$
Here are several examples of difference of squares factoring:
$x^2 - 1 = (x+1)(x-1) \\ 9x^2 -4 = (3x+2)(3x-2) \\ 36x^2 - 25y^2 = (6x +5y)(6x-5y)$
The next simplest factoring is quadratics without constant terms, which boils down to just spotting common factors. Ex: $9x^2 +3x = 3x(3x+1)$
Next there is factoring quadratics with a leading coefficient of 1, which amounts to pairing factors:
If $x^2 + hx +k = (x+a)(x+b)$, then $k = ab$ and $h = a+b$. This comes from a simple expansion of $(x+a)(x+b) = x^2 + ax + bx + ab = x^2 + (a+b)x + ab$
For example, $x^2 + 20x + 36$. The factors of $36$ are $(1,36) \ (2,18) \ (3,12) \ (4,9) \ (6,6)$ Notice that $20 = 2+18$. Thus, $x^2 + 20x + 26 = (x+2)(x+18)$. This can always be checked by re-expanding.
This even works with negative terms. For example, $x^2 - 4x -12$ The factors of $-12$ are $(-1, 12) \ (-2,6) \ (-3, 4) \ (-4,3) \ (-6, 2) \ (-12, 1)$. Notice that $-6 + 2 = -4$. Thus, $x^2 - 4x -12 = (x-6)(x+2)$
The last case is when the leading coefficient is anything other than $1$. This can be done with a method similar to the one above, but with slightly more computation. For any quadratic polynomial $ax^2 + bx + c$, if it has an expansion in integers $(px + q)(rx +s)$, then $a = pr$, $c = qs$, and $b = (ps + qr)$. Thus, this method of factoring involves finding all the factors of $a$ and $c$, and pairing them in such a way as to equal the middle term $b$. It's easier to demonstrate than to explain.
Let's use your example: $4x^2 - 4x - 3$. The factors of $4$ are $(1,4) \ (2,2)$ and the factors of $-3$ are $(-1,3) \ (1, -3)$. Let's begin pairing exhaustively.
$(1)(-1) + (4)(3) = 11 \\
(1)(3) + (4)(-1) = -1 \\
(1)(1) + (4)(-3) = -11 \\
(1)(-3) + (4)(1) = 1 \\
(2)(-1) + (2)(3) = 4 \\
(2)(1) + (2)(-3) = -4$
Every one of these expressions is equivalent to (a factor of $a$)(a factor of $b$) + (the corresponding factor of $a$)(the corresponding factor of $b$)
Notice that the last expression is the one we need. We need to take the numbers $2, 1, 2, -3$ and plug them into $(\_x +\_)(\_x+\_)$ such that the product is equal to $4x^2 -4x -3$
We know the coefficients of the $x$ terms will be $(2x+\_)(2x+\_)$, because they must multiply to be $4$. From here, we would place the $1, -3$ in the expression such that they match up with the appropriate term to multiply by. However in this case since it's just two $2$'s, the answer is $4x^2 - 4x -3 = (2x+1)(2x-3)$. Check this by expanding back out.
Another example of this method: $3x^2 -7x + 2$. The factors of $3$ are $(1,3)$ and the factors of $2$ are $(1,2)$. Notice that the middle term is negative, though, so we must use $(-1, -2)$ as our factors of $2$. Now we exhaustively pair.
$(1)(-2) + (3)(-1) = -5 \\
(1)(-1) + (3)(-2) = -7$.
The last expression is the one we need. Thus, the factored expansion is $(x+\_)(3x+\_)$. Notice here that for the expansion to be correct, the $3$ must multiply by $-2$. Thus, we plug in into the factor $(x+\_)$, otherwise $-2$ and $3$ would not be multiplied.
Thus, $3x^2 - 7x + 2 = (x-2)(3x-1)$. Check this by expanding back out.
If the method of factor-pairing by exhaustion reveals that there is no possible combination of factors which will sum to the middle term, then the expression has no factorization in integers.
Best Answer
You have $3x^5-6x^3=3x^3(x^2-2)$. Can you take it from here?