For factoring simple quadratic equations, it's simply a matter of remembering simple forms and tricks. For example, one of the easiest quadratic factorings is the difference of squares
$a^2 -b^2 = (a+b)(a-b)$
Here are several examples of difference of squares factoring:
$x^2 - 1 = (x+1)(x-1) \\ 9x^2 -4 = (3x+2)(3x-2) \\ 36x^2 - 25y^2 = (6x +5y)(6x-5y)$
The next simplest factoring is quadratics without constant terms, which boils down to just spotting common factors. Ex: $9x^2 +3x = 3x(3x+1)$
Next there is factoring quadratics with a leading coefficient of 1, which amounts to pairing factors:
If $x^2 + hx +k = (x+a)(x+b)$, then $k = ab$ and $h = a+b$. This comes from a simple expansion of $(x+a)(x+b) = x^2 + ax + bx + ab = x^2 + (a+b)x + ab$
For example, $x^2 + 20x + 36$. The factors of $36$ are $(1,36) \ (2,18) \ (3,12) \ (4,9) \ (6,6)$ Notice that $20 = 2+18$. Thus, $x^2 + 20x + 26 = (x+2)(x+18)$. This can always be checked by re-expanding.
This even works with negative terms. For example, $x^2 - 4x -12$ The factors of $-12$ are $(-1, 12) \ (-2,6) \ (-3, 4) \ (-4,3) \ (-6, 2) \ (-12, 1)$. Notice that $-6 + 2 = -4$. Thus, $x^2 - 4x -12 = (x-6)(x+2)$
The last case is when the leading coefficient is anything other than $1$. This can be done with a method similar to the one above, but with slightly more computation. For any quadratic polynomial $ax^2 + bx + c$, if it has an expansion in integers $(px + q)(rx +s)$, then $a = pr$, $c = qs$, and $b = (ps + qr)$. Thus, this method of factoring involves finding all the factors of $a$ and $c$, and pairing them in such a way as to equal the middle term $b$. It's easier to demonstrate than to explain.
Let's use your example: $4x^2 - 4x - 3$. The factors of $4$ are $(1,4) \ (2,2)$ and the factors of $-3$ are $(-1,3) \ (1, -3)$. Let's begin pairing exhaustively.
$(1)(-1) + (4)(3) = 11 \\
(1)(3) + (4)(-1) = -1 \\
(1)(1) + (4)(-3) = -11 \\
(1)(-3) + (4)(1) = 1 \\
(2)(-1) + (2)(3) = 4 \\
(2)(1) + (2)(-3) = -4$
Every one of these expressions is equivalent to (a factor of $a$)(a factor of $b$) + (the corresponding factor of $a$)(the corresponding factor of $b$)
Notice that the last expression is the one we need. We need to take the numbers $2, 1, 2, -3$ and plug them into $(\_x +\_)(\_x+\_)$ such that the product is equal to $4x^2 -4x -3$
We know the coefficients of the $x$ terms will be $(2x+\_)(2x+\_)$, because they must multiply to be $4$. From here, we would place the $1, -3$ in the expression such that they match up with the appropriate term to multiply by. However in this case since it's just two $2$'s, the answer is $4x^2 - 4x -3 = (2x+1)(2x-3)$. Check this by expanding back out.
Another example of this method: $3x^2 -7x + 2$. The factors of $3$ are $(1,3)$ and the factors of $2$ are $(1,2)$. Notice that the middle term is negative, though, so we must use $(-1, -2)$ as our factors of $2$. Now we exhaustively pair.
$(1)(-2) + (3)(-1) = -5 \\
(1)(-1) + (3)(-2) = -7$.
The last expression is the one we need. Thus, the factored expansion is $(x+\_)(3x+\_)$. Notice here that for the expansion to be correct, the $3$ must multiply by $-2$. Thus, we plug in into the factor $(x+\_)$, otherwise $-2$ and $3$ would not be multiplied.
Thus, $3x^2 - 7x + 2 = (x-2)(3x-1)$. Check this by expanding back out.
If the method of factor-pairing by exhaustion reveals that there is no possible combination of factors which will sum to the middle term, then the expression has no factorization in integers.
How wide is the shaded region? If the length of the larger rectangle is $a^3$, and the length of the right is $b^3$...
The width of the shaded region is $a^3-b^3$
How tall is the shaded region?
I assume the picture is indicating that the height is $a^2+ab+b^2$
What is the area of a rectangle described as in terms of width and height?
width times height
And what is that in this case?
$(a^3-b^3)(a^2+ab+b^2)$
What is the perimeter of a rectangle described as in terms of width and height?
twice the sum of width and height
And what is that in this case?
$2(a^3-b^3 + a^2+ab+b^2)$
Now... can we express any of these above in a more compact way via factoring? That is debatable, but I do recognize at least one simplification you can do for area:
Knowing that $a^3-b^3=(a-b)(a^2+ab+b^2)$ we have that the area can be described as $(a-b)(a^2+ab+b^2)^2$. Is that really more preferred than $(a^3-b^3)(a^2+ab+b^2)$? I don't see a big difference between the two, so personally I wouldn't care which. They are the same to me.
Using the same for perimeter, we can do the following:
$2(a^3-b^3 + a^2+ab+b^2) = 2((a-b)(a^2+ab+b^2)+1(a^2+ab+b^2)) = 2(a-b+1)(a^2+ab+b^2)$. Again, is this more useful than the first way of writing it? That is debatable...They are not fundamentally different.
Best Answer
If all the terms have even powers, you have a polynomial of half the degree in the square of the variable. $5x^4+24x^2-72=6((x^2)^2+4(x^2)-12)$, which is a quadratic that you should know how to factor. It might be easier to see if you define $y=x^2$ and write the polynomial as $6(y^2+4y-12)$. Once you have factored that, plug the $x^2$s back in and you will have two quadratic factors that you can factor.
Generally and here it is not useful to factor part of an expression. In your first example, you can write it as $(x^2+3x)(x^2-3x)+14$ (you have extra factors of $2$) but you are right that you don't know what to do with the $14$. You could complete the square to get $x^4-9x^2+14=(x^2-\frac 92)^2-\frac {25}4$ and then use the difference of squares.