Let's say the base is an $x \times x$ square, and call height $y$.
Since you know volume is length $\times$ width $\times$ height, we have that
$yx^2 = V\tag{1}$
You also know that you have $400$ square inches to work with.
The surface area of the box is $x^2$ (the area of the bottom of the box - no top since its to be an open box), plus $4\times x\times y$, the area of the 4 sides of the box.
So, $x^2 + 4xy = 400\tag{2}$
Now, we can use equatins $(1), (2)$ to put together a formula for maximizing Volume:
From $(2)$: $4xy = 400 - x^2 \iff y = \dfrac{400 - x^2}{4x}\tag{3}$
Substituting for $y$ from equation $(3)$ into equation $(1)$ gives us:
$$V = x^2y = x^2\left(\dfrac{400 - x^2}{4x}\right)$$
$$ \iff V = \frac{x(400-x^2)}{4} = \frac 14 x(400 - x^2) = 100x - \frac 14 x^3\tag{4}$$
Now, to maximize V, in terms of x, we differentiate $V$, set $V' = 0$, and determine critical points to test for maximums.
In your derivative, you'll find you get a difference of squares, which factors fairly nicely, revealing the possible roots for $x$: one positive, one negative. So your maximum is going to have to occur when $x = $ positive root.
Once you've found $x_{\text{max}}$, use this to solve for $y$ (height), using equation $(3)$, and once you have $y$, you're ready to compute the maximum possible volume of the box (given the constraint of the total available cardboard) using equation $(1)$ and substituting into that equation the values you obtain for $x_{\text{max}}$ and $y$ to solve for V = volume, in cubic inches.
The volume is $(A-2)(B-2)$. We want to maximize this given $AB=186$.
Now $(A-2)(B-2)=AB-2(A+B)+4=190-2(A+B)$. We want to minimize $A+B$, which is $A+\frac{186}{B}$. The minimum exists, at a place in "the middle." So the derivative there is $0$. Now it's over: we get the expected $A=B=\sqrt{186}$.
Best Answer
As the question fixes $h$ and seeks you to find $l$ and $w$ in terms of $h$, you should differentiate wrt. $w$ and not $h$.
$V = (62-w-h) w h$
$\frac{dV}{dw} = (62-w-h) h - wh = 0 \implies 62 - 2w - h = 0$
That leads to $w = 31 - \frac{h}{2}$ and $V = h(31 - \frac{h}{2})^2$.
Now to see for which $h$ the volume is max, differentiate wrt. $h$ and equate to zero.
You should get a cube with $l = w = h$. You can confirm it by AM-GM.
$V = (62-w-h) wh$
So by AM-GM,
$\frac{(62-w-h) + w + h}{3} \geq [(62-w-h) w h]^{1/3} = V^{1/3}$
$\implies V \leq (\frac{62}{3})^3$.
Equality occurs when $62-w-h = w = h$.