Let's say the base is an $x \times x$ square, and call height $y$.
Since you know volume is length $\times$ width $\times$ height, we have that
$yx^2 = V\tag{1}$
You also know that you have $400$ square inches to work with.
The surface area of the box is $x^2$ (the area of the bottom of the box - no top since its to be an open box), plus $4\times x\times y$, the area of the 4 sides of the box.
So, $x^2 + 4xy = 400\tag{2}$
Now, we can use equatins $(1), (2)$ to put together a formula for maximizing Volume:
From $(2)$: $4xy = 400 - x^2 \iff y = \dfrac{400 - x^2}{4x}\tag{3}$
Substituting for $y$ from equation $(3)$ into equation $(1)$ gives us:
$$V = x^2y = x^2\left(\dfrac{400 - x^2}{4x}\right)$$
$$ \iff V = \frac{x(400-x^2)}{4} = \frac 14 x(400 - x^2) = 100x - \frac 14 x^3\tag{4}$$
Now, to maximize V, in terms of x, we differentiate $V$, set $V' = 0$, and determine critical points to test for maximums.
In your derivative, you'll find you get a difference of squares, which factors fairly nicely, revealing the possible roots for $x$: one positive, one negative. So your maximum is going to have to occur when $x = $ positive root.
Once you've found $x_{\text{max}}$, use this to solve for $y$ (height), using equation $(3)$, and once you have $y$, you're ready to compute the maximum possible volume of the box (given the constraint of the total available cardboard) using equation $(1)$ and substituting into that equation the values you obtain for $x_{\text{max}}$ and $y$ to solve for V = volume, in cubic inches.
If we cut equal-sized square notches of side length $x$ from the corners of the sheet, then the volume of a rectangular prism from the sheet would be
$$V(x) = x (a-2 x) (b-2 x) = 4 x^3 - 2 (a+b) x^2 + a b x$$
where $a$ and $b$ are the lengths of the sides of the sheet. We maximize by solving $V'(x)=0$:
$$V'(x) = 12 x^2-4 (a+b) x+a b=0 \implies x_{\pm} = \frac{a+b}{6} \pm \frac16 \sqrt{a^2-a b+b^2}$$
For a max, $V''(x) \lt 0$, so that the volume is a maximum when $x=x_-$. In your case, $a=64$ cm and $b=51$ cm, so that each square notch is about $9.40$ cm on a side.
It is possible to maximize further by cutting rectangular notches, but I hope this illustrates how to go about it.
Best Answer
The volume is $(A-2)(B-2)$. We want to maximize this given $AB=186$.
Now $(A-2)(B-2)=AB-2(A+B)+4=190-2(A+B)$. We want to minimize $A+B$, which is $A+\frac{186}{B}$. The minimum exists, at a place in "the middle." So the derivative there is $0$. Now it's over: we get the expected $A=B=\sqrt{186}$.