Let's say the base is an $x \times x$ square, and call height $y$.
Since you know volume is length $\times$ width $\times$ height, we have that
$yx^2 = V\tag{1}$
You also know that you have $400$ square inches to work with.
The surface area of the box is $x^2$ (the area of the bottom of the box - no top since its to be an open box), plus $4\times x\times y$, the area of the 4 sides of the box.
So, $x^2 + 4xy = 400\tag{2}$
Now, we can use equatins $(1), (2)$ to put together a formula for maximizing Volume:
From $(2)$: $4xy = 400 - x^2 \iff y = \dfrac{400 - x^2}{4x}\tag{3}$
Substituting for $y$ from equation $(3)$ into equation $(1)$ gives us:
$$V = x^2y = x^2\left(\dfrac{400 - x^2}{4x}\right)$$
$$ \iff V = \frac{x(400-x^2)}{4} = \frac 14 x(400 - x^2) = 100x - \frac 14 x^3\tag{4}$$
Now, to maximize V, in terms of x, we differentiate $V$, set $V' = 0$, and determine critical points to test for maximums.
In your derivative, you'll find you get a difference of squares, which factors fairly nicely, revealing the possible roots for $x$: one positive, one negative. So your maximum is going to have to occur when $x = $ positive root.
Once you've found $x_{\text{max}}$, use this to solve for $y$ (height), using equation $(3)$, and once you have $y$, you're ready to compute the maximum possible volume of the box (given the constraint of the total available cardboard) using equation $(1)$ and substituting into that equation the values you obtain for $x_{\text{max}}$ and $y$ to solve for V = volume, in cubic inches.
Try to express $w = \dfrac{24}{l}$ in terms of $l = \sqrt{18} = 3\sqrt 2$, to get the precise value of $w$.
So $w = \dfrac{24}{l} = \dfrac{24}{3\sqrt 2} = \dfrac 8{\sqrt 2}$
And note that indeed, $A = w\cdot l = \dfrac{24}{3\sqrt 2}\cdot 3{\sqrt 2} = 24$
Using a calculator and rounding gets you a bit in the way of a loss of precision.
Lastly, you're calculation of the only possible solution to $A'(l) = 0$ is $l = \sqrt {18} = 3\sqrt 2$, since the alternative is negative, and since $l$ represents length, $l > 0$.
But even though the only place an extrema can occur in this case is at your solution $l$, and we can guess that since the problem is asking for maximization, that your solution must be a maximum, you really do need to show in your work that $A(\sqrt {18})$ is indeed the maximum value: Show that $A' > 0 $ on $0 < l < \sqrt{18}$, and $A' < 0$ for $l > \sqrt{18}$.
Best Answer
You have a cost function. Letting dimension of the box be $x \times y \times z$ such that the top and bottom have area $xy$, we have cost in cents given by $$C(x,y,z) = 2 (2xz + 2yz) + 3 (2xy)$$
You also have a constraint, the volume:
$$V(x,y,z) = xyz = 520$$
This problem hints at using the technique of Lagrange multipliers, which often arises in multivariable calculus course. To progress, you'll need to solve
$$\nabla C(x,y,z) = \lambda \nabla V(x,y,z) $$
under the constraint. You should find certain $\lambda$ that will lead to a set of $\{x,y,z\}$ to subtitute back into $C$, producing possible extremas.
As a solution to the system of equations created by the last step, I get
$$x = 2(\frac{130}{3})^{1/3}$$ $$y=x$$ $$z=\frac{3}{2}x$$
which when plugged into $C$ gives
$$C_{min} = 888.275 \mathtt{ cents}$$
This matches the answer given by Wolfram Alpha.