The following are postulates of thermodynamics (Callen, Thermodynamics, 1st ed.)
I. There exist particular states (called equilibrium states) of simple systems that are chracterized by their internal energy $U$, their volume $V$, and the particle numbers $N_1, \dots, N_r$ of their components.
III. The entropy is a monotonically increasing function of the internal energy $U$.
It follows that the internal energy $U$ can be written as a function of $S,V$ and the particle numbers $N_1, \dots, N_r$ and that therefore
\begin{align}
dU = \left(\frac{\partial U}{\partial S}\right)_{V, N_1, \dots N_r}dS + \left(\frac{\partial U}{\partial V}\right)_{S,N_1, \dots, N_r} dV + \sum_{i=1}^r\left(\frac{\partial U}{\partial N_i}\right)_{S, V, N_1, \dots, N_{i-1}, N_{i+1}, \dots, N_r}dN_i.
\end{align}
Now, the question becomes, which of these terms has anything to do with pressure? Well, suppose we accept that the first term can be identified as the heat transferred to a system during a quasistatic process and that we consider an adiabatic process during which the particle numbers $N_1, \dots, N_r$ are held constant, then we obtain
\begin{align}
dU = \left(\frac{\partial U}{\partial V}\right)_{S,N_1, \dots, N_r} dV\, \qquad (\text{quasistatic, adiabatic, $dN_i = 0$}).
\end{align}
On the other hand, if $P$ is the pressure of a given system, and if it undergoes such a process, then it should be pretty clear on physical grounds that if there is no heat transferred to the system, and no change in particle number of any of its species, then the only change in energy that can occur is if the system does some work due to the fact that when its volume is changing, the pressure it possesses causes it to exert a force on its boundaries. Furthermore, it's not hard to show (using some multivariable calculus and the definition of work from mechanics) that the work done due to this pressure for a given change $dV$ in the volume of the system is $PdV$. It follows that along any quasistatic, adiabatic process with unchanging particle numbers, we have
\begin{align}
P = -\left(\frac{\partial U}{\partial V}\right)_{S,N_1, \dots, N_r}.
\end{align}
Now simply note that every point in thermodynamic state space (namely every equilibrium state) lies on some such process, so that relation holds for every equilibrium state.
Your argument is quite correct. It's just that you have misinterpreted what is meant by the statement that the pressure on the liquid is atmospheric pressure.
The atmosphere exerts a downwards force on the air/water interface, and the water exerts an upwards force on water/air interface. Since the interface is stationary we know the two forces must be equal and opposite, and since they act on the same area the two pressures must be equal and opposite.
So the net force on the interface is zero, but it's zero because the pressure the atmosphere exerts on the water is equal to the pressure the water exerts on the atmosphere. Both pressures are real and non-zero. For example if you measured the density of the water accurately enough you'd find it has been slightly compressed by the pressure the atmosphere exerts on it.
Best Answer
The operational macrosopic definition of pressure in a solid is the same as that for a gas : when the solid's wall is in mechanical equilibrium with its surroundings, that means the force exerted on the wall by the surroundings is exactly compensated by the pressure within the solid times the area of the wall.