If everything is incompressible then pressure does not affect buoyancy.
If the object is less compressible than the fluid then the object will get more buoyant as the pressure increases. Conversely if the object is more compressible than the fluid then the object will get less buoyant as pressure increases.
Temperature and salinity can also vary the density of the water. Water at depth is likely to be a bit more dense.
So if you have an object that is pretty damn incompressible and is only slightly more dense than the surface water then it might find a depth and sit there. In practice few objects are likely to fall into this category.
I will focus on just a little bit of one of your questions - the relationship between compressibility, density and pressure - and per my comment, recommend that you narrow down the scope of your question.
As you know, in a gas we experience "pressure" because molecules hit the walls of the containing vessel. When I double the number of molecules in the same volume at the same temperature, I double the number of collisions (each imparting on average the same momentum) and thus double the pressure - this is the familiar ideal gas law.
Now when the size of the molecules becomes a sizable fraction of the volume, the rate of collisions goes up. Imagine a pingpong ball between two walls. If the distance between the walls is large compared to the size of the ball, the time for a round trip is inversely proportional to the size of the ball; but as the distance approaches the size of the ball, the rate of collisions goes up rapidly.
I think a similar thing happens with "nearly incompressible" liquids: there is a small amount of space between the molecules, but they are permanently bumping into each other and into the walls of the vessel. As you increase the pressure, they bounce more frequently as they have less far to travel before they collide with another molecule (or the wall).
All this is still treating the liquid like a non-ideal gas. In reality, not only do you have the finite size of the molecules, but also attractive forces between them. Both these things make the picture a bit more complex than I sketched. But I would say that the above reasoning nonetheless applies (with caveats).
As for the experiment you described with stoppers on the inside or outside - there are other things going on there as you go from the static (no flow) to the dynamic (flow) situation - the water needs to accelerate before it will flow out at a certain velocity. But I think all that should be the subject of another question.
Best Answer
The answer is no. Water, being a liquid, is nearly incompressible, meaning that the density changes very little with increasing pressure. In the very deep ocean, the pressure can approach $10^{8}$ Pa (about a thousand times greater than standard atmospheric pressure of $1.01\times10^{5}$ Pa). However, the bulk modulus $B$ of liquid water (the reciprocal of the compressibility) is even larger, at about $2\times10^{9}$ Pa. ($B$ actually varies with the pressure and temperature with the water, but not enough to make a practical difference.) The ratio of the ambient pressure $P$ to the bulk modulus $B$ gives you roughly how much fractional change there will be in the density of water when it is under the pressure $P$.
This ratio is about $0.05$, so even at the greatest depths of the ocean, the changes to the density of the water will be, at most, about five percent. Actual differences in ocean density often have more to do with differences in the salinity of the water (since the dissolved sodium and chloride ions adds extra mass); however, these changes are also small, also at the level of a few percent at most. To determine whether a penny will float or sink, we only need to compare the density of the penny to the benthic density of the water. Modern pennies are made mostly of zinc, while before 1982, they were mostly made of copper. The specific gravities of these metals are $7.0$ and $9.0$, respectively (under atmospheric pressure; they would be slightly greater at $10^{8}$ Pa), making them both several times more dense than seawater (specific gravity of $1.0$, plus or minus a few percent). So a penny would sink to the bottom. Only an object that is very close to the density of water at the surface can find an eventual equilibrium point where its density is equal to that of very deep water.