The most important concept relating Faraday cage hole size to cell phone signal attenuation is the idea of a cutoff frequency. For round holes, you would model them as cylindrical waveguides. For simplicity, we'll consider rectangular waveguides instead.
Matching the boundary conditions at the metal wall, you get so called transverse electric (TE) and transverse magnetic (TM) modes. These look like partially standing waves, with a traveling wave component for the third. For TE modes, they're of the form (for polarization in the y direction):
$$E = E_0 \sin(k_x x) \cos(k_y y) e^{i(k_z z-\omega t)}$$
There's a multitude of standing wave modes. These are described by different values of $k_x$ and $k_y$, which are solved by setting the above expression to zero at the walls of the waveguide (for the sine portion), or derivative zero (for the cosine portion). The solutions:
$$k_x = \frac{m\pi}{w}$$
$$k_y = \frac{n\pi}{h}$$
Where $w$ and $h$ are the width and height of the waveguide, and $m$ and $n$ are integers. Substituting the above expression into the wave equation, we get the relation between the different $k$ components and frequency.
$$\left( \frac{\omega}{c} \right)^2 = {k_x}^2+{k_y}^2+{k_z}^2 $$
The lowest possible such frequency is when $k_z = 0$ and $m=1,n=0$
$$f_c= \frac{c}{2w}$$
This is the cutoff frequency. Below this frequency, the signal exponentially decays as it propagates through the structure. To show this, solve for $k_z$, and write it in terms of cutoff frequency.
$$k_z = \frac{2 \pi}{c} \sqrt{f^2-{f_c}^2}$$
Evidently, at frequencies below cutoff, $k_z$ becomes imaginary. Substituting this into our travelling wave expression, it becomes exponential decay.
$$\alpha := \frac{2 \pi}{c} \sqrt{{f_c}^2-f^2}, f < f_c$$
$$E = E_0 \sin(k_x x) \cos(k_y y) e^{-\alpha z-i\omega t}$$
Notice that for our rectangular waveguide, the cutoff frequency depended only on the width. In general,
$$\lambda_c \approx \textrm{largest feature size}*2 $$
(This is exactly true for a rectangular waveguide, and should hold approximately for other shapes.)
For Faraday cages with openings in the size of centimeters, the cutoff frequency is around 20GHz, which is quite large compared cell phone signals in the range of 2GHz. We can approximate the decay constant $\alpha$
$$\alpha \simeq \frac{2 \pi f_c}{c} = \frac{2\pi}{\lambda _c} $$
To figure out the amount of decay, we need to assume some length $l$ to the opening (equivalently, thickness of the cage material), and then substitute $l$ for $z$ in the wave expression. Converting this to a decibel scale, we get the following power loss:
$$\frac{l}{\textrm{largest feature size}} (48.6 \textrm{dB})$$
Another important point is that the signal will mostly negatively interfere with itself in the cage, except at a few spots within the cage where it is effectively amplified (probably the center). If the cage features are fairly large, you might be able to notice this signal hotspot.
Edit: There are also complex effects where the fields in one hole can induce fields in another hole. The above analysis is a simplified description of a complex field problem, but I expect the general principles to hold.
Best Answer
I'll assume that you were using a radio tuned to a 1Mhz frequency ($\omega = 6.3\times 10^6$ s$^{-1}$) and that the radio was completely enclosed inside $t=3\,$mm of pure iron.
There are two important effects to consider. (i) How much power is reflected from the iron surface. (ii) How much of the transmitted power makes it through the iron.
To figure this out we need the properties of iron; a conductivity $\sigma = 10^7$ S/m, a relative permittivity $\epsilon_r \simeq 1$ and a relative permeability $\mu_r \simeq 10^4$ (for 99.9% pure iron).
First we check whether iron works as a good conductor at these frequencies by noting that $\sigma/{\epsilon_r \epsilon_0 \omega} = 1.8\times 10^{11}$; i.e. $\gg 1$ and therefore a good conductor.
The modulus of the impedance of a conductor is given by $\eta_{\rm Fe} = (\mu_r \mu_0 \omega / \sigma)^{1/2} = 0.089$ $\Omega$.
So, now the relevant equations are: Electric field transmission at the air/iron interface (assuming normal incidence)
$$\frac{E_t}{E_i} = \frac{2 \eta_{\rm Fe}}{\eta_0 + \eta_{\rm Fe}} \simeq 2\frac{\eta_{\rm Fe}}{\eta_0}\, ,$$. where $\eta_0 = 377$ $\Omega$.
The EM waves then propogate into the metal but are exponentially attenuated on a scale defined by the "skin depth" $\delta = (2/\mu_r \mu_0 \sigma \omega)^{1/2} = 1.59 \times 10^{-6}\,$m.
Thus, after traversing a thickness $t$, the E-field is attenuated by $\exp(-t/\delta)$.
Finally the wave emerges through the iron/air interface on the other side and we use the transmission formula again but with the labels swapped on the impedance values.
Hence the ratio of the net transmitted electric field to the incident electric field is given approximately by $$ R = 2 \frac{\eta_{\rm Fe}}{\eta_0}\, \exp(-t/\delta)\, 2 \frac{\eta_0}{\eta_0 + \eta_{\rm Fe}} = 4 \frac{\eta_{\rm Fe}}{\eta_0}\, \exp(-t/\delta)\, .$$
For the numbers I've assumed $R \simeq 0$ because the wave traverse $>1000$ skin depths to get through the iron! The transmitted power is $\propto R^2$.
So my conclusion is that enclosing within 3mm of pure iron would certainly block AM radio.
How might this not work? Perhaps the iron you used is very impure and the permeability is orders of magnitudes lower? If $\mu_r =1$ then $\eta_{\rm Fe} = 0.00089\,\Omega$, $\delta = 1.59\times 10^{-4}\,$m. Thus 3mm is still 18 skin depths. The conductivity I assumed is unlikely to be much lower, so I'm a bit confused as to why it wouldn't work.
The demo I use in my lectures is wrapping a mobile phone in aluminium foil. In principle, this is much more marginal because though the frequencies are higher, the foil thickness is much lower, but it certainly works.