Neural networks can be notoriously difficult to work with (too many potential difficulties, such as local minima, over-fitting etc.). A kernel method is likely to be easier to work with, such as kernel ridge regression, the support vector machine, kernel logistic regression or Gaussian process classifiers, using a radial basis function kernel/covariance function. As long as the hyper-parameters are tuned using a sensible procedure (e.g. cross-validation), they are likely to provide good results with much less effort/risk than neural networks.
(Disclaimer: I did not check the code, and that would be more appropriately asked on codereview.sx)
My problem is that I get unreliable results. I am assuming this is because the dataset is really small. What could I do to get more reasonable estimates?
There are (at least) two small sample size problems mixed here:
because you have too few training cases, your models (including the surrogate models of the cross validation) are not only bad on average, but also unstable (i.e. vary much if a few training instances are exchanged).
because you have too few test cases, your test results themselves are uncertain as well. So the results you observe may be (approximately) described as
learning curve (= average performance as function of $n_{train}$) +
model instability (= variance due to training sample size) +
test variance (= variance due to test sample size) +
testing bias (depending on how exactly you formulate your scenario, i.e. wrt. to $n_{train}$ or $n$)
For simulations, you can at least partially unmix the training and test sample size contributions to the total uncertainty by a reference measurement of the model performance with a separate large independent test set.
If there isn't any further data (i.e. real application, no simulation), make sure that the model itself is stable (low complexity/highly regularized and/or use an aggregated model): the instability contribution to variance is the one thing here that you can influence and you anyways need to do this, because unstable always means high generalization error. Stability can be checked by iterated/repeated cross validation or out-of-bootstrap validation. In addition, the variance contribution due to model instability cancels out over many runs of the cross/out of bootstrap validation.
Once you know that this component of the variance is negligible, you can attribute the variance you observe to the finite test sample size. Note that the $n$ is the number of distinct cases that have been tested, regardless of how many runs you did. With that, you can calculate e.g. hereconfidence intervals.
Obviously this will not reduce the uncertainty as it is fundamentally caused by too few samples - but it will allow you to judge what you can (or cannot) conclude from your data. With that (possibly + simulations), you may be able to convince you supervisor/boss/customer that more samples are needed.
I only have 3 instances in the test set. is that large enough to get reliable estimates?
Typically, no.
For classification with 0/1 loss function, the situation is even worse than for regression, and there we found that for sample sizes < 100 test cases the test error is typically so large that not much can be said at all. I'd expect the situation to be somewhat nicer for regression/calibration, but unless your are superlucky and have basically noiseless data, 3 test cases will not be sufficient to draw any kind of practical conclusions.
Here's a paper we wrote about the situation for classification:
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323
If you need to discuss in more detail, I'd be willing to help e.g. writing a shared paper about this. I'm in chemometrics, and the topic would certainly be of practical importance.
Best Answer
There are two potential issues with the size of your data: achieving statistical significance and finding a practically significant result.
In this particular case, statistical significance means showing that correlation you see between size and population is larger than might be expected by chance alone. If these two variables are strongly correlated, an appropriate test might be able to reject the null hypothesis ($\rho = 0$). If the relationship is weaker, however, you may need a larger sample size to detect it. Regardless, I'd ignore the $n>30$ bit; this is a rough heuristic for when one can assume a sample is normally distributed, but it's neither a particularly good rule in general or applicable to this particular situation. Instead, I'd consider performing a power calculation to determine how many samples you need, based on how big of a correlation you need to be able to detect (presumably a correlation of 0.00001 is essentially uninteresting, but something like 0.25 might be)
As for practical significance...your eventual goal is to use these data to convince someone of something. This might be difficult using data from six cities measured at six time points. If I were interested in demographics generally, I might be worried that there is something unrepresentative about the data from these six cities and that the correlation coefficient from your sample might not provide much information about other situations. Similar concerns would exist about having only six time points. On the other hand, if there is something special about these six towns--perhaps you're advising the mayor of a 7th nearby town--then this might be all the data you need, assuming it's enough demonstrate the effect.