Caveat: This answer assumes that the question is about interpreting bootstrapped p-values and CIs. A comparison between a traditional p-value (not bootstrapped) and a bootstrapped CI would be a different issue.
With a traditional (not bootstrapped) t-test, the 95%CI and the p-value's position relative to the .05 cutoff for significance will always tell you the same thing. That's because they're both based on the same information: the t-distribution for your degrees of freedom and the mean and standard error observed in your sample (or difference between means and standard error, in the case of a two-sample t-test). If your CI doesn't overlap with 0, then your p-value will necessarily be < .05 --- unless, of course, there's a bug in the software or a user error in implementation or interpretation of the test.
With a bootstrapped t-test, the CI and p value are both calculated directly from the empirical distribution generated by the bootstrapping: the p value is simply what percent of bootstrapped group differences are more extreme than the original observed difference; the 95%CI is the middle 95% of bootstrapped group differences. It is not impossible for the p-value and the CI to disagree about significance in a bootstrapped test.
Do you accept or reject the null hypothesis?
In the context of a bootstrapped test, the p-value (as compared to the CI) more directly reflects the spirit of the hypothesis test, so it makes the most sense to rely on that value to decide whether or not to reject the null at your desired alpha (generally .05). So in your case, where the p value is less than .05 but the 95%CI contains zero, I recommend rejecting the null hypothesis.
All of this skips over the big ideas about how important "significance" really should be and whether or not null hypothesis significance testing is actually that useful of a tool. Briefly, I always recommend complimenting any significance testing analysis with estimation of effect sizes (for a two-sample t-test, the best effect size estimate will probably be Cohen's d), which can provide some additional context to help you understand your results.
Related helpful post: What is the meaning of a confidence interval taken from bootstrapped resamples?
1 They don’t mean what people think they mean
Am I right that this is not a p-value (which is the probability to see this or more extreme value of a test statistic)? Is it a correct procedure for a statistical testing? I have a gut feeling that it is a wrong situation to apply hypothesis testing, but I can not formally answer why.
One could argue that technically speaking it is a p-value. But, it is a rather meaningless p-value. There are two ways to look at it as a meaningless p-value
Neyman and Pearson suggest that, in order to compute the p-value, you choose the region where the likelihood ratio (between the null hypothesis and alternative hypothesis) is the highest. You count observations as 'extreme' when a deviation from the null hypothesis would mean more likelihood to make that extreme observation.
This is not the case with the US citizen example. If the null hypothesis 'Robert is a US citizen' is false, then the observation 'Robert is a US senator' is in no way more likely. So from the viewpoint of Neyman's and Pearson's approach to hypothesis testing, this is a very bad type of calculation for a p-value.
From the viewpoint of Fisher's approach to hypothesis testing, you have a measurement of some effect and the point of the p-value is to quantify the statistical significance. It is useful as an expression of the precision of an experiment.
The p-value quantifies how good/accurate the experiment is in the quantification of the deviation. Statistically speaking effects will always occur to some extent due to random fluctuations in the measurements. An observation is seen as statistically significant when it is a fluctuation of a sufficiently large size such that it has a low probability that we observe a seemingly effect when there is actually no effect (when the null hypothesis is true). Experiments that have a high probability that we observe an effect while there is actually no effect are not very useful. We use p-values to express this probability.
By reporting p-values researchers can show that their experiments have sufficiently small noise and sufficiently large sample size, such that the observed effects are statistically significant (unlikely to be just noise).
Fisher's p-values are an expression of the noise and random fluctuations, they are a sort of expression of signal/noise ratio. The advice is to only reject a hypothesis when an effect is sufficiently large compared to the noise level.
Even though there is no alternative hypothesis in Fisher's viewpoint, when we express a p-value then this is done for the measurement of some effect as a deviation relative to a null (no effect) hypothesis. There must be some sense of a direction that can be considered to be an effect or a deviation.
In the case of the experiment with US citizenship, the measurement of 'Robert is a US senator' has nothing to do with the measurement of some effect or a deviation from the null hypothesis. Expressing a p-value for it is meaningless.
The example with US citizenship may be a bit weird and wrong. However, it is not meant to be correct. The point is to show that simply a p-value is not very meaningful and correct. What we need to consider is also the power of a test (and that is missing in the example with US citizenship). A low p-value might be nice, but what if the p-value would be just as well low, or even lower, for an alternative explanation? If you have a bad hypothesis test then we could 'reject a hypothesis' based on a (crappy) low p-value while actually, no alternative hypothesis is better suitable.
Example 1: Say you have two jars one with 50% gold and 50% silver coins, the other with 75% gold and 25% silver coins. You take 10 coins out of one jar, and they are all silver, which jar do we have? We could say that the prior odds were 1:1 and the posterior odds are 1:1024. We can say that the jar is very likely the one with 50:50 gold:silver, but both hypotheses are unlikely when we observe 10 silver coins and maybe we should mistrust our model.
Example 2: Say you have data that is distributed by a quadratic curve y = a + c x^2. But you fit it with a straight linear line y = a + b x. When we fit a model we find that the p-value is extremely low for a zero slope (no effect) since the data does not match a flat line (as it is following a quadratic curve). But does that mean that we should reject the hypothesis that the coefficient b is zero? The discrepancy, low p-value, is not because the null hypothesis is false, but because out entire model is false (that is the actual conclusion when the p-value is low, the null hypothesis and/or the statistical model is false).
2 They rely on hidden assumptions
It seems to be wrong, but the question is: can we say that non-parametric tests also rely on some regular statistical distributions? Not only they have assumptions, but also, technically, their statistics also follow some distributions
The point of non-parametric tests is that we make no assumptions about the data. But the statistic that we compute may follow some distribution.
Example: We wonder whether one sample is larger than another sample. Let's say that the samples are paired. Then without knowing anything about the distribution we can just count which of the pairs is larger. Independent of the distribution of the population from which the sample has been taken, this sign statistic will follow a binomial distribution.
So the point of non-parametric tests is not that the statistic that is being computed has no distribution, but that the distribution of the statistic is independent from the distribution of the data.
The point of this "They rely on hidden assumptions" is correct. However, it is a bit harsh and sketches the assumptions in a limited sense (as if assumptions are only simplifications to make computations easy).
Indeed many models are simplifications. But I would say that the parametric distributions are still useful, even when we have much more computation power nowadays and simplifications are not necessary. The reason is that parametric distributions are not always simplifications.
On the one hand: Bootstrapping or other simulations can approach the same result as a computation, and when the computation makes assumptions, approximations and simplifications then the bootstrapping may even do better.
On the other hand: The parametric distribution, if it is true, gives you information that bootstrapping can't give you. When you have only little amount of data then you can't get a proper estimate of p-values or confidence intervals. With parametric distributions you can fill the gap.
Example: if you have ten samples from a distribution, then you might estimate the quantile at multiples of 10%, but you won't be able to estimate smaller quantiles. If you know that the distribution can be approximated by some distribution (based on theory and previous knowledge such assumptions might not be bad) then you can use a fit with the parametric distribution to interpolate and extrapolate the ten samples to other quantiles.
Example 2: The representation of parametric tests as being only useful for making calculations easier is a straw man argument. It is not true because it is far from the only reason. The main reason why people use parametric tests is because they are more powerful. Compare for instance the parametric t-test with the non-parametric Mann-Whitney U test. The choice for the former is not because the computation is easier, but because it can be more powerfull.
3 They detract from the real questions
Can we, based on confidence intervals, say, what is an expected value? Is in this situation a clear decision? I always thought that confidence intervals are not necessarily symmetric, but I started to doubt here.
No, confidence intervals do not give full information. You should instead compute some cost function that quantifies all consideration in the decision (requiring the full distribution).
But confidence intervals may be a reasonable indication. The step from a single point estimate to a range is a big difference and adds an entire new dimension to the representation.
Your criticism here is also exactly the point of the author of the blogpost. You criticize the confidence intervals not giving full information. But the means 0.08 for action A and 0.001 for action B have even less information than the confidence intervals, and that is what the author is pointing out.
This third point is more a matter of point estimate versus interval estimates. Maybe p-values promote the use of point estimates, but it is a bit far-fetched to use it as criticism against p-values. The example is not even a case that is about p-values and it is about a Bayesian posterior for two situations.
Best Answer
As @MichaelChernick said in response to a comment on his answer to a linked question:
So this answer will address two associated issues: (1) why might presentations of bootstrap results seem more frequently to specify confidence intervals (CI) rather than p-values, as suggested in the question, and (2) when might both p-values and CI determined by bootstrap be suspected to be unreliable thus requiring an alternate approach.
I don't know data that specifically support the claim in this question on the first issue. Perhaps in practice many bootstrap-derived point estimates are (or at least seem to be) so far from test decision boundaries that there is little interest in the p-value of the corresponding null hypothesis, with primary interest in the point estimate itself and in some reasonable measure of the magnitude of its likely variability.
With respect to the second issue, many practical applications involve "symmetrical distribution of test statistic, pivotal test statistic, CLT applying, no or few nuisance parameters etc" (as in a comment by @XavierBourretSicotte above), for which there is little difficulty. The question then becomes how to detect potential deviations from these conditions and how to deal with them when they arise.
These potential deviations from ideal behavior have been appreciated for decades, with several bootstrap CI approaches developed early on to deal with them. The Studentized bootstrap helps provide a pivotal statistic, and the BCa method deals with both bias and skewness in terms of obtaining more reliable CI from bootstraps. Variance-stabilizing transformation of data before determining bootstrapped CI, followed by back-transformation to the original scale, also can help.
The example in this question on sampling from 14 heads out of 20 tosses from a fair coin is nicely handled by using CI from the BCa method; in R:
The other CI estimates pose the noted problem of being very close to or at the edge of the population value of 10 heads per 20 tosses. The BCa CI account for skewness (as introduced by binomial sampling away from even odds), so they nicely include the population value of 10.
But you have to be looking for such deviations from ideal behavior before you can take advantage of these solutions. As in so much of statistical practice, actually looking at the data rather than just plugging into an algorithm can be key. For example, this question about CI for a biased bootstrap result shows results for the first 3 CI shown in the above code, but excluded the BCa CI. When I tried to reproduce the analysis shown in that question to include BCa CI, I got the result:
where 'w' is involved in the bias correction. The statistic being examined has a fixed maximum value and the plug-in estimate that was bootstrapped was also inherently biased. Getting a result like that should indicate that the usual assumptions underlying bootstrapped CI are being violated.
Analyzing a pivotal quantity avoids such problems; even though an empirical distribution can't have useful strictly pivotal statistics, coming as close as reasonable is an important goal. The last few paragraphs of this answer provide links to further aids, like pivot plots to estimate via bootstrap whether a statistic (potentially after some data transformation) is close to pivotal, and the computationally expensive but potentially decisive double bootstrap.