They did some statistical tests on the data (look at the bottom of the article) which, AFAIK show that the number of boxes sent is big enough to show a significant difference between packages with atheist branding and those without them.

What is the methodological objection to the article claims?

Oh, yes, sorry. Thanks for pointing that out.

I think that a sample size of 85 (89 marked parcels; 89 unmarked parcels; with 4 removed because of late response) is too small to really know much.

And I think I'm a bit worried about the conclusions being drawn in this thread.

But the t-test says otherwise. Why is it that you find the t-test is wrong?

"140 parcels is a tiny number compared to what USPS handles per day."

Why does the size of the sample in relation to the size of the population mean that the numbers need to be better?

Because 140 parcels, not randomly selected, is nowhere near enough to meet the various statistical tests needed to draw meaningful conclusions.

Here's a more important question.

What's the p-value for their proposition, based on the data that's provided?

Does not answer my question. The commenter I was responding to mentioned the size of the population; you refer only to the size of the sample.

If the population size is 140 then a "sample" of 140 gives you a complete description. If the population size is 1.4e10 then a non-random sample of 140 is going to give very low confidence.

For the topics in between you're going to have to look up Wikipedia pages on statistics to see how sample size and population size relate, how that changes with the methodology used to select your sample, etc.

If you know statistics already and you have a point to make then just make it. "20 Questions" might work on Jabber or IRC but it doesn't on a forum like this.

"If the population size is 140 then a 'sample' of 140 gives you a complete description."

That would not be what we think of as a sample, but rather a census.

"If the population size is 1.4e10 then a non-random sample of 140 is going to give very low confidence."

If the sample is non-random then the conclusions are unreliable, regardless of the population size.

I don't understand most of the rest of your comment, except as angry sputtering designed to avoid having to correct yourself. As to my playing 20 questions, since I can not imagine in general how the population size could be relevant to statistical conclusions drawn from a random sample, I was wondering if those who imply the opposite could explain it to me. Because it seems obvious that if you want to measure the salinity of the ocean, you can scoop up a cup of water from it and analyse that. You don't have to use a different size cup for different size oceans, or even know how big your ocean is. But if I'm missing something, I'd love to learn what it is.

> If the sample is non-random then the conclusions are unreliable, regardless of the population size

That's not true. You may still get meaningful data, but it's less meaningful (i.e. the required interval to achieve a given level of confidence becomes wider, perhaps significantly so).

But either way, I mentioned "non-random" to pile onto the ridiculously low sample size. Even a random sample of such a small size would have given a low-confidence result.

> Because it seems obvious that if you want to measure the salinity of the ocean, you can scoop up a cup of water from it and analyse that. You don't have to use a different size cup for different size oceans, or even know how big your ocean is.

You're expressing a "population" parameter that doesn't actually exist. There's no such thing as "salinity of the ocean"; it changes depending on where (and what depth!) you are at. Sampling the salinity of the water in the cup tells you, at best, about the water where you're at.

Now you could probably talk about things like "mean salinity of the oceans", but to determine good bounds for that you would have to sample. And to figure out how much you must sample, you do indeed have to have an idea of the total population size, even if it's just to determine that the population size is so much larger than the sample size that you can ignore the population size and simply use the standard error formula.

If the population size is not much greater than the sample size then there is an adjustment you should make (the finite population correction).

Thanks - I was misremembering or had misunderstood how they had identified themselves (I was thinking they changed the name in the from address, not a much more visible piece of tape.)