So I was having a debate with a researcher about the definition of statistically significant which was given as:
“Highly unlikely to have occurred by chance”
My issue with it was the use of the word chance. I was arguing that there are many reasons why you would get significant results that don’t have to do with “chance”. Chance to me is a fuzzy word.
They brought up a sample size example using male vs female. They also pointed out that I need to think of what chance means in statistics vs laymen’s terms which I agree, but I have also seen non-laymen conflate chance with randomness. I did some digging and found this paper:
Probability and Chance in the Theory of Statistics
It focuses on how chance differs from probability which I think starts to gets towards the better definition:
p(data | null hypothesis or whatever assumption(s)/model)
Once given a particular premiss, it may be possible to give a probability a value with which everyone readily agrees; this value is, moreover, independent of further data or premisses unless these contradict the initial prermiss, that is, all other knowledge is irrelevant. The probability may then be said to be a chance. This invariant character of a chance depends always on the recognition that the particular premiss essential to it is inserted in the data.
I also found this discussing chance vs randomness:
Chance versus Randomness
Am I just being nit picky? To what degree is the original definition “right”? They were adamant that it was right and nothing really controversial about it.
Does anyone know of good papers/books which discuss the statistical definition or way “chance” is defined?
Any other considerations?