Double or Nothings (DON) and Statistical Power

[Nakamura]

With the rain beating down and feeling a bit sick, I was a bit bored today when I found my stats book. I've had something on my mind for a while now; I was wondering how many double or nothing tournaments do you need to play to be certain you are running at a particular level? If your mate is running at 7.5% over 2000 tournaments and you are running at 0% there is, statistically speaking, a good chance there is no difference in your ability. To understand why I have to give you a course in STATS101.

In statistics we create something called a null hypothesis (Ho), which assumes there is no difference between the data sets. In this case the null hypothesis would be that there is no difference between the mean percentage of tournaments your mate wins compared to you. The alternative hypothesis supposes that there are differences in the statistical behaviour of these two datasets.

The job of the statistician is to attempt to reject the null hypothesis, thereby accepting the alternative hypothesis. How do we do this? We use statistical tests to achieve this, but if our sample size is too small compared to the magnitude of the difference, we may not be able to detect the true difference.

Coming back to the hypotheses, since data is sampled at random, there is always a risk of reaching an incorrect conclusion. We can do this in two ways,

Type I error - The hypothesis is correct, but the test rejects it.
Type II error - The hypothesis is incorrect, but the test accepts it.

The probability of rejecting a false hypothesis is called the 'power' of a test and, you guessed it, there is a method for calculating this imaginatively called the power test.

Now, we are interested in the differences in ROI but we should look at the ITM %, since this the driving factor behind ROI. Conveniently, for DONs the result of any particular game is binary; either you lost the game or you finished ITM. So we can consider the ITM % as a proportion.

I think it suffices to say that there is a particular kind of power test for comparing proportions, which is based on the approximations with the normal distribution (the standard bell-shaped distribution that most random data conforms to). I have run this particular code in R, a free stats program that anyone can learn to use!

Here is the code:

> power.prop.test(power=0.85,p1=0.52,p2=0.53)

     Two-sample comparison of proportions power calculation 

              n = 44778.2
             p1 = 0.52
             p2 = 0.53
      sig.level = 0.05
          power = 0.85
    alternative = two.sided

 NOTE: n is number in *each* group

Eezy, peezy, Lionel squeezy, huh?

I have set the desired power to 85% (although it's up to the designer of the experiment). We are using a 95% confidence interval, which is where the significance level of 0.05 comes in (don't worry too much about this - it's pretty industry standard). I have set the first ITM percentage at 52%, which is exactly 0% ROI and the second ITM percentage at 53% or 1.9% ROI.

The power test indicates that you would need have 44778 under your belt to be sure that an ROI of 0% is statistically different to an ROI of 1.9% (at the 95% confidence interval). Now we can run with those ITM percentage and ask how many tournaments would you need to have under your belt to be sure that an ROI of X is different to an ROI of 0%. Here is the result.

ITM %      ROI %	      Tournaments
52	        0.0	           NA
53	        1.9	        44778
54	        3.8	        11181
55	        5.8	         4962
56	        7.7	         2786
57	        9.6	         1780
58	       11.5	         1233
59	       13.5	         904
60	       15.4	         690

Looking at it graphically, it looks like this.


Figure 1: The number of tournaments a player needs to play to be 95% certain that his ROI of 0% is statistically different to X ROI.

Obviously, the further apart the ROI figures are the fewer tournament you need to be certain that the ROI figures are in fact different. Interestingly this translates into a near perfect negative exponential line.

These results can perhaps be interpreted in a more practical manner with a player with a 5.8% ROI asking, "Can I be certain that my true ROI isn't 9.6%?" Since the difference of ITM finishes is 2% between those two ROI figures, we can simply read off the chart the numbers of tournaments you would need to play between an ITM% of 52 and and 54. In this case, you would need to play 11181 to be 95% certain that an ROI of 5.8% is statistically different to an ROI of 9.6%.

Well I hope this isn't too dense for folks to understand. I just did it as an exercise for myself and I hope some people find it interesting.

Edits: Just some housework with spelling.

[rong]

Intereseting stuff Nak. I often have ideas like this while at work and plan to do some math research for a post when I get home, but I never get round to doing it for one reason or another. A+ for effort.

[Nakamura]

Thanks Dan.

Another thought came to mind, which is this stuff obviously also applies to HUSNGS, since the outcome of a game is also binary.

[taipan168]

Thanks Rob, interesting analysis. When I have a little more time and headspace I'll run through your analysis in detail, I'm sure there are insights to be gained there!

[rong]

Thanks Dan.

Another thought came to mind, which is this stuff obviously also applies to HUSNGS, since the outcome of a game is also binary.

And 3-way (tripple-ups), which I think I mentioned earlier are seriously soft at low levels.

[Arjonius]

What's a decent ROI in DoNs anyway?

[rong]

One good thing that can be taken from this, is that if you are only just losing money or mayber breaking even, or even if you have lost considerably but don't have a huge data set, please continue playing.

It's probably not that you are crap at poker, it is likely you have just hit some variance and you are in fact a winning player. So continue playing for another, say 5000 SnG's before you either change your play or quit poker for good.


(Is that considered doing my bit for the poker community? I need donks in SnGs to make up for my small yet potentially large losses in ring games)

[kevster]

wow. just wow.

This post pleases me greatly. Very useful.

[Nakamura]

Thanks Kev.

I'm been bored while watching us clobber the hell out the Zimbabwe in the cricket, so I wondered what if we change the significance level. In other words, you lower the statistical significance level, which increases the chance of a Type II error (accepting the alternative hypothesis when it is actually incorrect). In our case, that would mean you accept that there is difference between two ROI's when, in fact, they are not different.

I played with significance levels of 80% and 70%, which, needless to say, are unusual in the stats realm. Here are the results displayed in a graphical layout (since I can't be bothered to align everything)

.
Figure 2. ROI versus number of tournaments at three different sigificance levels, 95%, 80%, 75.

It just so happens that if you want to be 80% certain that two ROIs are in fact different, you now only need a sample size 59.8% compared to that at the 95% sig. level. For example, when answering whether an ROI of 3.8% is different to 0%, you now only 6691 tournaments compared to 11181. However, the flip-side is that there is a greater chance you are concluding that two ROI's are different, when in fact they are not.

If you lower your standards even further, you can get away with less then half the number of tournaments you required at the 95% sig level, but now you are accepting a 30% risk of error.

I doubt this is the end of my thoughts, as I was thinking about the evolution of certainty between samples. For example, when am I am 10%, 20% certain etc., etc. But this is it for now.