The main practical drawback of permutation testing is that, except for very small sample sizes, the number of all possible permutations is usually impractically large. Although now fast and relatively cheap computing facilities are at our disposal, this problem is still interesting, in particular for applied statisticians. The main idea is that it is not necessary to compute all possible permutations to obtain a reliable p-value estimate of the test. To deal with this problem, one may approximate the exact p-value of the test by using a random sample from all permutations. The aim of this paper is to reply to this question: how many permutations should be considered in the p-value estimation procedure? We suggest to use 500-1000 permutations to estimate the size and power of a permutation test, via Monte Carlo simulations, at the alfa significance level of 5% and 2000-5000 when alfa = 1%. Moreover, we suggest to use 5000 permutations in actual applications when alfa = 5% and 10000 when alfa = 1%. These suggestions are based on a review of many papers, a simulation study and two applications to actual data sets.
Some Remarks about the Number of Permutations One Should Consider to Perform a Permutation Test
MAROZZI, Marco
2004
Abstract
The main practical drawback of permutation testing is that, except for very small sample sizes, the number of all possible permutations is usually impractically large. Although now fast and relatively cheap computing facilities are at our disposal, this problem is still interesting, in particular for applied statisticians. The main idea is that it is not necessary to compute all possible permutations to obtain a reliable p-value estimate of the test. To deal with this problem, one may approximate the exact p-value of the test by using a random sample from all permutations. The aim of this paper is to reply to this question: how many permutations should be considered in the p-value estimation procedure? We suggest to use 500-1000 permutations to estimate the size and power of a permutation test, via Monte Carlo simulations, at the alfa significance level of 5% and 2000-5000 when alfa = 1%. Moreover, we suggest to use 5000 permutations in actual applications when alfa = 5% and 10000 when alfa = 1%. These suggestions are based on a review of many papers, a simulation study and two applications to actual data sets.I documenti in SFERA sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
