Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
statistics:permutations [2010/12/13 6:37 pm PST]
John Colby
statistics:permutations [2010/12/18 9:03 pm PST] (current)
John Colby [Empirical p-value (FWE corrected)]
Line 55: Line 55:
 ===== Empirical p-value (FWE corrected) ===== ===== Empirical p-value (FWE corrected) =====
 Finally, we can apply permutation testing to control the [[wp>​Familywise_error_rate|family-wise type I error rate]] - that is, the chance of //any// false positives across //all// of our 100 multiple comparisons. This is done similar to above, but instead of recording all of the test statistics at the end of each round of permutation,​ only the maximum test statistic across all of the voxels is recorded. This lets us build up an empirical null distribution of the //maximum// test statistic, and comparing our example test statistic to this histogram allows us to generate a p-value that is corrected for multiple comparisons by controlling the family-wise error rate. Finally, we can apply permutation testing to control the [[wp>​Familywise_error_rate|family-wise type I error rate]] - that is, the chance of //any// false positives across //all// of our 100 multiple comparisons. This is done similar to above, but instead of recording all of the test statistics at the end of each round of permutation,​ only the maximum test statistic across all of the voxels is recorded. This lets us build up an empirical null distribution of the //maximum// test statistic, and comparing our example test statistic to this histogram allows us to generate a p-value that is corrected for multiple comparisons by controlling the family-wise error rate.
- 
-Notice how the histogram has shifted over to the right. As we expected, this means that any uncorrected p-value derived from an individual t-statistic will be less significant now that we have corrected for the multiple comparisons made by our many t-tests. For our example test statistic, we get: 
-  * Theoretical:​ p=0.007 
-  * Empirical: p=0.007 
-  * Empirical, corrected: p=0.50 
- 
-Another way of controlling for multiple comparisons is to use the [[wp>​Bonferroni_correction|Bonferroni correction]],​ which simply scales the alpha threshold and p-values according to how many comparisons you are making. The Bonferroni-corrected alpha=0.05 threshold is also shown below with a dashed line. Because the multiple tests in this example are independent,​ this correction aligns with that of our permutation methods. However, in real data where the tests are often correlated (like neuroimaging data), the Bonferroni correction can give overly-conservative results. 
  
 <code rsplus> <code rsplus>
Line 76: Line 69:
  
 {{:​statistics:​permutations:​empirical_corrected.png?​300}} {{:​statistics:​permutations:​empirical_corrected.png?​300}}
 +
 +Notice how the histogram has shifted over to the right. As we expected, this means that any uncorrected p-value derived from an individual t-statistic will be less significant now that we have corrected for the multiple comparisons made by our many t-tests. For our example test statistic, we get:
 +  * Theoretical:​ p=0.007
 +  * Empirical: p=0.007
 +  * Empirical, corrected: p=0.50
 +
 +Another way of controlling for multiple comparisons is to use the [[wp>​Bonferroni_correction|Bonferroni correction]],​ which simply scales the alpha threshold and p-values according to how many comparisons you are making. The Bonferroni-corrected alpha=0.05 threshold is also shown below with a dashed line. Because the multiple tests in this example are independent,​ this correction aligns with that of our permutation methods. However, in real data where the tests are often correlated (like neuroimaging data), the Bonferroni correction can give overly-conservative results.
  
 The R code for this example is available [[:​statistics:​permutations:​perms_example|here]]. The R code for this example is available [[:​statistics:​permutations:​perms_example|here]].
statistics/permutations.txt ยท Last modified: 2010/12/18 9:03 pm PST by John Colby
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki