This shows you the differences between two versions of the page.

Both sides previous revision Previous revision | |||

statistics:permutations [2010/12/13 6:37 pm PST] John Colby |
statistics:permutations [2010/12/18 9:03 pm PST] (current) John Colby [Empirical p-value (FWE corrected)] |
||
---|---|---|---|

Line 55: | Line 55: | ||

===== Empirical p-value (FWE corrected) ===== | ===== Empirical p-value (FWE corrected) ===== | ||

Finally, we can apply permutation testing to control the [[wp>Familywise_error_rate|family-wise type I error rate]] - that is, the chance of //any// false positives across //all// of our 100 multiple comparisons. This is done similar to above, but instead of recording all of the test statistics at the end of each round of permutation, only the maximum test statistic across all of the voxels is recorded. This lets us build up an empirical null distribution of the //maximum// test statistic, and comparing our example test statistic to this histogram allows us to generate a p-value that is corrected for multiple comparisons by controlling the family-wise error rate. | Finally, we can apply permutation testing to control the [[wp>Familywise_error_rate|family-wise type I error rate]] - that is, the chance of //any// false positives across //all// of our 100 multiple comparisons. This is done similar to above, but instead of recording all of the test statistics at the end of each round of permutation, only the maximum test statistic across all of the voxels is recorded. This lets us build up an empirical null distribution of the //maximum// test statistic, and comparing our example test statistic to this histogram allows us to generate a p-value that is corrected for multiple comparisons by controlling the family-wise error rate. | ||

- | |||

- | Notice how the histogram has shifted over to the right. As we expected, this means that any uncorrected p-value derived from an individual t-statistic will be less significant now that we have corrected for the multiple comparisons made by our many t-tests. For our example test statistic, we get: | ||

- | * Theoretical: p=0.007 | ||

- | * Empirical: p=0.007 | ||

- | * Empirical, corrected: p=0.50 | ||

- | |||

- | Another way of controlling for multiple comparisons is to use the [[wp>Bonferroni_correction|Bonferroni correction]], which simply scales the alpha threshold and p-values according to how many comparisons you are making. The Bonferroni-corrected alpha=0.05 threshold is also shown below with a dashed line. Because the multiple tests in this example are independent, this correction aligns with that of our permutation methods. However, in real data where the tests are often correlated (like neuroimaging data), the Bonferroni correction can give overly-conservative results. | ||

<code rsplus> | <code rsplus> | ||

Line 76: | Line 69: | ||

{{:statistics:permutations:empirical_corrected.png?300}} | {{:statistics:permutations:empirical_corrected.png?300}} | ||

+ | |||

+ | Notice how the histogram has shifted over to the right. As we expected, this means that any uncorrected p-value derived from an individual t-statistic will be less significant now that we have corrected for the multiple comparisons made by our many t-tests. For our example test statistic, we get: | ||

+ | * Theoretical: p=0.007 | ||

+ | * Empirical: p=0.007 | ||

+ | * Empirical, corrected: p=0.50 | ||

+ | |||

+ | Another way of controlling for multiple comparisons is to use the [[wp>Bonferroni_correction|Bonferroni correction]], which simply scales the alpha threshold and p-values according to how many comparisons you are making. The Bonferroni-corrected alpha=0.05 threshold is also shown below with a dashed line. Because the multiple tests in this example are independent, this correction aligns with that of our permutation methods. However, in real data where the tests are often correlated (like neuroimaging data), the Bonferroni correction can give overly-conservative results. | ||

The R code for this example is available [[:statistics:permutations:perms_example|here]]. | The R code for this example is available [[:statistics:permutations:perms_example|here]]. |

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International