Examples
Example 48.1. Cochran-Armitage Test with Permutation Resampling
This example, from Keith Soper at Merck, illustrates the exact permutation Cochran-Armitage test carried out on permutation resamples. In the following data set, each observation represents an animal. The binary variables S1 and S2 indicate two tumor types, with 0s indicating no tumor (failure) and 1 indicating a tumor (success); note that they have perfect negative association. The grouping variable is Dose .
data a; input S1 S2 Dose @@; datalines; 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 1 0 1 1 0 2 1 0 2 0 1 2 1 0 2 0 1 2 1 0 2 1 0 3 1 0 3 1 0 3 0 1 3 0 1 3 1 0 3 ; proc multtest data=a permutation nsample=10000 seed=36607 outperm=pmt pvals; test ca(S1 S2 / permutation=10 uppertailed); class Dose; contrast 'CA Linear Trend' 0 1 2; run; proc print data=pmt; run;
The PROC MULTTEST statement requests 10,000 permutation resamples. The OUTPERM=PMT option creates an output SAS data set for the exact permutation distribution computed for the CA test.
The TEST statement specifies an upper-tailed Cochran-Armitage linear trend test for S1 and S2 . The cutoff for exact permutation calculations is 10, as specified with the PERMUTATION= option in the TEST statement. Since S1 and S2 have ten and eight successes, respectively, PROC MULTTEST uses exact permutation distributions to compute the p -values for both variables.
The groups for the CA test are the levels of Dose from the CLASS statement. The trend coefficients applied to these groups are 0, 1, and 2, respectively, as specified in the CONTRAST statement.
Finally, PROC PRINT displays the SAS data set containing the permutation distributions.
The results from this analysis are listed in Output 48.1.1 through Output 48.1.5.
 |
The Multtest Procedure Model Information Test for discrete variables Cochran-Armitage Exact permutation distribution used Everywhere Tails for discrete tests Upper-tailed Strata weights None P-value adjustment Permutation Number of resamples 10000 Seed 36607
| |
| |
The Multtest Procedure Contrast Coefficients Dose Contrast 1 2 3 CA Linear Trend 0 1 2
| |
| |
The Multtest Procedure Discrete Variable Tabulations Variable Dose Count NumObs Percent S1 1 2 6 33.33 S1 2 4 6 66.67 S1 3 4 6 66.67 S2 1 4 6 66.67 S2 2 2 6 33.33 S2 3 2 6 33.33
| |
| |
The Multtest Procedure p-Values Variable Contrast Raw Permutation S1 CA Linear Trend 0.1993 0.4058 S2 CA Linear Trend 0.9220 1.0000
| |
| |
Obs _contrast_ _var_ _value_ upper_p 1 CA Linear Trend S1 0 1.00000 2 CA Linear Trend S1 1 1.00000 3 CA Linear Trend S1 2 1.00000 4 CA Linear Trend S1 3 1.00000 5 CA Linear Trend S1 4 1.00000 6 CA Linear Trend S1 5 0.99966 7 CA Linear Trend S1 6 0.99609 8 CA Linear Trend S1 7 0.97827 9 CA Linear Trend S1 8 0.92205 10 CA Linear Trend S1 9 0.80070 11 CA Linear Trend S1 10 0.61011 12 CA Linear Trend S1 11 0.38989 13 CA Linear Trend S1 12 0.19930 14 CA Linear Trend S1 13 0.07795 15 CA Linear Trend S1 14 0.02173 16 CA Linear Trend S1 15 0.00391 17 CA Linear Trend S1 16 0.00034 18 CA Linear Trend S1 17 0.00000 19 CA Linear Trend S1 18 0.00000 20 CA Linear Trend S1 19 0.00000 21 CA Linear Trend S1 20 0.00000 22 CA Linear Trend S2 0 1.00000 23 CA Linear Trend S2 1 1.00000 24 CA Linear Trend S2 2 1.00000 25 CA Linear Trend S2 3 0.99966 26 CA Linear Trend S2 4 0.99609 27 CA Linear Trend S2 5 0.97827 28 CA Linear Trend S2 6 0.92205 29 CA Linear Trend S2 7 0.80070 30 CA Linear Trend S2 8 0.61011 31 CA Linear Trend S2 9 0.38989 32 CA Linear Trend S2 10 0.19930 33 CA Linear Trend S2 11 0.07795 34 CA Linear Trend S2 12 0.02173 35 CA Linear Trend S2 13 0.00391 36 CA Linear Trend S2 14 0.00034 37 CA Linear Trend S2 15 0.00000 38 CA Linear Trend S2 16 0.00000
| |
You should check the preceding table to verify that the analysis specifications are correct.
The preceding table lists the label and coefficients from the CONTRAST statement.
The preceding table contains summary statistics for the two test variables, S1 and S2 . The Count column lists the number of successes for each level of the class variable, Dose . The NumObs column is the sample size , and the Percent column is the percentage of successes in the sample.
The Raw column in the preceding p -Values table contains the p -values from the CA test, and the Permutation column contains the permutation-adjusted p -values.
This table shows that, for S1 , the adjusted p -value is almost twice the raw p -value. In fact, from theoretical considerations, the permutation-adjusted p -value for S1 should be 2 — 0.1993 = 0.3986; the difference is due to resampling error. For S2 , the raw p -value is 0.9220, and the adjusted p -value equals 1, as you would expect from theoretical considerations. The permutation p -values for S1 and S2 also happen to be the Bonferroni-adjusted p -values for this example.
The preceding table lists the OUTPERM= data set, which contains the exact permutation distributions for S1 and S2 in terms of cumulative probabilities.
Example 48.2. Freeman-Tukey and t-Tests with Bootstrap Resampling
The data for the following example are the same as for Example 48.1, except that a continuous variable T , which indicates the time of death of the animal, has been added.
data a; input S1 S2 T Dose @@; datalines; 0 1 104 1 1 0 80 1 0 1 104 1 0 1 104 1 0 1 100 1 1 0 104 1 1 0 85 2 1 0 60 2 0 1 89 2 1 0 96 2 0 1 96 2 1 0 99 2 1 0 60 3 1 0 50 3 1 0 80 3 0 1 98 3 0 1 99 3 1 0 50 3 ; proc multtest data=a bootstrap nsample=10000 pvals seed=37081 outsamp=res; test ft(S1 S2 / lowertailed) mean(T / lowertailed); class Dose; contrast 'Linear Trend' 0 1 2; run; proc print data=res(obs=36); run;
The BOOTSTRAP option in the PROC MULTTEST statement requests bootstrap resampling, and NSAMPLE=10000 requests 10,000 bootstrap samples. The seed for the random number generation is 37081. The OUTSAMP=RES option creates an output SAS data set containing the 10,000 bootstrap samples.
The TEST statement specifies the Freeman-Tukey test for S1 and S2 and specifies the t -test for T . Both tests are lower-tailed. The grouping variable in the CLASS statement is Dose , and the coefficients across the levels of Dose are 0, 1, and 2, as specified in the CONTRAST statement. PROC PRINT displays the first 36 observations of the Res data set containing the bootstrap samples.
The results from this analysis are listed in Output 48.2.1 through Output 48.2.5.
| |
The Multtest Procedure Model Information Test for discrete variables Freeman-Tukey Test for continuous variables Mean t-test Tails for discrete tests Lower-tailed Tails for continuous tests Lower-tailed Strata weights None P-value adjustment Bootstrap Center continuous variables Yes Number of resamples 10000 Seed 37081
| |
| |
The Multtest Procedure Contrast Coefficients Dose Contrast 1 2 3 Linear Trend 0 1 2
| |
| |
The Multtest Procedure Discrete Variable Tabulations Variable Dose Count NumObs Percent S1 1 2 6 33.33 S1 2 4 6 66.67 S1 3 4 6 66.67 S2 1 4 6 66.67 S2 2 2 6 33.33 S2 3 2 6 33.33 Continuous Variable Tabulations Standard Variable Dose NumObs Mean Deviation T 1 6 99.3333 9.6056 T 2 6 87.5000 14.4326 T 3 6 72.8333 22.7017
| |
| |
The Multtest Procedure p-Values Variable Contrast Raw Bootstrap S1 Linear Trend 0.8547 1.0000 S2 Linear Trend 0.1453 0.4471 T Linear Trend 0.0070 0.0253
| |
| |
Obs _sample_ _class_ _obs_ S1 S2 T 1 1 1 11 0 1 8.5000 2 1 1 16 0 1 25.1667 3 1 1 16 0 1 25.1667 4 1 1 14 1 0 22.8333 5 1 1 18 1 0 22.8333 6 1 1 14 1 0 22.8333 7 1 2 4 0 1 4.6667 8 1 2 12 1 0 11.5000 9 1 2 8 1 0 27.5000 10 1 2 7 1 0 2.5000 11 1 2 3 0 1 4.6667 12 1 2 12 1 0 11.5000 13 1 3 13 1 0 12.8333 14 1 3 5 0 1 0.6667 15 1 3 8 1 0 27.5000 16 1 3 5 0 1 0.6667 17 1 3 13 1 0 12.8333 18 1 3 6 1 0 4.6667 19 2 1 8 1 0 27.5000 20 2 1 3 0 1 4.6667 21 2 1 9 0 1 1.5000 22 2 1 13 1 0 12.8333 23 2 1 14 1 0 22.8333 24 2 1 12 1 0 11.5000 25 2 2 14 1 0 22.8333 26 2 2 18 1 0 22.8333 27 2 2 15 1 0 7.1667 28 2 2 6 1 0 4.6667 29 2 2 13 1 0 12.8333 30 2 2 1 0 1 4.6667 31 2 3 7 1 0 2.5000 32 2 3 7 1 0 2.5000 33 2 3 6 1 0 4.6667 34 2 3 13 1 0 12.8333 35 2 3 4 0 1 4.6667 36 2 3 6 1 0 4.6667
| |
The information in the preceding table corresponds to the specifications in the invocation of PROC MULTTEST.
The preceding table shows the coefficients from the CONTRAST statement, and they model a linear trend.
The summary statistics in the preceding table for S1 and S2 are the same as those from Example 48.1. The variables S1 and S2 are discrete, and T is a continuous variable. The mean, standard deviation, and sample size for each level of Dose is listed in the table for T . The p -values for S1 and S2 are from the Freeman-Tukey test, and the p -values for T are from the t -test.
The p -values are listed in the preceding table. The Raw column contains the results from the tests on the original data, and the Bootstrap column contains the bootstrap resampled adjustment to raw_p . Note that the adjusted p -values are larger than the raw p -values for all three variables. The adjusted p -values more accurately reflect the correlation of the raw p -values, the small size of the data, and the multiple testing.
The preceding table lists the first 36 observations of the SAS data set resulting from the OUTSAMP=RES option in the PROC MULTTEST statement. The entire data set has 180,000 observations, which is 10,000 times the number of observations in the data set. The _sample_ variable is the sample indicator and _class_ indicates the resampling group , that is, the level of the CLASS variable Dose assigned to the new observation. The number of the observation in the original data set is represented by _obs_ . Also listed are the values of the original test variables, S1 and S2 , and the mean-centered values of T .
Example 48.3. Peto Mortality-Prevalence Test
This example illustrates the use of the Peto mortality-prevalence test. The test is a combination of analyses about the prevalence of incidental tumors in the population and mortality due to fatal tumors .
In the data set, each observation represents an animal. The variables S1 ˆ’ S3 are three tumor types, with a value of 0 indicating no tumor, 1 indicating an incidental (nonlethal) tumor, and 2 indicating a lethal tumor. The time variable T indicates the time of death of the animal, a strata variable B is constructed from T , and the grouping variable Dose is drug dosage.
data a; input S1-S3 T Dose @@; if T<=90 then B=1; else B=2; datalines; 0 0 0 104 0 2 0 1 80 0 0 0 1 104 0 0 0 0 104 0 0 2 0 100 0 1 0 0 104 0 2 0 0 85 1 2 1 0 60 1 0 1 0 89 1 2 0 1 96 1 0 0 0 96 1 2 0 1 99 1 2 1 1 60 2 2 0 0 50 2 2 0 1 80 2 0 0 2 98 2 0 0 1 99 2 2 1 1 50 2 ; proc multtest data=a notables out=p stepsid; test peto(S1-S3 / permutation=20 time=T uppertailed); class Dose; strata B; contrast 'mort-prev' 0 1 2; run; proc print data=p; run;
The NOTABLES option in the PROC MULTTEST statement suppresses the display of the summary statistics for each variable. The OUT=P option creates an output SAS data set containing all p -values and intermediate statistics. The STEPSID option is used to adjust the p -values.
The TEST statement specifies an upper-tailed Peto test for S1 ˆ’ S3 . The mortality strata are defined by TIME= T , the death times. The CLASS statement contains the grouping variable Dose . The prevalence strata are defined by the STRATA statement as the blocking variable B . The CONTRAST statement lists the default linear trend coefficients. PROC PRINT displays the requested p -value data set.
The results from this analysis are listed in Output 48.3.1 through Output 48.3.4.
| |
The Multtest Procedure Model Information Test for discrete variables Peto Exact permutation distribution used Everywhere Tails for discrete tests Upper-tailed Strata weights Sample size P-value adjustment Stepdown Sidak
| |
| |
The Multtest Procedure Contrast Coefficients Dose Contrast 0 1 2 mort-prev 0 1 2
| |
| |
The Multtest Procedure p-Values Stepdown Variable Contrast Raw Sidak S1 mort-prev 0.0681 0.0814 S2 mort-prev 0.5000 0.5000 S3 mort-prev 0.0363 0.0781
| |
| |
Obs _test_ _var_ _contrast_ _strat_ _tstrat_ _value_ _exp_ _se_ raw_p stpsid_p 1 PETO S1 mort-prev 1 0 0 0.00000 0.00000 . . 2 PETO S1 mort-prev 2 0 0 0.62500 0.85696 . . 3 PETO S1 mort-prev 50 1 4 2.00000 1.12022 . . 4 PETO S1 mort-prev 60 1 3 1.75000 1.06654 . . 5 PETO S1 mort-prev 80 1 2 1.57143 1.04978 . . 6 PETO S1 mort-prev 85 1 1 0.75000 0.72169 . . 7 PETO S1 mort-prev 96 1 1 0.70000 0.78102 . . 8 PETO S1 mort-prev 98 1 0 0.00000 0.00000 . . 9 PETO S1 mort-prev 99 1 1 0.42857 0.72843 . . 10 PETO S1 mort-prev 100 1 0 0.00000 0.00000 . . 11 PETO S2 mort-prev 1 0 6 5.50000 1.05221 . . 12 PETO S2 mort-prev 2 0 0 0.00000 0.00000 . . 13 PETO S2 mort-prev 50 1 0 0.00000 0.00000 . . 14 PETO S2 mort-prev 60 1 0 0.00000 0.00000 . . 15 PETO S2 mort-prev 80 1 0 0.00000 0.00000 . . 16 PETO S2 mort-prev 85 1 0 0.00000 0.00000 . . 17 PETO S2 mort-prev 96 1 0 0.00000 0.00000 . . 18 PETO S2 mort-prev 98 1 0 0.00000 0.00000 . . 19 PETO S2 mort-prev 99 1 0 0.00000 0.00000 . . 20 PETO S2 mort-prev 100 1 0 0.00000 0.00000 . . 21 PETO S3 mort-prev 1 0 6 5.50000 1.05221 . . 22 PETO S3 mort-prev 2 0 4 2.22222 1.08298 . . 23 PETO S3 mort-prev 50 1 0 0.00000 0.00000 . . 24 PETO S3 mort-prev 60 1 0 0.00000 0.00000 . . 25 PETO S3 mort-prev 80 1 0 0.00000 0.00000 . . 26 PETO S3 mort-prev 85 1 0 0.00000 0.00000 . . 27 PETO S3 mort-prev 96 1 0 0.00000 0.00000 . . 28 PETO S3 mort-prev 98 1 2 0.62500 0.85696 . . 29 PETO S3 mort-prev 99 1 0 0.00000 0.00000 . . 30 PETO S3 mort-prev 100 1 0 0.00000 0.00000 . . 31 PETO S1 mort-prev . . 12 7.82500 2.42699 0.06808 0.08140 32 PETO S2 mort-prev . . 6 5.50000 1.05221 0.50000 0.50000 33 PETO S3 mort-prev . . 12 8.34722 1.73619 0.03627 0.07811
| |
The preceding information corresponds to the PROC MULTTEST invocation. In this case the totals for all prevalence and fatality strata are less than 20, so exact permutation tests are used everywhere, and the STEPSID adjustments are computed from these permutation distributions.
The contrast trend coefficients are listed in the preceding table. They happen to be the same as the levels of the Dose variable.
In the preceding p -Values table, the p -values for the Peto tests are listed in the Raw column, and the stepdown Sidak adjusted p -values are in the Stepdown Sidak column.
Significant p -values support the claim that higher dosage levels promote higher mortality and prevalence. The raw Peto test is significant at the 5% level for S3 , but the adjusted S3 test is no longer significant at 5%. The raw and adjusted p -values for S2 are the same because of the stepdown technique.
The preceding table lists the OUT= data set. The first 30 observations correspond to intermediate statistics used to compute the Peto p -values. The _test_ variable lists the name of the test, the _var_ variable lists the name of the TEST variables, and the _contrast_ variable lists the CONTRAST label. The _strat_ variable lists the level of the STRATA variable, and the _tstrat_ variable indicates whether or not the stratum corresponds to values of the TIME= variable. The _value_ variable is the observed contrast for a stratum and the _exp_ variable is its expected value. The variable _se_ contains the square root of the variance terms summed to form the denominator of the Peto statistics.
The final three observations correspond to the three Peto tests, with their p -values listed under the raw_p variable. The stpsid_p variable contains the stepdown Sidak adjusted p -values.
Example 48.4. Fisher Test with Permutation Resampling
These data, from Brown and Fears (1981), are the results from an 80-week carcino-genesis bioassay with female mice. Six tissue sites are examined at necropsy; 1 indicates the presence of a tumor and 0 the absence. A frequency variable Freq is included. A control and four different doses of a drug (in parts per milliliter) make up the levels of the grouping variable Dose .
data a; input Liver Lung Lymph Cardio Pitui Ovary Freq Dose$ @@; datalines; 1 0 0 0 0 0 8 CTRL 0 1 0 0 0 0 7 CTRL 0 0 1 0 0 0 6 CTRL 0 0 0 1 0 0 1 CTRL 0 0 0 0 0 1 2 CTRL 1 1 0 0 0 0 4 CTRL 1 0 1 0 0 0 1 CTRL 1 0 0 0 0 1 1 CTRL 0 1 1 0 0 0 1 CTRL 0 0 0 0 0 0 18 CTRL 1 0 0 0 0 0 9 4PPM 0 1 0 0 0 0 4 4PPM 0 0 1 0 0 0 7 4PPM 0 0 0 1 0 0 1 4PPM 0 0 0 0 1 0 2 4PPM 0 0 0 0 0 1 1 4PPM 1 1 0 0 0 0 4 4PPM 1 0 1 0 0 0 3 4PPM 1 0 0 0 1 0 1 4PPM 0 1 1 0 0 0 1 4PPM 0 1 0 1 0 0 1 4PPM 1 0 1 1 0 0 1 4PPM 0 0 0 0 0 0 15 4PPM 1 0 0 0 0 0 8 8PPM 0 1 0 0 0 0 3 8PPM 0 0 1 0 0 0 6 8PPM 0 0 0 1 0 0 3 8PPM 1 1 0 0 0 0 1 8PPM 1 0 1 0 0 0 2 8PPM 1 0 0 1 0 0 1 8PPM 1 0 0 0 1 0 1 8PPM 1 1 0 1 0 0 2 8PPM 1 1 0 0 0 1 2 8PPM 0 0 0 0 0 0 19 8PPM 1 0 0 0 0 0 4 16PPM 0 1 0 0 0 0 2 16PPM 0 0 1 0 0 0 9 16PPM 0 0 0 0 1 0 1 16PPM 0 0 0 0 0 1 1 16PPM 1 1 0 0 0 0 4 16PPM 1 0 1 0 0 0 1 16PPM 0 1 1 0 0 0 1 16PPM 0 1 0 1 0 0 1 16PPM 0 1 0 0 0 1 1 16PPM 0 0 1 1 0 0 1 16PPM 0 0 1 0 1 0 1 16PPM 1 1 1 0 0 0 2 16PPM 0 0 0 0 0 0 14 16PPM 1 0 0 0 0 0 8 50PPM 0 1 0 0 0 0 4 50PPM 0 0 1 0 0 0 8 50PPM 0 0 0 1 0 0 1 50PPM 0 0 0 0 0 1 4 50PPM 1 1 0 0 0 0 3 50PPM 1 0 1 0 0 0 1 50PPM 0 1 1 0 0 0 1 50PPM 0 1 0 0 1 1 1 50PPM 0 0 0 0 0 0 19 50PPM ; proc multtest data=a order=data notables out=p permutation nsample=1000 seed=764511; test fisher(Liver Lung Lymph Cardio Pitui Ovary / lowertailed); class Dose; freq Freq; run; proc print data=p; run;
In the PROC MULTTEST statement, the ORDER=DATA option is required to keep the levels of Dose in the order in which they appear in the data set. Without this option, the levels are sorted by their formatted value, resulting in an alphabetic ordering. The NOTABLES option suppresses the display of summary statistics, and the OUT=P option requests an output data set containing p -values. The PERMUTATION option specifies permutation resampling, NSAMPLE=1000 requests 1000 samples, and SEED=764511 provides a starting value for the random number generator. You should specify a seed if you need to duplicate resampling results.
To test for higher rates of tumor occurrence in the treatment groups compared to the control group, the LOWERTAILED option is specified in the TEST statement to produce a lower-tailed Fisher exact test for the six tissue sites. The Fisher test is appropriate for comparing a treatment and a control, but multiple testing can be a problem. Brown and Fears (1981) use a multivariate permutation to evaluate the entire collection of tests. PROC MULTTEST adjusts the p -values by simulation.
The treatments make up the levels of the grouping variable Dose , listed in the CLASS statement. Since no CONTRAST statement is specified, PROC MULTTEST uses the default pairwise contrasts with the first level of Dose . The FREQ statement is used since this is summary data containing frequency counts of occurrences.
The results from this analysis are listed in Output 48.4.1 through Output 48.4.4.
| |
The Multtest Procedure Model Information Test for discrete variables Fisher Tails for discrete tests Lower-tailed Strata weights None P-value adjustment Permutation Number of resamples 1000 Seed 764511
| |
| |
The Multtest Procedure Contrast Coefficients Dose Contrast CTRL 4PPM 8PPM 16PPM 50PPM CTRL vs. 4PPM 1 1 0 0 0 CTRL vs. 8PPM 1 0 1 0 0 CTRL vs. 16PPM 1 0 0 1 0 CTRL vs. 50PPM 1 0 0 0 1
| |
| |
The Multtest Procedure p-Values Variable Contrast Raw Permutation Liver CTRL vs. 4PPM 0.2828 0.9690 Liver CTRL vs. 8PPM 0.3069 0.9750 Liver CTRL vs. 16PPM 0.7102 1.0000 Liver CTRL vs. 50PPM 0.7718 1.0000 Lung CTRL vs. 4PPM 0.7818 1.0000 Lung CTRL vs. 8PPM 0.8858 1.0000 Lung CTRL vs. 16PPM 0.5469 1.0000 Lung CTRL vs. 50PPM 0.8498 1.0000 Lymph CTRL vs. 4PPM 0.2423 0.9430 Lymph CTRL vs. 8PPM 0.5898 1.0000 Lymph CTRL vs. 16PPM 0.0350 0.2480 Lymph CTRL vs. 50PPM 0.4161 0.9960 Cardio CTRL vs. 4PPM 0.3163 0.9770 Cardio CTRL vs. 8PPM 0.0525 0.3570 Cardio CTRL vs. 16PPM 0.4506 1.0000 Cardio CTRL vs. 50PPM 0.7576 1.0000 Pitui CTRL vs. 4PPM 0.1250 0.7260 Pitui CTRL vs. 8PPM 0.4948 1.0000 Pitui CTRL vs. 16PPM 0.2157 0.9050 Pitui CTRL vs. 50PPM 0.5051 1.0000 Ovary CTRL vs. 4PPM 0.9437 1.0000 Ovary CTRL vs. 8PPM 0.8126 1.0000 Ovary CTRL vs. 16PPM 0.7760 1.0000 Ovary CTRL vs. 50PPM 0.3689 0.9950
| |
| |
Obs _test_ _var_ _contrast_ _xval_ _mval_ _yval_ _nval_ raw_p perm_p sim_se 1 FISHER Liver CTRL vs. 4PPM 14 49 18 50 0.28282 0.969 0.005481 2 FISHER Liver CTRL vs. 8PPM 14 49 17 48 0.30688 0.975 0.004937 3 FISHER Liver CTRL vs. 16PPM 14 49 11 43 0.71022 1.000 0.000000 4 FISHER Liver CTRL vs. 50PPM 14 49 12 50 0.77175 1.000 0.000000 5 FISHER Lung CTRL vs. 4PPM 12 49 10 50 0.78180 1.000 0.000000 6 FISHER Lung CTRL vs. 8PPM 12 49 8 48 0.88581 1.000 0.000000 7 FISHER Lung CTRL vs. 16PPM 12 49 11 43 0.54685 1.000 0.000000 8 FISHER Lung CTRL vs. 50PPM 12 49 9 50 0.84978 1.000 0.000000 9 FISHER Lymph CTRL vs. 4PPM 8 49 12 50 0.24228 0.943 0.007332 10 FISHER Lymph CTRL vs. 8PPM 8 49 8 48 0.58977 1.000 0.000000 11 FISHER Lymph CTRL vs. 16PPM 8 49 15 43 0.03498 0.248 0.013656 12 FISHER Lymph CTRL vs. 50PPM 8 49 10 50 0.41607 0.996 0.001996 13 FISHER Cardio CTRL vs. 4PPM 1 49 3 50 0.31631 0.977 0.004740 14 FISHER Cardio CTRL vs. 8PPM 1 49 6 48 0.05254 0.357 0.015151 15 FISHER Cardio CTRL vs. 16PPM 1 49 2 43 0.45061 1.000 0.000000 16 FISHER Cardio CTRL vs. 50PPM 1 49 1 50 0.75758 1.000 0.000000 17 FISHER Pitui CTRL vs. 4PPM 0 49 3 50 0.12496 0.726 0.014104 18 FISHER Pitui CTRL vs. 8PPM 0 49 1 48 0.49485 1.000 0.000000 19 FISHER Pitui CTRL vs. 16PPM 0 49 2 43 0.21572 0.905 0.009272 20 FISHER Pitui CTRL vs. 50PPM 0 49 1 50 0.50505 1.000 0.000000 21 FISHER Ovary CTRL vs. 4PPM 3 49 1 50 0.94372 1.000 0.000000 22 FISHER Ovary CTRL vs. 8PPM 3 49 2 48 0.81260 1.000 0.000000 23 FISHER Ovary CTRL vs. 16PPM 3 49 2 43 0.77596 1.000 0.000000 24 FISHER Ovary CTRL vs. 50PPM 3 49 5 50 0.36889 0.995 0.002230
| |
The preceding table lists the PROC MULTTEST specifications.
The preceding table lists the default contrasts for the Fisher test. Note that each dose is compared with the control.
The preceding p -Values table lists p -values for the Fisher exact tests and their permutation-based adjustments. As noted by Brown and Fears, only one of the twenty-four tests is significant at the 5% level (Lymph, CTRL vs. 16PPM). Brown and Fears report a 12% chance of observing at least one significant raw p -value for 16PPM and a 9% chance of observing at least one significant raw p -value for Lymph (both at the 5% level). Adjusted p -values exhibit much lower chances of false significances. For this example, none of the adjusted p -values are close to significant.
The preceding table lists the OUT= data set. The _test_ , _var_ ,and _contrast_ variables provide the TEST name, TEST variable, and CONTRAST label, respectively. The _xval_ , _mval_ , _yval_ ,and _nval_ variables contain the components used to compute the Fisher exact tests from the hypergeometric distribution. The raw_p variable contains the p -values from the Fisher exact tests, and the perm_p variable contains their permutation-based adjustments. The variable sim_se is the simulation standard error from the permutation resampling.
Example 48.5. Inputting Raw p-Values
This example illustrates how to use PROC MULTTEST to multiplicity-adjust a collection of raw p -values obtained from some other source. This is a valuable option for those cases where PROC MULTTEST cannot compute the raw p -values directly.
data a; input Test$ Raw_P; datalines; test1 .09108 test2 .69122 test3 .00177 test4 .57181 test5 .03121 test6 .01413 ; proc multtest pdata=a holm hoc fdr; run;
Note that there are no statements other than the PROC MULTTEST statement using the p -value input mode. In this example, the raw p -values are adjusted using the Holm, Hochberg, and Benjamini and Hocherg (FDR) methods .
The output from this analysis is listed in Output 48.5.1.
| |
The Multtest Procedure p-Values False Stepdown Discovery Test Raw Bonferroni Hochberg Rate 1 0.0911 0.2732 0.2732 0.1366 2 0.6912 1.0000 0.6912 0.6912 3 0.0018 0.0106 0.0106 0.0106 4 0.5718 1.0000 0.6912 0.6862 5 0.0312 0.1248 0.1248 0.0624 6 0.0141 0.0707 0.0707 0.0424
| |
Note that the adjusted p -values for the Hochberg method ( hoc_p ) are less than or equal to those for the Holm method ( stpbon_p ). In turn , the adjusted p -values for the Benjamini and Hochberg method ( fdr_p ) are less than or equal to those for the Hochberg method. These comparisons hold generally for all p -value configurations. The FDR method controls the false discovery rate and not the familywise error rate. The Hochberg method controls the familywise error rate under independence. The Holm method controls the familywise error rate without assuming independence.
- ERP Systems Impact on Organizations
- Challenging the Unpredictable: Changeable Order Management Systems
- Distributed Data Warehouse for Geo-spatial Services
- A Hybrid Clustering Technique to Improve Patient Data Quality
- Relevance and Micro-Relevance for the Professional as Determinants of IT-Diffusion and IT-Use in Healthcare