SGoF+ program web page

 

Rationale of the SGoF+ approach

 

SGoF and SGoF+ tests control for FWER (FamilyWise Error Rate) in the weak sense. FWER control means that what is aimed to be controlled is the probability of committing any (one or more than one) type I error in families of (simultaneous) comparisons. Weak control means that the FWER is maintained below a given error level under the intersection (or complete) null hypothesis. Given a number S of tests, in Bonferroni technique the error rate per comparison is fixed to α/S which warrants strong FWER control (i.e. under all configurations of the true and false hypotheses). Therefore the per-comparison error rate diminishes with the number of tests. The problem here is that the power of each test also depends on the significance level. With very stringent significance level we will have very low power. SGoF method just performs an exact binomial test onto the expected (γ*S) under the complete null hypothesis (all nulls are true) and the observed proportion of tests with p-values below γ. The binomial test is performed at the α level. By default SGoF uses γ=α.

 

There is a complementary strategy to SGoF called SGoficance. We could look in different places of the original p-value distribution and "paint" the trace of effects detected by SGoF and the FDR commited in each case. That is, for a number S of tests we compute the expectation γ*S and test at the α level to compare with the number of observed p-values that are minor or equal than γ. So, we relax the SGoF condition γ=α just to paint and see the different number of detected effects and FDR commited at different γ's. This picture can be obtained using the R script for the SGoFicance graphical tool.
NOTE that if you automatize the choosing of gamma parameter as the maximizer of the Kolmogorov-Smirnov statistic i.e. you use the γ value that maximize the difference with expectations then you should correct with the Kolmogorov-Smirnov test as in SGoF+. So, you should use the SgoF+ output instead the SGoF one in this case.

 

In the case of SGoF+ a data-driven choice of γ is performed so the difference between the expected and the observed proportion of significants is maximized. According to this, the null hypothesis for the binomial test is settled by computing the Kolmogorov-Smirnov statistic onto the p-value distribution. The test is then performed at an independent level α by comparing the number of observed rejections with the critical value of the one-sided Kolmogorov-Smirnov test for uniformity. This warrants weak FWER control at the α level (0.05 by default). Therefore SGoF+ provides a power increase of about 7-10% over SGoF. Importantly, in the case of SGoF and SGoF+, the per test error rate is proportional to a factor that increases with the number of tests resolving in this way the trade-off between type I error and statistical power.

 

Alternatively, the false discovery rate (FDR) based methods as the Benjamini-Hochberg one aim to control the proportion of false positives among the total ones (i.e. the proportion of the rejected null hypotheses which are erroneously rejected). Therefore, FDR methods suppose an important improvement compared to Bonferroni allowing a substantial gain in power. However such methods are strongly dependent on the magnitude of the deviations of the alternative hypotheses from the null one, the relative frequency of false hypothesis and on the sample size. When deviations are weak or intermediate and number of effects (for example number of genes under selection, number of significant protein interactions from a proteome) are relatively low, the power of Benjamini-Hochberg method under biological sample sizes is low and diminishes as the number of tests becomes higher. So, this should make SGoF and SGoF+ interesting strategies for multitest adjustment when working with high-dimensional biological data.

 

Another method with similar behaviour as SGoF+ is the Sequential Fisher combined probability test (SFisher). The null hypothesis for the SFisher metatest is that all the nulls are true. To test this, all the p-values are transformed and summed to give a single chi-square value via the so-called the Fisher's combined probability test. This results in a meta test p-value. If it is significant at the chosen meta test α level (eg α=0.05) then it can be concluded that at least one of the tests in the list has a null hypothesis that is false. The best candidate is the test with the lowest p-value. The combining probabilities procedure is then repeated but with the exclusion of the lowest p-value. The sequential procedure continues with exclusion of successive p-values until the meta test p-value is no longer significant. This method has proven to have similar ROC curves and pFDR values as SGoF+ (unpublished data).

 

SGoF and SGoF+ are calculated by an exact binomial test when the number of tests is lower than 10 and a G test with the Williams' correction in any other case.

 

For more detailed explanations please see the papers:

SGoF: Carvajal-Rodríguez et al (BMC Bioinformatics 10:209, 2009) [PDF]
SGoFicance: de Uña-Alvarez & Carvajal-Rodríguez (PLoS ONE 5 (12):e15930, 2010) [PDF]
SFisher: AP. Diz et al (Molecular & Cellular Proteomics 10: M110.004374) [Online early]
SGoF+: Carvajal-Rodriguez A, de Uña-Alvarez J (PLoS ONE 6(9): e24700, 2011) [Journal Link]

 

 

Computation of the proportion of true nulls

 

When computing q-values it is necessary to estimate the proportion of true null hypotheses (π₀). There are different methods to do so and the SGoF+ software incorporates some of them. The methods are:

 

• Bootstrap (B): The bootstrap method (Storey et al 2004).

   

• SDPB (D): The standard deviation proportional bounding method (Meinshausen and Rice 2006)

   

• LBE (L): Location Based Estimator (Dalmasso et al 2005).

   

• Smooth (S): Cubic Spline method (Storey & Tibshirani 2003). This is performed without limiting the degrees of freedom.

 

By default SGoF+ performs all these methods and compute π₀ as the mathematical mode of them. The mode is computed by grouping the values in intervals of 5% longitude beginning from the highest π₀ value. If the set of π₀'s is multimodal the highest mode is selected. Alternatively, SGoF+ also permits to select as π₀ the highest from the different computed values or just the value obtained by any of the methods.

 

 

Computation of q-values

 

Once the value of π₀ is estimated the q-value of a given p-value p(i) is computed as:

 

where S is the number of tests, i is the position of p(i) in the list of sorted p-values and C = 1.0 - (1.0-p(t))^S if the robust option was selected or, on the contrary, C = 1 (by default).