Feature selection by higher criticism thresholding achieves the optimal phase diagram.

*(English)*Zbl 1185.62113Summary: We consider two-class linear classification in a high-dimensional, small-sample-size setting. Only a small fraction of the features are useful, these being unknown to us, and each useful feature contributes weakly to the classification decision. This was called the rare/weak (RW) model in our previous study, Proc. Natl Acad. Sci. USA 105, 14790–14795 (2008). We select features by thresholding feature Z-scores. The threshold is set by higher criticism (HC). For \(1\leq i\leq N\), let \(\pi_i\) denote the \(p\)-value associated with the \(i\)th \(Z\)-score and \(\pi (i)\) denote the ith order statistic of the collection of \(p\)-values. The HC threshold (HCT) is the order statistic of the \(Z\)-score corresponding to index \(i\) maximizing \((i/N-\pi_{i})/\sqrt{i/N(1-i/N)}\). The ideal threshold optimizes the classification error.

In that previous study, we showed that HCT was numerically close to the ideal threshold. We formalize an asymptotic framework for studying the RW model, considering a sequence of problems with increasingly many features and relatively fewer observations. We show that, along this sequence, the limiting performance of ideal HCT is essentially just as good as the limiting performance of ideal thresholding. Our results describe two-dimensional phase space, a two-dimensional diagram with coordinates quantifying ‘rare’ and ‘weak’ in the RW model. The phase space can be partitioned into two regions-one where ideal threshold classification is successful, and one where the features are so weak and so rare that it must fail. Surprisingly, the regions where ideal HCT succeeds and fails make exactly the same partition of the phase diagram. Other threshold methods, such as false (feature) discovery rate (FDR) threshold selection, are successful in a substantially smaller region of the phase space than either HCT or ideal thresholding. The FDR and local FDR of the ideal and HC threshold selectors have surprising phase diagrams, which are also described. Results showing the asymptotic equivalence of HCT with ideal HCT can be found in a forthcoming paper.

In that previous study, we showed that HCT was numerically close to the ideal threshold. We formalize an asymptotic framework for studying the RW model, considering a sequence of problems with increasingly many features and relatively fewer observations. We show that, along this sequence, the limiting performance of ideal HCT is essentially just as good as the limiting performance of ideal thresholding. Our results describe two-dimensional phase space, a two-dimensional diagram with coordinates quantifying ‘rare’ and ‘weak’ in the RW model. The phase space can be partitioned into two regions-one where ideal threshold classification is successful, and one where the features are so weak and so rare that it must fail. Surprisingly, the regions where ideal HCT succeeds and fails make exactly the same partition of the phase diagram. Other threshold methods, such as false (feature) discovery rate (FDR) threshold selection, are successful in a substantially smaller region of the phase space than either HCT or ideal thresholding. The FDR and local FDR of the ideal and HC threshold selectors have surprising phase diagrams, which are also described. Results showing the asymptotic equivalence of HCT with ideal HCT can be found in a forthcoming paper.

##### MSC:

62H30 | Classification and discrimination; cluster analysis (statistical aspects) |

62G30 | Order statistics; empirical distribution functions |

##### Keywords:

asymptotic rare/weak model; false discovery rate; linear classification; phase diagram; Fisher’s separation measure; feature selection by thresholding##### References:

[1] | 34 pp 584– (2006) · Zbl 1092.62005 |

[2] | J R STAT SOC B 57 pp 289– (1995) |

[3] | BERNOULLI 10 pp 989– (2004) · Zbl 1064.62073 |

[4] | 35 pp 2421– (2007) · Zbl 1360.62113 |

[5] | Monthly Notices of the Royal Astronomical Society 362 pp 826– (2005) |

[6] | PERSPECTIVES IN MATHEMATICAL SCIENCES I PROBABILITY AND STATISTICS 7 pp 109– (2009) |

[7] | 32 pp 962– (2004) · Zbl 1092.62051 |

[8] | 34 pp 2980– (2006) · Zbl 1114.62010 |

[9] | PNAS 105 (39) pp 14790– (2008) · Zbl 1357.62212 |

[10] | PROB THEORY RELATED FIELDS 2 pp 277– (1994) |

[11] | J R STAT SOC B 54 pp 41– (1992) |

[12] | 99 pp 96– (2001) |

[13] | Fan 36 (6) pp 2605– (2008) · Zbl 1360.62327 |

[14] | Golub, Science 286 (5439) pp 531– (1999) |

[15] | 36 pp 381– (2008) · Zbl 1139.62049 |

[16] | J R STAT SOC B 70 pp 158– (2008) |

[17] | Hedenfalk, New England Journal of Medicine 344 (8) pp 539– (2001) |

[18] | MATH METHODS STAT 6 pp 47– (1997) |

[19] | PHIL TRANS R SOC A 367 pp 4427– (2009) · Zbl 1185.62115 |

[20] | 35 pp 2018– (2007) · Zbl 1126.62030 |

[21] | PNAS 106 (22) pp 8859– (2009) · Zbl 1203.68064 |

[22] | EURASIP J APPL SIGNAL PROCESS 15 pp 2470– (2005) |

[23] | 32 pp 1594– (2004) · Zbl 1047.62008 |

[24] | Tibshirani, PNAS 99 (10) pp 6567– (2002) |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.