• Login
    View Item 
    •   Athenaeum Home
    • BioMed Central Open Access Articles
    • Open Access Articles by UGA Faculty
    • View Item
    •   Athenaeum Home
    • BioMed Central Open Access Articles
    • Open Access Articles by UGA Faculty
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Optimally splitting cases for training and testing high dimensional classifiers

    Thumbnail
    View/Open
    1755-8794-4-31.xml (67.01Kb)
    1755-8794-4-31.pdf (363.6Kb)
    1755-8794-4-31-S1.PDF (424.1Kb)
    Date
    2011-04-08
    Author
    Dobbin, Kevin K
    Simon, Richard M
    Metadata
    Show full item record
    Abstract
    Abstract Background We consider the problem of designing a study to develop a predictive classifier from high dimensional data. A common study design is to split the sample into a training set and an independent test set, where the former is used to develop the classifier and the latter to evaluate its performance. In this paper we address the question of what proportion of the samples should be devoted to the training set. How does this proportion impact the mean squared error (MSE) of the prediction accuracy estimate? Results We develop a non-parametric algorithm for determining an optimal splitting proportion that can be applied with a specific dataset and classifier algorithm. We also perform a broad simulation study for the purpose of better understanding the factors that determine the best split proportions and to evaluate commonly used splitting strategies (1/2 training or 2/3 training) under a wide variety of conditions. These methods are based on a decomposition of the MSE into three intuitive component parts. Conclusions By applying these approaches to a number of synthetic and real microarray datasets we show that for linear classifiers the optimal proportion depends on the overall number of samples available and the degree of differential expression between the classes. The optimal proportion was found to depend on the full dataset size (n) and classification accuracy - with higher accuracy and smaller n resulting in more assigned to the training set. The commonly used strategy of allocating 2/3rd of cases for training was close to optimal for reasonable sized datasets (n ≥ 100) with strong signals (i.e. 85% or greater full dataset accuracy). In general, we recommend use of our nonparametric resampling approach for determing the optimal split. This approach can be applied to any dataset, using any predictor development method, to determine the best split.
    URI
    http://dx.doi.org/10.1186/1755-8794-4-31
    http://hdl.handle.net/10724/19647
    Collections
    • Open Access Articles by UGA Faculty

    About Athenaeum | Contact Us | Send Feedback
     

     

    Browse

    All of AthenaeumCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

    My Account

    LoginRegister

    About Athenaeum | Contact Us | Send Feedback