On the optimistic performance evaluation of newly introduced bioinformatic methods
Abstract Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2021-05-01
|
| Series: | Genome Biology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s13059-021-02365-4 |
| id |
doaj-af463957588640b9954e81012da31227 |
|---|---|
| record_format |
Article |
| spelling |
doaj-af463957588640b9954e81012da312272021-05-11T15:00:25ZengBMCGenome Biology1474-760X2021-05-012211810.1186/s13059-021-02365-4On the optimistic performance evaluation of newly introduced bioinformatic methodsStefan Buchka0Alexander Hapfelmeier1Paul P. Gardner2Rory Wilson3Anne-Laure Boulesteix4Institute for Medical Information Processing, Biometry and Epidemiology, LMUInstitute of Medical Informatics, Statistics and Epidemiology, School of Medicine, TUMDepartment of Biochemistry, University of OtagoResearch Unit Molecular Epidemiology, Institute of Epidemiology, Helmholtz Zentrum München, German Research Center for Environmental HealthInstitute for Medical Information Processing, Biometry and Epidemiology, LMUAbstract Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray.https://doi.org/10.1186/s13059-021-02365-4BenchmarkingOptimistic biasNeutral comparison studyIllumina HumanMethylation450K BeadChipNormalization |
| collection |
DOAJ |
| language |
English |
| format |
Article |
| sources |
DOAJ |
| author |
Stefan Buchka Alexander Hapfelmeier Paul P. Gardner Rory Wilson Anne-Laure Boulesteix |
| spellingShingle |
Stefan Buchka Alexander Hapfelmeier Paul P. Gardner Rory Wilson Anne-Laure Boulesteix On the optimistic performance evaluation of newly introduced bioinformatic methods Genome Biology Benchmarking Optimistic bias Neutral comparison study Illumina HumanMethylation450K BeadChip Normalization |
| author_facet |
Stefan Buchka Alexander Hapfelmeier Paul P. Gardner Rory Wilson Anne-Laure Boulesteix |
| author_sort |
Stefan Buchka |
| title |
On the optimistic performance evaluation of newly introduced bioinformatic methods |
| title_short |
On the optimistic performance evaluation of newly introduced bioinformatic methods |
| title_full |
On the optimistic performance evaluation of newly introduced bioinformatic methods |
| title_fullStr |
On the optimistic performance evaluation of newly introduced bioinformatic methods |
| title_full_unstemmed |
On the optimistic performance evaluation of newly introduced bioinformatic methods |
| title_sort |
on the optimistic performance evaluation of newly introduced bioinformatic methods |
| publisher |
BMC |
| series |
Genome Biology |
| issn |
1474-760X |
| publishDate |
2021-05-01 |
| description |
Abstract Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray. |
| topic |
Benchmarking Optimistic bias Neutral comparison study Illumina HumanMethylation450K BeadChip Normalization |
| url |
https://doi.org/10.1186/s13059-021-02365-4 |
| work_keys_str_mv |
AT stefanbuchka ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods AT alexanderhapfelmeier ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods AT paulpgardner ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods AT rorywilson ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods AT annelaureboulesteix ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods |
| _version_ |
1721443787785371648 |