Subjective Ranking of Quality of Research by Subject Matter Area

2017

This is a subjective ranking of topical areas by the typical quality of research published in the area. Keep in mind that top-quality research can occur in any area when the research team is multi-disciplinary, team members are at the top of their game, and peer review is functional.

Author

Affiliation

Frank Harrell

Vanderbilt University
School of Medicine
Department of Biostatistics

Published

March 16, 2017

While being engaged in biomedical research for a few decades and watching reproducibility of research as a whole, I’ve developed my own ranking of reliability/quality/usefulness of research across several subject matter areas. This list is far from complete. Let’s start with a subjective list of what I perceive as the areas in which published research is least likely to be both true and useful. The following list is ordered in ascending order of quality, with the most problematic area listed first. You’ll notice that there is a vast number of areas not listed for which I have minimal experience. Some excellent research is done in all subject areas. This list is based on my perception of the proportion of publications in the indicated area that are rigorously scientific, reproducible, and useful.

Comments

Subject Areas With Least Reliable/Reproducible/Useful Research

any area where there is no pre-specified statistical analysis plan and the analysis can change on the fly when initial results are disappointing
behavioral psychology
studies of corporations to find characteristics of “winners”; regression to the mean kicks in making predictions useless for changing your company
animal experiments on fewer than 30 animals
discovery genetics not making use of biology while doing large-scale variant/gene screening
nutritional epidemiology
electronic health record research reaching clinical conclusions without understanding confounding by indication and other limitations of data
pre-post studies with no randomization
non-nutritional epidemiology not having a fully pre-specified statistical analysis plan [few epidemiology papers use state-of-the-art statistical methods and have a sensitivity analysis related to unmeasured confounders]
prediction studies based on dirty and inadequate data
personalized medicine
biomarkers
observational treatment comparisons that do not qualify for the second list (below)
small adaptive dose-finding cancer trials (3+3 etc.)

Subject Areas With Most Reliable/Reproducible/Useful Research

The most reliable and useful research areas are listed first. All of the following are assumed to (1) have a prospective pre-specified statistical analysis plan and (2) purposeful prospective quality-controlled data acquisition (yes this applies to high-quality non-randomized observational research).

randomized crossover studies
multi-center randomized experiments
single-center randomized experiments with non-overly-optimistic sample sizes
adaptive randomized clinical trials with large sample sizes
physics
pharmaceutical industry research that is overseen by FDA
cardiovascular research
observational research [however only a very small minority of observational research projects have a prospective analysis plan and high enough data quality to qualify for this list]

Some Suggested Remedies

Peer review of research grants and manuscripts is done primarily by experts in the subject matter area under study. Most journal editors and grant reviewers are not expert in biostatistics. Every grant application and submitted manuscript should undergo rigorous methodologic peer review by methodologic experts such as biostatisticians and epidemiologists. All data analyses should be driven by a prospective statistical analysis plan, and the entire self-contained data manipulation and analysis code should be submitted to journals so that potential reproducibility and adherence to the statistical analysis plan can be confirmed. Readers should have access to the data in most cases and should be able to reproduce all study findings using the authors’ code, plus run their own analyses on the authors’ data to check robustness of findings. Medical journals are reluctant to (1) publish critical letters to the editor and (2) retract papers. This has to change.

In academia, too much credit is still given to the quantity of publications and not to their quality and reproducibility. This too must change. The pharmaceutical industry has FDA to validate their research. The NIH does not serve this role for academia.

Rochelle Tractenberg, Chair of the American Statistical Association Committee on Professional Ethics and a biostatistician at Georgetown University said in a 2017-02-22 interview with The Australian that many questionable studies would not have been published had formal statistical reviews been done. When she reviews a paper she starts with the premise that the statistical analysis was incorrectly executed. She stated that “Bad statistics is bad science.”

Discussion Archive (2017)

Reuse

CC BY 4.0

--- title: Subjective Ranking of Quality of Research by Subject Matter Area author: - name: Frank Harrell url: https://hbiostat.org affiliation: Vanderbilt University<br>School of Medicine<br>Department of Biostatistics date: 2017-03-16 modified: 2017-03-27 categories: [2017] description: "This is a subjective ranking of topical areas by the typical quality of research published in the area. Keep in mind that top-quality research can occur in any area when the research team is multi-disciplinary, team members are at the top of their game, and peer review is functional." --- While being engaged in biomedical research for a few decades and watching reproducibility of research as a whole, I've developed my own ranking of reliability/quality/usefulness of research across several subject matter areas. This list is far from complete. Let's start with a subjective list of what I perceive as the areas in which published research is least likely to be both true and useful. The following list is ordered in ascending order of quality, with the most problematic area listed first. You'll notice that there is a vast number of areas not listed for which I have minimal experience. **Some excellent research is done in all subject areas.** This list is based on my perception of the *proportion* of publications in the indicated area that are rigorously scientific, reproducible, and useful. [[Comments](https://hbiostat.org/comment.html)]{.aside} #### Subject Areas With Least Reliable/Reproducible/Useful Research 1. any area where there is no pre-specified statistical analysis plan and the analysis can change on the fly when initial results are disappointing 2. behavioral psychology 3. studies of corporations to find characteristics of "winners"; regression to the mean kicks in making predictions useless for changing your company 4. animal experiments on fewer than 30 animals 5. discovery genetics not making use of biology while doing large-scale variant/gene screening 6. nutritional epidemiology 7. electronic health record research reaching clinical conclusions without understanding confounding by indication and other limitations of data 8. pre-post studies with no randomization 9. non-nutritional epidemiology not having a fully pre-specified statistical analysis plan [few epidemiology papers use state-of-the-art statistical methods and have a sensitivity analysis related to unmeasured confounders] 10. prediction studies based on dirty and inadequate data 11. personalized medicine 12. biomarkers 13. observational treatment comparisons that do not qualify for the second list (below) 14. small adaptive dose-finding cancer trials (3+3 etc.) #### Subject Areas With Most Reliable/Reproducible/Useful Research The most reliable and useful research areas are listed first. All of the following are assumed to (1) have a prospective pre-specified statistical analysis plan and (2) purposeful prospective quality-controlled data acquisition (yes this applies to high-quality non-randomized observational research). 1. randomized crossover studies 2. multi-center randomized experiments 3. single-center randomized experiments with non-overly-optimistic sample sizes 4. adaptive randomized clinical trials with large sample sizes 5. physics 6. pharmaceutical industry research that is overseen by FDA 7. cardiovascular research 8. observational research [however only a very small minority of observational research projects have a prospective analysis plan and high enough data quality to qualify for this list] #### Some Suggested Remedies Peer review of research grants and manuscripts is done primarily by experts in the subject matter area under study. Most journal editors and grant reviewers are not expert in biostatistics. Every grant application and submitted manuscript should undergo rigorous methodologic peer review by methodologic experts such as biostatisticians and epidemiologists. All data analyses should be driven by a prospective statistical analysis plan, and the entire self-contained data manipulation and analysis code should be submitted to journals so that potential reproducibility and adherence to the statistical analysis plan can be confirmed. Readers should have access to the data in most cases and should be able to reproduce all study findings using the authors' code, plus run their own analyses on the authors' data to check robustness of findings. Medical journals are reluctant to (1) publish critical letters to the editor and (2) retract papers. This has to change. In academia, too much credit is still given to the quantity of publications and not to their quality and reproducibility. This too must change. The pharmaceutical industry has FDA to validate their research. The NIH does not serve this role for academia. Rochelle Tractenberg, Chair of the American Statistical Association Committee on Professional Ethics and a biostatistician at Georgetown University said in a 2017-02-22 interview with *The Australian* that many questionable studies would not have been published had formal statistical reviews been done. When she reviews a paper she starts with the premise that the statistical analysis was incorrectly executed. She stated that "Bad statistics is bad science." ---- ## [Discussion Archive](disqus.pdf) (2017)