Ten steps toward a better personality science – how quality may be rewarded more in research evaluation

Supplementary materials [Review] to: Leising, D., Thielmann, I., Glöckner, A., Gärtner, A., & Schönbrodt, F. (2022). Ten steps toward a better personality science: How quality may be rewarded more in research evaluation. Personality Science, 3, Article e6029. https://doi.org/10.5964/ps.6029

We point out ten steps that we think will go a long way in improving personality science. The first five steps focus on fostering consensus regarding (1) research goals, (2) terminology, (3) measurement practices, (4) data handling, and (5) the current state of theory and evidence. The other five steps focus on improving the credibility of empirical research, through (6) formal modelling, (7) mandatory pre-registration for confirmatory claims, (8) replication as a routine practice, (9) planning for informative studies (e.g., in terms of statistical power), and (10) making data, analysis scripts, and materials openly available. The current, quantity-based incentive structures in academia clearly stand in the way of implementing many of these practices, resulting in a research literature with sometimes questionable utility and/or integrity. As a solution, we propose a quality-based reward scheme that explicitly weights published research by its Good Science merits. Scientists need to be increasingly rewarded for doing good work, not just lots of work.

We respond to the comments (https://doi.org/10.5964/ps.9227) on our “Ten Steps” paper (https://doi.org/10.5964/ps.6029), focusing on the most prominent themes: (1) What motivates scientists?, (2) Consensus-building (Is our field ready? May there be adverse side-effects? How shall we do it?), (3) How may institutional change be facilitated?, (4) Diversity (of participants, stimuli, methodology, measures, and among researchers), (5) The reliability of our proposed scoring system, and (6) The real-world relevance of personality research. We stand by our call for more concerted consensus-building and offer a few clarifications in this regard. We also issue four specific calls to action to our colleagues in the field: (a) specify legitimate paths to greater consensus, (b) explicate what constitutes good “qualitative” research, (c) help establish a widely used, public domain item database, and (d) determine what the most important contemporary goals of personality research are.

The current collection comprises several comments to "Leising, D., Thielmann, I. Glöckner, A., Gärtner, A., & Schönbrodt, F. (2022). Ten steps toward a better personality science: How quality may be rewarded more in research evaluation" (https://doi.org/10.5964/ps.6029). In their target paper, Leising et al. presented a number of steps that personality researchers (and others) may take to improve the scientific standards in their field. Leising et al. answer to these comments in a rejoinder (https://doi.org/10.5964/ps.7961).