The Impact of Open Science Practices on Research Methodology: A Case Study for Research in Judgment and Decision Making
Author(s) / Creator(s)
Jekel, Marc
Glöckner, Andreas
Fiedler, Susann
Allstadt Torras, Ramona
Dorrough, Angela
Mischkowski, Dorothee
Franke, Nicole
Goltermann, Janik
Miketta, Stefanie
Abstract / Description
In 2011, the journal Judgment and Decision Making introduced as one of the first journals in psychology a standard request for data at submission of an article, which are also used for checks in the review process. In a sample of 71 articles published in this journal we investigate the effectiveness of this policy with respect to the prevalence of direct and indirect open science measures and their development between 2012 and 2018. For 100% of the articles data was available, 80% of the original authors responded positively to student requests for cooperation in conducting replications of their publications, for 94% of the articles materials were available or shared by the authors on request, and 96% of the original analyses were reproducible also by students, 30% of them with minor deviations or after further clarification. The usage of a priori power analyses (10%) and the reporting of effect sizes (66%) were considerably lower but increased over time. For only 4% of the articles analysis scripts were directly available, and none of the studies pre-registered hypotheses or used a pre-registered report format. There was no indication of the usage of small underpowered studies (Md(N) = 193, average effect size r = 0.30) and the p-values showed the expected right-skewed distribution without a bunching of p-values just below p = .05. Overall, adoption rates of open science principle are higher than in other fields and the journals policies were successful in fostering adherence to open science principles. A general culture of openness to reproducing analyses and replication of findings was established, which allows for a cumulative development of knowledge to which also student research can contribute.
Persistent Identifier
Date of first publication
2019-03-12
Is part of
Open Science 2019, Trier, Germany
Publisher
ZPID (Leibniz Institute for Psychology Information)
Citation
Jekel, M., Glöckner, A., Fiedler, S., Allstadt Torras, R., Dorrough, A., Mischkowski, D., … Miketta, S. (2019, March 12). The Impact of Open Science Practices on Research Methodology: A Case Study for Research in Judgment and Decision Making. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2389
-
b_2_Jekel_Trier_Open_Science_final.pdfAdobe PDF - 1.63MBMD5: 079c0036020a033ae708662624914420Description: Conference Talk
-
There are no other versions of this object.
-
Author(s) / Creator(s)Jekel, Marc
-
Author(s) / Creator(s)Glöckner, Andreas
-
Author(s) / Creator(s)Fiedler, Susann
-
Author(s) / Creator(s)Allstadt Torras, Ramona
-
Author(s) / Creator(s)Dorrough, Angela
-
Author(s) / Creator(s)Mischkowski, Dorothee
-
Author(s) / Creator(s)Franke, Nicole
-
Author(s) / Creator(s)Goltermann, Janik
-
Author(s) / Creator(s)Miketta, Stefanie
-
PsychArchives acquisition timestamp2019-04-01T14:35:08Z
-
Made available on2019-04-01T14:35:08Z
-
Date of first publication2019-03-12
-
Abstract / DescriptionIn 2011, the journal Judgment and Decision Making introduced as one of the first journals in psychology a standard request for data at submission of an article, which are also used for checks in the review process. In a sample of 71 articles published in this journal we investigate the effectiveness of this policy with respect to the prevalence of direct and indirect open science measures and their development between 2012 and 2018. For 100% of the articles data was available, 80% of the original authors responded positively to student requests for cooperation in conducting replications of their publications, for 94% of the articles materials were available or shared by the authors on request, and 96% of the original analyses were reproducible also by students, 30% of them with minor deviations or after further clarification. The usage of a priori power analyses (10%) and the reporting of effect sizes (66%) were considerably lower but increased over time. For only 4% of the articles analysis scripts were directly available, and none of the studies pre-registered hypotheses or used a pre-registered report format. There was no indication of the usage of small underpowered studies (Md(N) = 193, average effect size r = 0.30) and the p-values showed the expected right-skewed distribution without a bunching of p-values just below p = .05. Overall, adoption rates of open science principle are higher than in other fields and the journals policies were successful in fostering adherence to open science principles. A general culture of openness to reproducing analyses and replication of findings was established, which allows for a cumulative development of knowledge to which also student research can contribute.en_US
-
CitationJekel, M., Glöckner, A., Fiedler, S., Allstadt Torras, R., Dorrough, A., Mischkowski, D., … Miketta, S. (2019, March 12). The Impact of Open Science Practices on Research Methodology: A Case Study for Research in Judgment and Decision Making. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2389en
-
Persistent Identifierhttps://hdl.handle.net/20.500.12034/2021
-
Persistent Identifierhttps://doi.org/10.23668/psycharchives.2389
-
Language of contentengen_US
-
PublisherZPID (Leibniz Institute for Psychology Information)en_US
-
Is part ofOpen Science 2019, Trier, Germanyen_US
-
Dewey Decimal Classification number(s)150
-
TitleThe Impact of Open Science Practices on Research Methodology: A Case Study for Research in Judgment and Decision Makingen_US
-
DRO typeconferenceObjecten_US
-
Visible tag(s)ZPID Conferences and Workshops