Reliability Generalization Meta-Analysis of the Padua Inventory-Revised (PI-R)

Background: Obsessive–compulsive disorder (OCD) is a mental disorder characterized by the presence of obsessions, compulsions, or both. The Padua Inventory (PI) of Sanavio is one of the measurement instruments most widely used to assess obsessive-compulsive symptoms (Sanavio, 1988). A number of shorter versions of the PI can also be found in the literature. This is the case of the Padua Inventory Revised (PI-R) developed by Van Oppen, Hoekstra, and Emmelkamp (1995), which consists of 41 items and five subscales adapted to Dutch language: Impulses (7 items), Washing (10 items), Checking (7 items), Rumination (11 items) and Precision (6 items). Higher scores indicate greater severity of obsessive–compulsive symptoms. Reliability of psychological tests depends on the composition and characteristics of the samples of participants and the application context. Since reliability varies in each test administration, meta-analysis is a suitable method to statistically integrate the reliability estimates obtained in different applications of a test. Vacha-Haase (1998) coined the term reliability generalization (RG) to refer to this type of meta-analysis. Objectives: An RG meta-analysis of the empirical studies that applied the PI-R (Van Oppen et al. 1995) was carried out in order to: (a) estimate the average reliability (for the total scale and subscales); (b) examine the variability among the reliability estimates; and (c) search for characteristics of the studies (moderators) that can be statistically associated to the reliability coefficients. Method: To be included in the meta-analysis, each study had to fulfil the following criteria: (a) to be an empirical study where the PI-R, or an adaptation maintaining the 41 items, was applied to a sample of at least 10 participants; (b) to report any reliability estimate based on the study-specific sample; (c) the paper had to be written in English or Spanish; (d) samples of participant from any target population were accepted (community, clinical of subclinical populations); and (e) the paper might be published or unpublished. The search period of relevant studies covered from 1988 to September 2017 inclusive. The following databases were consulted: PROQUEST, PUBMED, and Google Scholar. In the electronic searches, the keywords “Padua Inventory” were used to be found in the full-text of the documents. Internal consistency was the type of reliability investigated in this RG meta-analysis, so that alpha coefficients reported in the primary studies were extracted. A random-effects model was assumed estimating the between-studies variance by restricted maximum likelihood (López-López, Botella, Sánchez-Meca, & Marín-Martínez, 2013; Sánchez-Meca, López-López, & López-Pina, 2013). The 95% confidence interval around each overall reliability estimate was computed with the improved method proposed by Hartung (1999). All statistical analyses were carried out with the metafor package in R (Viechtbauer, 2010). Results: The search yielded a total of 1,335 references, out of which 1,234 were removed for different reasons. The remaining 101 references were empirical studies that had applied the PI-R and out of them, 24 were included in the meta-analysis. The 24 estimates reported for the total scale yielded a mean coefficient alpha of .926 (95%CI: .913 and .937), ranging from .830 to .960. Subscales exhibited lower mean reliability coefficients than that of the total scale, with Washing yielding the largest estimates (mean = .889; 95%CI: .853 and .916), followed by Checking (mean = .879; 95%CI: .862 and .894), and Rumination (mean = .870; 95%CI: .845 and .890). Impulses (mean = .793; 95%CI: .762 and .820) and Precision (mean = .727; 95%CI: .678 and .768) were the subscales with the poorest average reliabilities. Alpha coefficients presented a large heterogeneity, with I2 Indices over 80% in all cases. The large variability exhibited by the reliability coefficients obtained in different applications of the PI-R was investigated by analyzing the influence of potential moderator variables. Concretely, the standard deviation of test scores exhibited a statistically significant relationship with coefficient alpha and with a percentage of variance accounted for of 33%. In particular, this predictor exhibited a positive relationship with alpha coefficients, so that larger coefficients alpha were obtained as the standard deviation of the scores increased. Furthermore, statistically significant differences were found when comparing the mean alpha coefficients grouped by the test version (p = .034), with a 36% of variance of variance explained, the mean reliability being larger for Turkish studies. Conclusions: Several guidelines have been proposed in the psychometric literature to assess the adequacy and relevance of reliability coefficients. In general, it is accepted that coefficients alpha must be over .70 for exploratory research, over .80 for general research purposes, and over .90 when the test is used for taking clinical decisions (Nunnally & Bernstein, 1994). Based on these guidelines, our findings demonstrated the good reliability of the PI-R total scores, both for screening and clinical purposes. The results also demonstrate how reliability depends on the application context and the composition and variability of the samples. In particular, as expected form psychometric theory, a strong positive relationship was found with the standard deviation of test scores. Another characteristics of the studies that exhibited a statistical relationship with alpha coefficients was the test version. References: Hartung, J. (1999). An alternative method for meta-analysis. Biometrical Journal, 41, 901-916. López-López, J. A., Botella, J., Sánchez-Meca, & Marín-Martínez, F. (2013). Alternatives for mixed-effects meta-regression models in the reliability generalization approach: A simulation study. Journal of Educational and Behavioral Statistics, 38, 443-469. Nunnally J. C., & Bernstein I. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill. Sanavio E. (1988). Obsessions and compulsions: The Padua Inventory. Behaviour Research and Therapy, 26, 169–177. Sánchez-Meca, J., López-López, J. A., & López-Pina, J. A. (2013). Some recommended statistical analytic practices when reliability generalization (RG) studies are conducted. British Journal of Mathematical and Statistical Psychology, 66, 402-425. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20. Van Oppen, P., Hoekstra, R.J., & Emmelkamp, P.M.G. (1995). The structure of obsessive-compulsive symptoms. Behaviour Research and Therapy, 33, 15-23. Viechtbauer, W. (2010). Conducting meta-analyses in R with the metaphor package. Journal of Statistical Software, 36, 1–48.