Measuring the Human, Not Just the Team: Validity-Centered Conversational Psychometrics for Epistemic Agency

Evaluation of human–AI systems remains structurally one-sided: models and team outcomes are measured extensively, while the human epistemic work that produces those outcomes is rarely measured at all. We propose validity-centered conversational psychometrics, a framework for treating dialogue and action traces as evidence about human conduct without treating them as transparent windows into cognition. Its central rule is no opportunity, no inference: a behavior is interpretable only when the task and the model afforded a meaningful chance to frame the problem, scrutinize evidence and omissions, revise under counter-evidence, or regulate delegation. We distinguish three inferential targets—conduct in a specific interaction, human–model coupling, and portable person-level epistemic agency—the last requiring cross-task, cross-model, and AI-absent transfer evidence. We organize the construct as three general epistemic processes plus an AI-specific delegation-governance layer, and outline an evidence-centered design with controlled observation opportunities (including error-free controls that separate scrutiny from indiscriminate skepticism), explicit non-observation states, plural human reference coding, and versioned automated extraction that never serves as the sole criterion. Validity is organized as a chain of inferences—opportunity, scoring, generalization, attribution, extrapolation, and use—rather than as a checklist of thresholds. A deployment failure case shows why outcome disagreement alone cannot identify who performed the epistemic work. We close with governance constraints for formative, contestable, and non-punitive use, including an audit of opportunity fairness: whether people of equal capacity were given equal chances to show it.