Search logs, argumentative essays, and performance profiles: Positioning multiple source use in a classroom context

Objectives: This study investigated the relation between undergraduates’ online search behaviors in a multiple source use (MSU) task and their argumentative essay writing. An innovative search log was employed to record students’ decision-making during source selection, which was examined in relation to essay quality and self-identified performance profiles. Theoretical Framework: The Integrated Framework for Multiple Texts (List & Alexander, 2019) describes three phases for tasks involving multiple sources: preparation, execution, and production. In the IF- MT, strategies implemented in the execution stage are presumed to influence outcomes in the production stage. The behavioral and (meta)cognitive strategies enacted for searching, selecting, and processing information sources in the execution stage influence the cognitive and affective outcomes externalized in the written products in the production stage. The execution of information search strategies can differ depending on the profile students adopt in the preparation stage (List & Alexander, 2017). However, little research has sought to link students’ decision- making in source selection to their written products, nor have such search processes been examined in relation to students’ adopted profiles. To address this gap, we developed a method to capture students’ processing in both the execution and production phases to investigate the interrelations between these phases. Method: Undergraduates (n = 132) completed an online MSU task as an assignment in an educational psychology course. Participants performed open searches and selected at least four sources to summarize for the purpose of writing argumentative essays on a controversial topic, students’ over-reliance on technology. During the search, they logged the search terms they used; number of sources scanned, opened, and read; and reasons for including or excluding each read source. Unlike automatically generated log files, these search logs were students’ records of their decision making throughout the search process. Students were also required to write summaries of each source they read in their class journals—a routine activity. They used these summaries to compose their argumentative essays. Essays were scored based on components of argumentation quality (claim, counterargument, sources, justification, and writing quality) and integration (critical analysis, synthesis, and overall cohesion), each on a 0–2 scale (interrater reliability [ICC = .96], maximum score = 16). In a post-task questionnaire, students’ selected one of four profile descriptions to represent their performance stance (see Table 3). Results: Descriptive statistics of the rich data collected via the search logs allowed for important insights into students’ online search behaviors. Specifically, students conduct on average few searches (3.9), read 7.2 sources, and summarized 4.3 sources. Most students used biased, often negative search terms for their searches, and most used less sophisticated search phrases rather than concise search terms. Further, as reasons for reading a source, students mostly reported making epistemic judgments (i.e., relevance, credibility, viewpoints) and to a lesser extent non-epistemic judgments (i.e., order on Google page, length, interestingness). However, in selecting a source for inclusion in their essay, both epistemic and non-epistemic judgments were made. Future analyses will subject these rich search log data, as well as the essay outcome data, to cluster analysis. As such, we will explore whether meaningful typifications of groups of students with specific MSU orientations can be made. Significance: Writing was integral to this MSU task in terms of the search log, written summaries, and argumentative essays. This study provides insight into multiple document use within an ecologically valid context, which rarely occurs. Moreover, the rich descriptive data of students’ source use in an unconstrained search task are novel contributions to the literature. Further, attempts to typify students’ MSU behavior in the preparation and production stage may be a crucial first step toward an intervention.