3. Experiment 2
In the previous experiment, we found no evidence of any behavioral differences between the Refresh and Remove instruction groups. Thus, Experiment 1 offered no support for, and in fact some support against, a functional distinction between the refresh and remove processes, although that task design did not allow us to look for certain potential patterns of differences, such as differential effects on irrelevant items during the main task. Specifically, we did not ever probe irrelevant items during the main task, so we could not observe whether there was an RT benefit for relevant over irrelevant items during the initial task, or whether the magnitude of such a benefit might differ between instruction groups. Thus, Experiment 2 was conducted in the same manner as Experiment 1, except for the inclusion of invalid probes (i.e., occasional probes of irrelevant words) in the LDT (see Figure 1a). Adding these invalid probes allowed us to examine the fate of the irrelevant words during the main task and further illuminate any potential differences between instruction groups. We predicted faster RTs to relevant than irrelevant items in both instruction groups, but that any difference between the refresh and removal processes would manifest as an interaction. If there were such a difference between the refresh and removal processes, presumably RTs to irrelevant items in the Remove instruction group would be particularly slowed (versus their simply not being refreshed in the Refresh instruction group), which would drive the hypothesized interaction.
3.1. Method
3.1.1. Participants
Sixty-nine undergraduate students (N = 54 female), between the ages of 18 and 26 (M = 19.74, SD = 1.66), from the University of Nebraska-Lincoln participated in a one-hour experiment for course credit. All participants provided informed consent prior to the study, and procedures were approved by the Institutional Review Board at the University of Nebraska- Lincoln. In Experiment 2, each participant was randomly assigned to either the Refresh or Remove instruction group (N = 30 and 39, respectively). Participants that had already participated in Experiment 1 were ineligible to take part in Experiment 2. As in Experiment 1, participants were not informed that there were different instruction groups until after the experiment had ended; each participant only received instructions for his or her own group (Refresh or Remove). An additional 18 participants were excluded from the analyses, four due to poor compliance with task instructions (2 Refresh group, 2 Remove group) and 14 due to LDT accuracy below 75% correct (5 Refresh group, 9 Remove group) in any of the three probe type conditions. Exclusions based on accuracy, with the exception of one participant, were due to accuracy specifically on invalid trials. Accuracies for remaining participants were near ceiling (see 3.2. Results).
3.1.2. Procedure and Tasks
Experiment 1 was fully counterbalanced for all of the key trial variables in order to control for potential confounds, rather than relying on randomization to produce equivalence between conditions. Given the introduction of invalid trials and the resulting increase in the number of permutations of stimuli, the roles and positions of specific words within Experiment 2 were simply randomized rather than counterbalanced. The same constraints on trial presentation remained; no more than three consecutive trials having the same cue position or probe type (valid, invalid, or non-word). Participant instructions were changed slightly to accommodate the invalid trials. Other researchers (Lewis-Peacock et al., 2018) have noted that the presence of invalid probes may complicate the study of WM removal, as invalid trials might negate any incentive to remove. In order to address the presence of invalid probes and incentivize participants to comply with removal cues, we acknowledged that we would rarely probe the irrelevant words; however, since we were measuring reaction time, we informed participants that they would still be faster overall if they followed the cue instructions. Thus, as it was framed to participants, in service of the overall goal of low average reaction times, it was most effective to remove as cued, given the higher percentage of valid and nonword probe trials. In any case, if the existence of invalid trials indeed de-incentivized participants from complying with removal cues, that should become apparent in the data as a reduction in LTM differences between relevant and irrelevant items. As we did ultimately find a preserved LTM benefit for relevant items (see Results), in this case we do not find reason to suspect that the presence of invalid trials presented an issue for participants' compliance with the instructions.
In the first experiment, LDT word probes had an equal probability of being the relevant word or a non-word; the irrelevant words were never tested. The probability of encountering each probe type in the second experiment was 40% non-word, 40% valid, 20% invalid. While this would change the overall ratio of word to non-word probes from equal (as in Experiment 1) to 60% word, we felt that this change would be unnoticeable to participants. This afforded us the opportunity to test a sufficient number of invalid trials while keeping the proportion of invalid to valid trials low enough that invalid probes would not influence participants' strategy. Total trials were decreased from 108 (Experiment 1) to 90 for Experiment 2. Accordingly, the LTM task in Experiment 2 consisted of all 270 words encountered in the main task and 180 foils, keeping the ratio of previously seen words to foils the same as in Experiment 1.
3.2. Results
Response time to the main task LDT probe was tested with a 2 (instruction group: Refresh or Remove) × 3 (probe type: valid, invalid, non-word) mixed ANOVA (see Figure 3a). RTs shorter than 200ms and/or more than three standard deviations away from the participant's mean RT were excluded from the analysis, as in Experiment 1. RTs corresponding to incorrect LDT responses were also excluded. In total, 6.02% of all trials from analyzed participants were excluded from analysis. As noted above, participants with < 75% accuracy in one or more probe type conditions were excluded from analysis; accuracy was high in the remaining participants (95.53% on average). There was a main effect of probe type, F(1.828, 122.465) = 102.583, p < .001, Greenhouse-Geisser corrected, with faster responses to valid probes (M = 548.379, SD = 116.661) than to non-word probes (M = 663.781, SD = 135.021), p < .001, and invalid probes (M = 679.576, SD = 124.733), p < .001. Responses to non-word probes were not significantly different than to invalid probes, p = .300. The main effect of instruction group was not significant, F(1, 67) = 3.727, p = .058 (Remove group: M = 660.763, SD = 111.773; Refresh group: M = 607.359, SD = 115.525). There was no significant probe type by instruction group interaction, F(1.829, 122.465) = 0.821, p = .433.
LTM confidence ratings were tested with a 2 (instruction group: Refresh or Remove) × 8 (word role) mixed ANOVA (see Figure 3b). Word roles were the same as in Experiment 1, with the addition of three new roles corresponding to the words that had been seen on invalid probe trials (wherein the probe was one of the irrelevant words; thus, the new word roles for invalid probe trials were the relevant word, the irrelevant word that was probed, and the irrelevant word that was not probed). LTM responses faster than 200ms were deemed to be accidental keypresses, resulting in 0.03% of total responses excluded from analysis.
For the LTM confidence ratings, there was a main effect of word role, F(4.430, 296.799) = 118.537, p < .001, Greenhouse-Geisser corrected. There was no main effect of instruction group (Refresh versus Remove), F(1, 67) = 1.454, p = .232, and no group by word role interaction, F(4.430, 296.799) = 0.538, p = .726, Greenhouse-Geisser corrected. A planned comparison testing the difference in confidence ratings between relevant (M = 2.651, SD = 0.420) and irrelevant (M = 2.448, SD = 0.365) words from nonword-probe trials was significant, t(68) = 6.271, p < .001, Cohen's d = 0.755. This replicated the pattern seen in Experiment 1, and again was consistent with both traditional directed forgetting and traditional refreshing LTM effects. No attenuation of the effects was observed in this experiment, suggesting that the inclusion of invalid probe trials in the main task did not generally cause non-compliance with the remove cue instructions. The addition of invalid probe trials in Experiment 2 allowed a second test of the relevant versus irrelevant words that were not subsequently probed, which revealed that relevant, unprobed words from invalid-probe trials (M = 2.682, SD = 0.485) were rated higher than irrelevant, unprobed words from invalid-probe trials (M = 2.496, SD = 0.467), t(68) = 3.959, p = .005, Cohen's d = 0.477, thus paralleling the effects seen on nonword-probe trials.
As in the previous experiment, further interpretation of the results was explored with Bayesian analyses. BANOVAs of main task RT and LTM both had broadly similar patterns of results to Experiment 1. For main task RT (see Table 3), the best model did include both a main effect of probe type (valid, invalid, nonword) and a main effect of instruction group (BF10 = 4.422 × 1024 507 ; extreme evidence favoring this model over the null), although this model's likelihood was only slightly higher than the model including only a main effect of probe type (BF = 1.455). This is consistent with the non-significant trend for a main effect of instruction group (p = .058) we saw using NHST; thus, we cannot rule out the possibility of a main effect of group in Experiment 2, but the evidence is not conclusive either way. However, the model including an interaction of instruction group and probe type (as well as both main effects) had lower likelihood than either the model including both main effects (BF = 6.429) or the model including only a main effect of probe type (BF = 4.418). Thus, the magnitude of the evidence against an interaction between probe type and instruction group was similar to that found in Experiment 1.
An LTM task BANOVA (see Table 4) found that the best model was a main effect model of word role alone (BF10 = 5.668 × 1097 518 ; overwhelming support versus the null model), and including instruction group in the model penalized it by a factor of BF = 1.730. Adding another term for an interaction of instruction group and word role further lowered the likelihood of the model (BF = 93.808) relative to the model including both main effects. Thus, the simplest word- role-only model was favored by a factor of 1.7 over the model including an effect of group, and by a factor of 162.288 (1.730 × 93.808) over the model that also included an interaction. Thus, the Bayesian analyses suggest that the lack of a significant interaction found with NHST should be interpreted as extremely strong support against the existence of such an interaction.
A final pair of mixed ANOVAs (main task RT, LTM confidence) were performed to directly compare the results of Experiments 1 and 2. For these analyses, only valid-probe and nonword-probe trials were considered, as there were no invalid-probe trials in Experiment 1. For main task RT, the three-way interaction between probe type, Refresh/Remove instruction group, and experiment was non-significant, F(1, 137) = 0.320, p = .572. For LTM ratings, the three- way interaction of word role, Refresh/Remove instruction group, and experiment was also non- significant, F(2.472, 338.635) = 0.380, p = .728, Greenhouse-Geisser corrected. Taken together, these results suggest that none of the changes between Experiments 1 and 2 substantively affected the results for any of the conditions shared between the two experiments, and specifically that the inclusion of invalid probes in Experiment 2 did not alter the way in which participants approached the task.
3.3. Discussion
In the main task, as in Experiment 1, participants were faster to respond to valid probes than to nonword probes. Response times to invalid probes were slower than to valid probes, reflecting the expected directed forgetting effect (Remove instruction group) or refresh advantage (Refresh instruction group). Response times were slower overall in Experiment 2 than Experiment 1 (main effect of experiment: p < .001), suggesting that the inclusion of invalid probes added some degree of difficulty or ambiguity to the task overall.