ABSTRACT

Visual attention is induced to shift to another person’s gaze direction. We employed a two-frame stimulus presentation procedure to investigate the effect of pre-presentation of a frontal face on this attentional shift. The first frame showed a frontal-gaze face, while the second frame showed an averted-gaze face. Reaction time (RT) to a target suddenly appearing to the left or right of the averted face image was measured. The results showed that an averted face with averted gaze induced an attentional shift in the gaze direction, while presentation of a frontal-gaze face as the first-frame stimulus resulted in two different effects on RTs. The RTs were generally shorter when the frontal face was presented irrespective of the gaze-target directional congruency, while prepresenting the frontal face for 300 ms elongated the RTs. The former might arise from a priming effect and the latter might arise from re-processing of the face from which gaze direction is estimated. We concluded that the first-frame stimulus could facilitate the general response to the target, but could not facilitate an attentional shift, in spite of the existence of an apparent motion cue from the head motion.

Keywords:Visual Attention, Gaze Direction, Reaction Time, Face Processing

1. Introduction

Our attention can be captured by another person’s direction of gaze (Friesen & Kingstone, 1998). This property of visual function can be useful for divining important information from various kinds of visual objects surrounding us and can be thought of as a form of joint attention (e.g., Bruner, 1983; Hood et al., 1998). Many experiments have been conducted in this field, and in most of these previous studies, a cuing paradigm was used to assess the locus of visual attention (e.g., Friesen & Kingstone, 1998; Hietanen, 1999; Driver et al., 2005). The experimental procedure involved the presentation of a face with averted gaze, followed by presentation of a target stimulus, typically on the left or right of the face. The task of the participants was to respond as quickly and correctly as possible to the target. The reaction time (RT) was employed as a measure to estimate attentional status. When the directions of the captured attention and the target appearance are the same, the RTs should be shorter than those when the directions are different. By comparing the RTs, the effect of captured attention can be quantitatively investigated (Posner, 1980).

However, we find non-negligible inconsistencies in measurement procedures used in previous research. For example, Hietanen (1999) presented only an averted gaze face before target presentation, while Driver et al. (1999) and Bayliss et al. (2005) presented a frontal face with or without direct gaze before the presentation of an averted-gaze face. This difference in face presentation produces totally different impressions of the stimulus’ appearance. The two-frame stimuli make dynamic impressions, i.e., presenting a frontal-gaze face before the averted-gaze face produces apparent motion, instead of the visual transition caused by the sudden appearance of the averted-gaze face in one-frame stimuli. In fact, Hietanen (1999) suggested that dynamic stimuli could induce differences in the effect of the captured attention they reported. One problem in the two-frame face presentation stimuli previously used is that the movement in the stimuli is limited to the eye image. The procedure may be useful in enhancing the effect of gaze on attentional shift. However, this may not be natural as a human behavior, because we usually turn our head to the attention direction almost simultaneously with the eye movement.

In this study we examined whether the effect of presenting a frontal face as the first stimulus had an effect on the attentional shift induced by the averted-gaze face presented second. There were two possible hypotheses. One is that the first-frame frontal face can facilitate the response to the second-frame averted-gaze face as a prime stimulus. When this is true, the RT should be shorter when the frontal face is presented than when it is not presented, irrespective of the directional congruency between the gaze and the target. Another possible hypothesis is that apparent motion created by the two-frame stimuli had some effects on the attentional shift. It is possible that the direction of apparent motion of the face itself could additionally induce attentional shift. When this is the case, the RTs when the gaze and target directions are congruent should be shorter when the two frames are presented rather than when only a one-frame face image is presented. Conversely, when the gaze and target directions are incongruent, the RTs should be longer in the two-frame condition.

2. Experiment 1

We examined the effect of a frontal face with direct gaze as the first stimulus on the attentional shift. We modulated the length of the frame. It should be noted that when the duration of the first-frame presentation was 0 ms, only the second frame (i.e., averted face with averted gaze) was presented.

2.1. Method

2.1.1. Participants

Sixteen volunteers (8 males and 8 females) participated in Experiment 1. They all had normal or correctedto-normal vision. None of them knew the four people appearing in the stimulus photos. Except for one female, the participants were not aware of the purpose of the experiment. An experimenter obtained informed consent from each participant before conducting the experiment.

2.1.2. Apparatus and Stimuli

Examples of face stimuli are shown in Figure 1. Headshots of four people (two males and two females) were taken using a digital camera (Canon IXY30s, 3648 × 2736 pixels). To create averted face images, the four people turned their face to a marker placed at the 30 degree oriented position and gazed at the 60 degree oriented marker in the rightward (or leftward) orientation from the front. All photographs were converted to grey-scale

images. The photo images were scaled so that the width of the head was approximately 8degrees of visual angle. The stimuli were presented on a 21 inch-CRT monitor (2048 × 1536, 75 Hz, View Sonic P227f). The stimuli consisted of two-frame face presentation. For example, in the first frame, a headshot with frontal gaze was presented and in the second frame, a headshot of the averted face with averted gaze was presented. The facial image was centered by placing the nose in the center of the screen. The red fixation mark (1 deg × 1 deg) was presented at the center of the screen. The size of the target stimulus (an asterisk) was 1 deg × 1 deg. The color of the target was grey with a luminance of 63 cd/m². It was presented 9.5 degrees apart (right or left) from the center of the screen. The background was white with a luminance of 95 cd/m². The viewing distance was 57.3 cm.

2.1.3. Procedure

A cuing paradigm was used in this experiment; the flow of the procedure is shown in Figure 2. A fixation point was presented all through the experiment. The trial sequence proceeded as follows. During the first 1000 ms, only the fixation point was presented. Then, a frontal headshot with frontal gaze of one of the four people was presented for one of nine durations (first-frame duration): 0 ms, 50 ms, 100 ms, 150 ms, 200 ms, 300 ms, 600 ms, 1200 ms and 1800 ms. Following this, a headshot of the person’s averted face with averted gaze oriented rightward or leftward was presented. The target stimulus was presented either in the right or left visual field with an SOA (stimulus onset asynchrony) of 100 ms or 1000 ms. The task of the participants was to respond to the target position (right or left) as correctly and quickly as possible using two corresponding buttons. The second-frame image remained to be presented until the response was obtained or 2000 ms passed from the target appearance.

There were 288 trials in total for each participant (9 conditions of first frame duration, 2 conditions of SOA, 2 conditions of consistency between the target and gaze direction, and 8 repetitions). After half a session was finished the participants took a 10-minute rest. The order of the trials was randomized for each participant. Each participant performed more than 30 trials as practice before formal data collection.

2.2. Results and Discussion

Table 1 and Figure 3 show the results. The RTs of error responses were excluded from the data analysis. We considered an extreme RT as an outlier where the deviation from the average was greater than three times the standard deviation. However, no outlier was acquired in Experiment 1. The vertical axis indicates the RTs (ms) and the horizontal axis indicates the first-frame duration. The SOAs between the second-frame stimulus and the target are indicated by the color difference of the lines.

When the SOA was 1000 ms, RTs were the longest when the first-frame duration was 0 ms. In all these conditions except for the 0 ms first-frame condition, the RTs were almost constant. However, when the SOA was 100 ms, there was a peak in the 300 ms first-frame condition and the shapes of the curves were greatly different from those when the SOA was 1000 ms. On the other hand, the directional congruency affected the RTs (i.e., RTs were shorter for the congruent conditions) only in the 100 ms SOA conditions.

We conducted a three-way ANOVA. The three factors were first-frame duration, SOA between the second stimulus and the target, and the congruency of the target and the gaze directions. There were significant main

effects in all three factors (duration, [F(8,120) = 5.9986, p < 0.0001]; SOA, [F(1,15) = 139.4413, p < 0.0001]; congruency, [F(1,15) = 20.4883, p = 0.0004]). There were significant interactions between the first-frame duration and SOA ([F(8,120) = 6.731, p < 0.0001]) and between SOA and congruency ([F(1,15) = 36.2412, p < 0.0001]). Simple main effects tests revealed simple main effects of first-frame duration in each SOA condition (100 ms, p < 0.0001; 1000 ms, p = 0.0002). Simple main effects of SOA were found, except for the 1800 ms first-frame condition (0 ms, p = 0.0001; 50 ms, p = 0.0028; 100 ms, p < 0.0001; 150 ms, p = 0.0134; 200 ms, p = 0.0004; 300 ms, p < 0.0001; 600 ms, p < 0.0001; 1200 ms, p < 0.0001). Simple main effects of SOA were also found in each congruency condition (Congruent, p < 0.0001; Incongruent, p < 0.0001). A significant simple main effect of congruency was only found when the SOA was 100 ms (p < 0.0001). However, multiple comparisons (Holm’s Sequentially Rejective Bonferroni Procedure) revealed that when the SOA was 100 ms, there was no significant difference in RTs between the 0 ms and 300 ms first-frame duration conditions.

First we should note that in the 0 ms first-frame duration condition, i.e., no first-frame condition, the RTs were longest. This means that participants’ responses were facilitated by the presentation of the first-frame stimulus. However, this effect was obtained both when the direction suggested by the second-frame stimulus and the target position were congruent and when they were incongruent. Thus, although presenting the first-frame stimulus could attract attention, its effect seems to be the facilitation of visual processing or button-pressing action, not to emphasize the attention shift by the averted gaze/face. This may indicate that the first-frame presentation acted as a prime stimulus and did not enhance attentional shift by additional motion information. Thus, we could reject the hypothesis that apparent motion induces an additional attentional-shift effect.

The main effect of SOA indicates that the RTs were shorter in the 1000 ms condition than in the 100 ms condition. This result indicates that participants were ready to respond to the target well before the target appearance in the 1000 ms SOA condition. The difference between congruent and incongruent conditions was not found here because the effect of attentional shift induced by gaze direction may fade away in the long SOA conditions (Friesen & Kingstone, 1998). However, even in the 1000 ms SOA condition, the effect of the first-frame presentation noted above clearly appeared. On the other hand, there was no significant difference in the effects of SOA when the first-frame duration was 1800 ms. This result indicates that the long first-frame duration substituted for the second-frame presentation duration. One possible cause of this phenomenon is that the sudden appearance of a face image draws attention to the face and that longer presentation of the image gradually releases the attraction. We believe that there are several (as opposed to a single) attention control factors in this simple stimulus presentation.

The RTs acquired under the 0 ms and 300 ms first-frame duration conditions were approximately the same when the SOA was 100 ms. This result suggests that the 300 ms continuous presentation of the first-frame stimulus reset the response facilitation effect of the first-frame presentation. When the first-frame duration was between 300 ms and 1800 ms, the shrinkage of the RTs was shallower than that arose when the first-frame duration was between 0 ms and 200 ms. This result may indicate that the mechanisms of the shrinkage of RTs were different between the short (less than 200 ms) and long (more than 300 ms) first-frame duration conditions. Although the large shrinkage of RTs by the first-frame presentation was also obtained when the SOA was 1000 ms, the reset of the facilitation effect noted above was not found. We guessed that the long RTs under the 300 ms first-frame presentation condition had some relation to the processing of the first-frame face image. This point is further investigated in Experiment 2.

The results of Experiment 1 showed that the presentation of the first-frame stimulus itself was effective in reducing the RTs to the target. However, the directional information of apparent motion seems to be trivial or even neglected. We tested the hypothesis by employing grey silhouettes as the first-frame stimuli. Although the silhouette could suggest the existence of the person, there was no information about the eyes or the face, and thus no explicit facial image in the first-frame stimulus or motion information. If our hypothesis was valid, the facilitation effect on RTs by the existence of the first-frame could also be obtained when we use the silhouette. On the other hand, if the effect of the 300 ms first-frame duration were something to do with face processing, the effect would disappear when the silhouette was presented as the first-frame stimulus.

3. Experiment 2

We examined whether the same effect of the first-frame stimulus could be obtained without using the frontal face with straight gaze as the first-frame stimulus. Instead, we used the silhouette of a face (i.e., without face or gaze information) as the first-frame stimulus. In Experiment 2, we used a 100 ms SOA with varied first-frame durations.

3.1. Method

3.1.1. Participants

Eighteen volunteers (8 males and 10 females) participated in Experiment 2. They all had normal or corrected-to-normal vision. None of the participants knew the four people appearing in the stimulus photos. Except for one male and three female, they had participated in Experiment 1. Except for one male and one female, the participants were not aware of the purpose of the experiment. An experimenter obtained informed consent from each participant before conducting the experiment.

3.1.2. Apparatus and Stimuli

We used the same apparatus as used in Experiment 1. We employed silhouettes of four people for the first-frame stimulus as shown in Figure 4. Each silhouette was painted in grey in the mean luminance of the face image. There was no information about the positions of the eyes, nose, mouth, etc.

3.1.3. Procedure

The procedure was the same as that used in Experiment 1, except that the SOA was fixed at 100 ms. In total, 144 trials were conducted for each participant (9 conditions in the first-frame duration, 2 conditions of congruency, and 8 repetitions for each condition). Each participant performed more than 30 trials as practice before formal data collection.

3.2. Results and Discussion

The results are shown in Table 2 and Figure 5. Some RTs were excluded from the data analysis with the same

criterion as used in Experiment 1. When the first-frame duration was 0 ms, i.e., without the first-frame stimulus, RTs were the longest. In other words, the existence of the first-frame stimulus reduced RTs. This result is consistent with the results in Experiment 1. However, the RTs monotonically became shorter when the first frame was presented longer. No elongation at 300 ms first-frame duration was found.

We conducted a two-way ANOVA. The factors were the first-frame duration and the congruency between the directions of gaze and the target. There were significant main effects of both factors [duration, F(8,136) = 28.9545, p < 0.0001; congruency, F(1,17) = 45.3055, p < 0.0001]. There was no significant interaction between the two [F(8,136) = 0.3737, p = 0.933]. Multiple comparisons revealed that RTs were significantly longer in the 0 ms first-frame duration condition than in the other first-frame duration conditions (50 ms, p = 0.0045; 100 ms, p = 0.0006; 150 ms, p = 0.0001; 200 ms, p = 0.0001; 300 ms, p = 0.0001; 600 ms, p < 0.0001; 1200 ms, p < 0.0001;1800 ms, p < 0.0001).

One of the purposes of Experiment 2 was to investigate the effect of presentation of the first stimulus itself rather than the information from the face in the first-frame stimulus. The results from the 100 ms second-frame averted-gaze conditions in Experiment 1 and the results from Experiment 2 were similar when the first-frame duration was shorter than 200 ms. This result may indicate that the first-frame stimulus could capture attention and facilitate the responses as a prime stimulus even without face information in the image.

The other purpose of Experiment 2 was to investigate the effect of face image in the first-frame stimulus. In Experiment 1, a 300 ms presentation of the first-frame frontal-face stimulus reset the facilitation effect noted above. However, in Experiment 2, this phenomenon was not confirmed. The only difference between the corresponding conditions in Experiments 1 and 2 was that the first-frame stimulus in Experiment 2 was painted in uniform grey. The reset of the facilitation effect in Experiment 1 could be due to presenting the frontal face information.

There are two possibilities for the reset of the facilitation effect achieved by presenting the frontal face. First, it is plausible that finishing the processing of the frontal face at 300 ms released visual attention, resulting in the delay of the response to the second-frame stimulus. This may correspond to the “attentional blink” phenomenon (Raymond, Shapiro, & Arnell, 1992). Second, it is possible that head orientation processing was re-conducted by the appearance of a new averted face image after finishing the processing of the frontal face. It has been reported that detection of gaze direction is determined in relation to head orientation (Hietanen, 1999). According to this work, re-determination of head orientation and eye-gaze direction relative to the head may be required after the appearance of the new averted face image. This could delay the response to the second-frame stimulus. These two possibilities are totally different from the view of the mechanisms involved, although both could potentially delay the response. In Experiment 3, we investigated which was the case.

We used the same frontal faces from Experiment 1 as the first-frame stimulus and also presented the second-frame frontal face stimulus with a change in averted gaze direction. If visual attention was released after the processing of the first-frame stimulus at 300 ms after the onset of the first-frame stimulus in Experiment 1, the same effect should be obtained in Experiment 3. On the contrary, if the phenomenon that appeared at 300 ms was due to the re-determination of the gaze direction caused by the presentation of the averted face as the second-frame stimulus in Experiment 1, the reset of the facilitation effect at 300 ms would not be obtained in Experiment 3, because there was no change of head orientation in the condition used in Experiment 3.

4. Experiment 3

We investigated the reason for the re-set of the facilitation effect. In this experiment we used the frontal faces from Experiment 1, and presented the face with the gaze direction averted and keeping the head direction fixed.

4.1. Method

4.1.1. Participants

Twenty volunteers (10 males and 10 females) participated in Experiment 3. They all had normal or corrected-to-normal vision. None of the participants knew the four people appearing in the stimulus photos. Except for two males, all had participated in Experiment 2. Except for one male and one female, the participants were not aware of the purpose of the experiment. An experimenter obtained informed consent from each participant before conducting the experiment.

4.1.2. Apparatus and Stimuli

Except for the difference in the second-frame face stimuli, the procedure was the same as that in Experiments 1 and 2. In the second-frame face stimulus, the head was fixed in the frontal orientation and only the gaze direction deviated by 30 degrees to the left or right. The examples of the second-frame stimuli are shown in Figure 6.

4.1.3. Procedure

There were 144 trials in total (9 duration conditions in the first-frame stimulus, 2 conditions of congruency, and 8 repetitions). Each participant performed more than 30 trials as practice before formal data collection.

4.2. Results and Discussion

The results are shown in Table 3 and Figure 7. Some RTs were excluded from the data analysis with the same criterion as used in Experiment 1. The congruent and incongruent conditions are represented by filled or open circles, respectively. The vertical axis indicates the RTs (ms) and the horizontal axis indicates the first stimulus duration. The shapes of the curves were similar to the results of Experiment 2 (superimposed in Figure 7).

We conducted a two-way ANOVA. The two factors were first-frame duration and congruency between gaze

direction and target position. There were two significant main effects (duration, [F(8,152) = 22.6031, p < 0.0001]; congruency, [F(1,19) = 54.4705, p < 0.0001]). The interaction was not significant. Multiple comparisons revealed that, in the 0 ms first-frame condition, RTs were significantly shorter than those in the other firstframe duration conditions (50 ms, p = 0.0319; 100 ms, p = 0.0001; 150 ms, p < 0.0001; 200 ms, p = 0.0005; 300 ms, p = 0.0007; 600 ms, p < 0.0001; 1200 ms, p < 0.0001; 1800 ms, p < 0.0001).

The results clearly showed that there was no reset of the facilitation effect in the 300 ms first-frame duration condition in this experiment. This suggests that the reset of the facilitation effect obtained in Experiment 1 is related to the re-determination of the gaze direction based on the averted face that appeared just after the face processing of the first-frame stimulus was completed. In fact, the direction of the head is important for the detection of gaze direction (Cline, 1967; Langton, 2000). The attentional shift based on the gaze direction would be referenced by the head direction (Hietanen, 1999). Thus, the need to re-process the second-frame stimulus to account for a change in head direction is very natural.

5. General Discussion

In Experiment 1, we confirmed that a frontal face as a first-frame stimulus facilitated RTs to the target. That facilitation effect was reset by the relatively shorter first-frame duration, i.e., 300 ms, in the shorter SOA conditions. This was true both for the congruent and incongruent conditions. In Experiment 2, we examined whether this facilitation effect was caused by the presentation of the frontal face or by the existence of the first-frame stimulus itself. The results showed that even a silhouette stimulus caused the effect. This result indicated that the existence of the first-frame stimulus itself was important, rather than the facial information. However, a reset of the facilitation effect was not obtained in Experiment 2. It was suggested that the facial information was important for the reset of the facilitation effect. In Experiment 3, the head orientation was kept frontal to the observer in both the first and second frame stimuli, and in the second stimulus only the gaze directions were averted. In this experiment, the reset of the facilitation effect was not obtained. Therefore, this phenomenon is obtained only when the head orientation is changed from a frontal face after some duration.

These results in sum suggested that the addition of the first-frame stimulus could capture the attention for visual information. Based on this effect (similar to a priming effect), RTs were shortened. If a frontal face was used as the first-frame stimulus, the processing of the face and attentional concentration may start simultaneously. Seemingly, the attention captured by the first-frame stimulus was kept on purpose even when the first-frame presentation duration was longer, because of expectation that the visual stimulus would change. However, when there is a change of head orientation between the first and second stimuli, re-processing is required for determination of the gaze direction. For this reason, the facilitation effect of RT is reset when the second-frame averted face is presented at a certain timing, possibly at the timing of the release of attention after the completion of the first-frame stimulus processing. When presenting the first-frame frontal face longer, we speculate that attention is re-focused on the stimulus via the top-down pathway, based on the participants’ awareness that the second stimulus would come.

If this is the case, why was the reset of the facilitation effect obtained only when the head orientation changed, and not when the gaze direction was changed? This might be related to the fact that the head orientation can function as a reference frame for the gaze direction (Hietanen, 1999). When only the gaze directions were changed, the reference frame was stable, thus observers could determine the gaze direction from the local information. On the contrary, when the head orientation changed, the reference frame changed too. Then, the observer had to detect both the head orientation and the gaze direction.

In this study, we examined the effect of presenting a frontal face before an averted-gaze stimulus in the attentional shift paradigm. Although we expected that the apparent motion would facilitate RTs to the directionally congruent target, the results showed that even when the directions of the gaze and target were inconsistent the facilitation effect was obtained, thus this hypothesis was rejected. By presenting the averted gaze stimulus during the processing time of the first-frame stimulus, the RTs were elongated because of the re-processing of head orientation. In conclusion, the first-frame stimulus can facilitate the general response to the target, but cannot facilitate the attentional shift.

Acknowledgements

This study was partly supported by KAKENHI (23243076, 25540170).

References

NOTES

^*Corresponding author.