ABSTRACT

In order to measure real life prospective memory (PM) ability in laboratory setting, previous studies have used photograph-based, video-based and virtual reality (VR)-based to simulate real life environment. In this study, we used the Functional Near-Infrared Spectroscopy (fNIRS) to record participants’ brain activities when they are performing PM tasks in immersive VR environment. 10 participants were asked to perform PM tasks in a virtual environment while wearing a VR headset and an fNIRS device. By comparing the hemodynamic changes of rostral prefrontal cortex (BA10) of the PM component and the ongoing component, the result shows that VR PM task can induce the activation of BA10. In addition, by making use of the fNIRS technique, this study provides a feasible way to measure BA10 activity in simulated real life environment using immersive VR device.

Keywords:

Prospective Memory, Virtual Reality, BA10, fNIRS

1. Introduction

Taking medicine after a meal or meeting with old friends at a certain time requires the use of prospective memory (PM). PM refers to a form of memory that defines one’s ability to remember to take a planned action at a particular time in the future. As a critical ability frequently used in our daily lives, PM has attracted attentions in the field of memory and neuroscience [1] .

Advancements in neuroimaging technology, including Electroencephalography (EEG; [2] ), functional Magnetic Resonance Imaging (fMRI; [3] ) and functional near-infrared spectroscopy (fNIRS) have been widely used in memory research studies, including PM related studies. However, a number of PM studies that make use of fNIRS are still quite limited. In 1998, Okuda [4] published a PM neuroimaging study that made use of a slide-based PM task in which participants were asked to press a key when a specific word was shown. He concluded that BA10 was connected to the performance of PM tasks. In 2003, Burgess [5] designed three different tasks to investigate the role of BA10 in PM. He used positron emission tomography (PET) to monitor regional cerebral blood flow changes in Brodmann’s area (BA) 10 during the experiment and the result showed that activation at BA10 correlates with PM performance. Further, BA10 was related to the maintenance cognitive processes, and after the PM cue was detected, that is a delay period of the BA10 activation. His findings were later reproduced in other studies [6] [7] . Several fMRI studies also link prefrontal lobe activity directly to PM task performance, and shown that each time participants respond appropriately to a PM cue, the cognitive processing resources would reallocation [8] [9] [10] . As a result, for the current study, we aim to further investigate this relationship in an immersive virtual reality environment.

Slide-based task is a common method often used when evaluating PM performance [11] . However, researchers have pointed out that as a basic laboratory psychological paradigm, such approach cannot reflect all the PM processes in real-world conditions [12] [13] [14] [15] . In order to measure the PM performance of real world conditions in laboratory settings, Titov and Knight [12] used video-based technology to design the Prospective Remembering Video Procedure (PRVP) for assessing PM performance by stimulating everyday memory-based activities with video recordings. Knight found that PRVP task performance varied depending on the task environment. In his later study, Knight et al. [16] linked a set of 1,200 photographs to build a platform that simulated a shopping street environment. Knight found that not only age impacted task performance, the task environment also has an impact on performance. He also pointed out that using computer-based environments to measure functional cognitive deficits should be encouraged. The advancement in 3D graphics and virtual reality technology encourage the use of computer-generated virtual environment to measure PM performance in simulated real-life conditions. Okahashi et al. [17] developed a virtual shopping test (VST) task with 3D modeling technology to evaluate patient’s everyday memory performance. The VST task has shops on both sides of a Japanese style shopping street virtual environment. Participants are required to make purchase and perform several sets of actions. Mitrovic et al. [18] used a 3D virtual model-based task to try to improve and measure PM performance. The task is a virtual house with household objects and a garden. Participants perform tasks and interact with objects. To enhance PM performance in senior subjects, Debarnot et al. [19] used the theta burst stimulation (TBS) technique to stimulate the BA10 and the behavioral effects after the stimulation were measured by a 3D model-based task. In this study, Debarnot designed a virtual city environment. The city had one road and was connected to 12 scenes. The environment included common city objects, such as buildings and stores. Participants were asked to stop walking at the appropriate time or place, and tell the experimenter what action they had to do.

In recent years, significant advancement in immersive Virtual Reality (VR) technology encouraged researchers to adopt the technology in many studies, especially in the field of psychology and engineering. Immersive VR refers to the presentation of a virtual environment that simulates users’ real-world surroundings such that they are able to fully engage with the artificial environment close to reality. Unlike traditional VR technology, immersive VR generally requires the use of a head-mounted display in order to create a life-like experience for the user. To evaluate PM performance using immersive VR technology, Dong et al. [20] designed a virtual environment shopping task and compared the result against a traditional slide-based PM task. The result suggested that the immersive VR task was able to reliably better assess real-life PM ability. In order to discover the potential relevance regarding VR PM task and PM performance, and to further investigate the cognitive processes of VR PM task, we developed a methodology that involves neuroimaging technique to explore the neural correlation of VR PM task.

However, there are several limitations and problems in most neuroimaging technology such as EEG and fMRI when they are used in an immersive VR experiment. Immersive VR device is usually worn on a participant’s head and the head band covers the top and the two sides of head. This prevents the participant from wearing an EEG cap and fitting the scalp electrodes accurately. In addition, the EEG cap greatly limits the head and body movement of the participant. In fact, such movement also interferes with the EEG signal. Using fMRI to study the neural correlation of immersive VR PM task also has several problems: 1) The fMRI machine is large and expensive, and it increases the cost of the experiment; 2) The noise of the fMRI machine may affect the performance of the participant, which has been shown by Knight’s study [16] ; 3) It is likely that the electro-magnetic field will interfere with the VR equipment. Past studies used a mirror to reflect the image of a monitor placed outside of fMRI machine to address this problem [21] . However, such workaround cannot provide a true immersive VR experience to the participants.

On the other hand, using fNIRS can address the above problems. fNIRS has been widely used to measure local brain activation and changes in hemoglobin oxygenation state. The technique is non-invasive, portable and inexpensive, and it can measure cerebral blood flow in real-time. Compared to EEG, fNIRS allows the participant to move his/her head and body easier. Previous study also used fNIRS for outdoor experiments [22] . Since fNIRS is generally worn on the forehead to measure the activities in the prefrontal cortex, it leaves enough space for the immersive VR headset to be worn simultaneously. In addition, fNIRS can measure BA10 activation, which has been shown to link with PM performance [5] . In our study, we constructed a 3D virtual environment with an immersive VR device to trigger PM function in daily life condition. The system was designed to mimic a typical shopping experience in a virtual shopping area. The VR task also contains ongoing and PM components, and conforms to the task design request [5] [23] . The fNIRS system we used to monitor prefrontal cortex activity consists of a headset and a processing box that is connected to a computer. The aim of this study was to investigate prefrontal cortex function during a PM task in an immersive VR environment.

2. Materials & Methods

2.1. Participants

11 healthy young participants were recruited from Kobe University (10 males, mean age: 25.4, age range: 23-34). The forehead of one of the participants was too narrow and the fNIRS failed to obtain any measurements in 8 channels, thus his data was excluded from the experiment. All participants wrote with their right hands, and had normal color vision, corrected-to-normal or normal vision. None of the participants was taking medication or had psychiatric disorders. The informed consent agreement was signed by all participants before the experiment started.

2.2. VR PM Task and Experimental Procedure

All participants performed VR task while wearing the fNIRS device during the entire experiment. The experiment took around 40 minutes (5 minutes for training task, 10 seconds for fNIRS baseline, and 35 minutes for VR task) and was completed in the same day. Figure 1 shows the timeline of the experiment. Participants were told that they can stop the experiment at any time for the rest. Before the VR task started, each participant was required to perform a training task to ensure that they could follow the instructions and perform the task in a stabilized emotional condition. Participants were using joystick to move forward or backward. Turning around in VR task required natural body rotation on the chair. In VR task, VR environment was composed of a shopping street with 12 shops, two special action points and an exit. By using the joystick and making body rotation, participants moved in the VR shopping street, buying items and taking actions.

To reduce light disturbance to the fNIRS probes and the head mounted display, the light in the room was turned off during the task. The laboratory room was kept quiet, and the participant was seated on a comfortable swivel chair. The VR system included an immersive VR device (Oculus Rift Development Kit 2) and a computer with a desktop monitor. The desktop monitor allowed the re-

searcher to observe the participant’s movement in the VR environment and the progress of the task. The participant used a controller to move forward and backward. Turning around in the VR environment required natural body rotation on the swivel chair. The VR environment was developed with Unity 5.0.

At the beginning of the experiment, the researcher put the fNIRS device on the participant and calibrated it according to the manufacturer’s instructions. In order to minimize interference, a soft headband was tied to the participant’s head to further secure the fNIRS device firmly in the correct position (Figure 2).

The participant was then asked to complete a training task (Figure 1). The training task contained the same basic components as the formal task but only had two training shops and one special action point. In the formal VR task, it had twelve shops and two special action points. The participant was required to buy three items and take two special actions in the VR environment.

At the beginning of each task, the participant was shown a shopping list on the screen. The instruction described the shopping items and the required special actions. The shopping items were described with image and text. The special actions were only described in text. Both the shopping items and special actions represented the PM component.

As the ongoing component, the instruction also asked the participant to press a button when he passed each store. He must also read aloud a number under the store signboard. (Figure 3) The font size of the number was set to small to prevent the participant from reading the number from afar. Thus, when a par-

ticipant read out the number on the signboard, one could expect that he was standing near the store, and had also read the name of the store and realized what was available. It was up to the participant to choose whether to move to the green entrance to trigger the store menu. (Figure 3) When the store menu was triggered, the participant can use his head movement to control an on-screen selector to choose an item to purchase. The mechanism was the same for the special action point.

Task-related data were recorded in the background automatically into a log file. They included the participant’s head movement, walking path, and timing of all the button presses. Participant’s voice was also recorded and we received permission from all the participants for the use of their voice data.

2.3. fNIRS Settings and Data Analysis

The fNIRS system (OEG-16, Spectratech Inc., Tokyo, Japan) was a multichannel fNIRS sensor with 16 measurement points and each point corresponded to one channel (Figure 4). The fNIRS system used Beer-Lambert law to calculate the relative changes in oxyhemoglobin (oxyHb) and deoxyhemoglobin (deoxyHb). Wavelengths of 770 nm and 840 nm were used to measure Δoxyhemoglobin (ΔoxyHb) and Δdeoxyhemoglobin (ΔdeoxyHb) concentration. fNIRS baseline was taken after each participant finished reading the shopping list and rested for at least 10 seconds.

Linear fitting was used to correct the fNIRS data, and the individual hemodynamic responses were corrected by the baseline data. Data was extracted from the log file to measure the timing of the ongoing and PM components. As shown in Figure 5, the data can be divided into three categories: ongoing component data (when participants read the name of the store but the required item was not available), PM component data (when participants read the name of the store, and the required item was available), and PM-hit component data (when participants buy items or take actions). When an event occurred, the ΔoxyHb from the previous 6 seconds and after 6 seconds were extracted and the mean was calculated to represent the component [24] .

The PM component (shopping and action-taking task) in this study is regarded as event-based PM. As suggested by past studies [24] [25] [26] , brain ac-

tivities of event-based PM may be biased towards left prefrontal cortex near channel 10 to 15 (Figure 4). Therefore, the ΔoxyHb signal changes from channel 10 to 15 were analyzed. Subsequently, the difference between the maximum and minimum values of the ΔoxyHb were calculated. In order to examine the general variation of hemodynamic in BA10, the mean values of the difference between the maximum and minimum values ofΔoxyHb from channel 10 to 15 were compared.

3. Result

SPSS (version 22.0) was used to analyze the fNIRS data. For statistical analysis, we performed a series of 2-tailed paired-sample t-test on the ΔoxyHb to examine the hemodynamic change of ongoing component, PM component and PM-hit component (Figure 6 & Figure 7). In order to ensure the paired sample observations being tested were independent of each other, we excluded the comparison of PM and PM-hit. It was because the timing of the PM task and PM-hit were too close and the data was likely overlapping with each other.

From Table 1, we observed the difference of PM component was significantly greater than the difference of ongoing component (t = −2.37, p < 0.05, Cohen’s d = 0.76), which means the hemodynamic changes in the PM component is signif-

OG: the mean values of difference in ongoing component from 10 to 15 channels, PM: the mean values of difference in PM component from 10 to 15 channels, PMH: the mean values of difference in PM hit component from 10 to 15 channels. *p < 0.05.

icantly greater than the hemodynamic change in the ongoing component. In addition, the difference between the ongoing component and the PM-hit component was not significant (t = −1.98, p = 0.08, Cohen’s d = 0.63).

4. Discussion

In this study, we used fNIRS technique to record ΔoxyHb data when participants performed immersive VR task, and analyzed the result by using the paired-sam- ple t-test on the ongoing, PM and PM-hit components. We also analyzed the mean values of difference between the maximum and minimum from channel 10 to 15. Our result showed that the hemodynamic changes of BA10 in the PM component was significantly greater than the ones in the ongoing component.

Our previous study [20] compared the conventional slide-based PM task against the VR PM task and demonstrated the benefits of using immersive VR to create a real-life environment in a laboratory setting. Although, In VR task, participants could explore the virtual environment but only through devices. Since they were seated, they lacked proprioceptive information, and this makes a difference as compared to real world settings. However, that is the difficulty to control variables and strategies when participants performing a “real-life task”. Therefore, in laboratory settings, using head-mounted display (immersive VR) to create a life-like experience is a more promising method. In present study, the VR PM task was further examined with fNIRS and statistically significant hemodynamic changes of BA10 was observed. This novel approach of using immersive VR environment should serve as a reference when future study of BA10 activity and PM performance are conducted.

Although the mean of PM-hit component was considerably greater than the one of the ongoing component as shown in Figure 7, our analysis showed that the two components were not statistically significant. Since the PM-hit component normally appeared after the PM component had ended, the hemodynamic changes caused by the PM component may have influenced the ones of PM-hit, since it generally takes time for the hemodynamic changes to return to normal level. In other words, the large mean of the PM-hit component may have caused by a partial contribution of the PM component. It is possible that, if the participant is given enough time to restore to pre-PM level, the mean of PM-hit component may actually be smaller than what was observed. In order to confirm this theory, we are planning to develop a new experiment to separate the variables.

Using fNIRS with immersive VR headset posed several challenges during the experiment. The hair of the participant can reduce the transmission of infrared light and affect the result of the fNIRS measurement. And it is due to this reason that we had to exclude one of the participant’s data from experiment. Another problem was the cables coming from the VR headset and the fNIRS. Since the participants were allowed to rotate freely in a swivel chair to navigate in the shopping area, the cables from the equipment may tangle and prevent their movement if they rotated more than 360 degrees. During the experiment, the researchers had to constantly monitor the motion of the participants and carefully adjust the cables. A more efficient cable managing system is being considered until wireless fNIRS and wireless VR headsets are available in the future. Previous studies attempt to eliminate the motion noise from data by employing various techniques, such as motion-noise filtering from raw data [27] . Our study attempt to reduce motion noise by forcing participant to remain in a swivel chair in order to prevent large body motion.

5. Conclusion

This study designed an experiment that record hemodynamic changes when participants perform immersive VR PM task. Unlike previous studies, we used an fNIRS device to measure prefrontal cortex activity, and an immersive VR device to present an event-based PM VR task. By measuring the event of the ongoing components, PM components and PM-hit components, the fNIRS data of these three components was extracted. The paired-sample t-test showed that the immersive VR PM task was able to induce prefrontal cortex activity in the BA10 area that was associated to PM performance. With the use of fNIRS, this study provides a novel and feasible approach to measure prefrontal cortex activity in real-life immersive VR environment. This is also the first time that confirmed the activation of BA10 in an immersive VR PM task by using fNIRS technique. Since PM performance is sensitive to environmental change and ages, future study should focus on how environmental changes and age differences may affect PM cognition process in immersive VR environment.

Cite this paper

Dong, D., Wong, L.K.F. and Luo, Z.W. (2017) Assessment of Prospective Memory using fNIRS in Immersive Virtual Reality Environment. Journal of Behavioral and Brain Science, 7, 247-258. https://doi.org/10.4236/jbbs.2017.76018

References