Open Journal of Depression
Vol.06 No.03(2017), Article ID:78060,11 pages
10.4236/ojd.2017.63007

Can We Find Any Sustained Neurofunctional Alteration in Remitted Depressive Patients with a History of Modified Electroconvulsive Therapy?

Ryo Saito1, Kazuyuki Fujihara1, Masato Kasagi1, Tomokazu Motegi1, Yusuke Suzuki1, Kosuke Narita1*, Koichi Ujita2, Masato Fukuda1

1Department of Psychiatry and Neuroscience, Gunma University Graduate School of Medicine, Gunma, Japan

2Department of Diagnostic Radiology and Nuclear Medicine, Gunma University Graduate School of Medicine, Gunma, Japan

Copyright © 2017 by authors and Scientific Research Publishing Inc.

This work is licensed under the Creative Commons Attribution International License (CC BY 4.0).

http://creativecommons.org/licenses/by/4.0/

Received: June 20, 2017; Accepted: July 28, 2017; Published: July 31, 2017

ABSTRACT

Background and Aim: Recent magnetic resonance imaging (MRI) studies have revealed structural and functional differences in the human brain before and after an electroconvulsive therapy (ECT) series in patients with depression. The hippocampus is one of the well-documented subcortical areas that show an increase in cortical volume following an ECT series. Moreover, some resting-state functional MRI studies have shown that an ECT series alters the connectivities of the hippocampus with several brain areas. However, it remains unknown whether ECT can induce some sustained neurofunctional alterations in the human brain. Methods: In this study of the depressive patients with and without a history of modified ECT (N = 5 and 9, respectively) using a cross-sectional design, we investigated the functional connectivity between the left and right hippocampi in terms of blood-oxygenation-level dependent signal changes with time by resting-state functional MRI. Results: Voxel-based morphometry showed no significant structural differences in hippocampal volume between the ECT and non-ECT groups. The ECT group showed a significantly lower functional connectivity between the same brain areas than the non-ECT group (p < 0.01). Conclusions: Although their clinical significance remains unclear, our results suggest that an ECT series induces a sustained neurofunctional alteration in the human brain.

Keywords:

Resting-State Functional MRI, Modified Electroconvulsive Therapy, Depression, Hippocampus, Connectivity

1. Introduction

The hippocampus (HPC), which is located in the subcortical cortex, is one of the key regions that play significant roles in both cognition and mood function in patients with major depressive disorder (Duman, 2004; Sapolsky, 2004; Snyder, Soumier, Brewer, Pickel, & Cameron, 2011). Several previous studies have shown a smaller gray matter (GM) volume of the left and/or right HPC in depressive patients than in healthy controls (Bell-McGinty et al., 2002; Steffens et al., 2000; Mervaala et al., 2000). Consistently, memory functions and neuroplasticity in HPC have been shown by animal and human research to be likely impaired by stress (Videbech & Ravnkilde, 2004; Kim & Diamond, 2002).

Electroconvulsive therapy (ECT) is one of the major methods of treating drug-resistant depression. Recent neuroimaging studies have revealed structural and functional differences in the human brain before and after an ECT series (Dukart et al., 2014; Jorgensen et al., 2016; Ota et al., 2015). Some studies have shown an increase in the GM volume of bilateral HPCs after ECT (Jorgensen et al., 2016; Ota et al., 2015); others have shown that an ECT series significantly contributes to the increase in the GM volume of the right HPC, but not to that of the left HPC (Dukart et al., 2014; Abbott et al., 2014). Moreover, recent functional magnetic resonance imaging (MRI) studies of ECT in depression have shown some laterality of neuronal connectivity of HPC. That is, the right hippocampal connectivity with the left superior temporal lobe increased after an ECT series and correlated with the amelioration of depressive symptom, but no such increase in the left hippocampal connectivity with any brain regions was found (Abbott et al., 2014).

The left and right HPCs are linked anatomically and functionally. Moreover, several significant differences in morphology and neuronal circuitry are observed between the left and right HPCs (Goto et al., 2010), e.g., GM volume and NMDA receptor distribution. Thus, it is possible that anatomical and functional differences in response to ECT series can exist between the left and right HPCs. On the basis of this assumption, ECT might alter the connectivity between the left and right HPCs, but knowledge about this alteration is still limited. In addition, most of the previous studies of morphological or functional alterations after ECT were conducted with short observational durations from a week to a few months, and there are only a few research studies focusing on the long-term effect of ECT on a patient’s brain.

In this structural and resting-state functional MRI study using a cross-sec- tional design, we examined the morphological and functional symmetries between the left and right HPCs in elderly patients with and without a history of ECT. First, we performed a group comparison of the GM volume of the HPC between the ECT and non-ECT groups. Then, we compared between the ECT and non-ECT groups regarding the functional connectivity between the left and right HPCs in terms of blood-oxygenation-level-dependent (BOLD) signal changes with time.

2. Methods

2.1. Subjects

Five remitted depressive patients with a history of modified ECT [aged 51 - 85 years (mean, 64.0 ± 15.6 years); all females] and nine remitted depressive patients without a history of ECT [aged 54 - 68 years (mean, 60.4 ± 5.1 years); all males] were enrolled in this study between April and September in 2014 in Gunma University Hospital (7 inpatients and 7 outpatients). The details of demographic data were shown in Table 1. All the subjects enrolled in this study were right-handed as assessed using the Edinburgh Handedness Inventory ( Oldfield, 1971 ). For the evaluation of their past and current psychological conditions, the DSM-IV-TR Axis I Disorder ( American Psychiatric Association, 2000 ) was used. We used the Hamilton depression Rating Scale (HAM-D) to assess their depression grade ( Hamilton, 1967 ). To attenuate the effect of severity of depressive symptoms on the analysis of MRI data, only those who showed HAM-D scores below 10 (i.e., defined as remitted condition) were enrolled as subjects in this study. Moreover, the Japanese version of Mini-Mental State Examination (MMSE) was conducted for cognitive function assessment ( Folstein, Folstein, & McHugh, 1975 ; Ideno, Takayama, Hayashi, Takagi, & Sugai, 2012 ). All the subjects in the ECT and non-ECT groups had taken antidepressants

Table 1. Demographic characteristics of study subjects.

ECT, electroconvulsive therapy; MMSE, mini-mental state examination; HAM-D, Hamilton rating scale for depression; YMRS, Young mania rating scale; SSRI, Selective serotonin reuptake Inhibitor; SNRI, Serotonin Noradrenaline Reuptake Inhibitor; NaSSA, Noradrenergic and specific serotonergic antidepressant.

and/or psychotropic agents (see Table 1). All the subjects provided their written informed consent. The study protocol was approved by the Ethics Committee of Gunma University. This research has an observational and retrospective study design, and the patients had received treatments based only on clinical significance.

2.2. ECT Methods

All the patients were administered modified ECT 3 times a week using a brief-pulse square wave ECT device (Thymatron System IV, Somatics LLC, Lake Bluff, IL, USA). In all the patients, propofol was administered for anesthesia induction, and succinylcholine for muscle relaxation. All the patients were administered modified ECT bilaterally by bitemporal electrode placement at an energy dosage 1.5 times above the seizure threshold. The seizure threshold is defined as the minimum stimulus dosage required generating an EEG seizure for at least 25 s. The initial dosage was determined by the half-age method.

2.3. T1-Weighted Anatomical Imaging and Functional MRI Data Acquisition

Brain MRI was performed using Siemens 3-T Trio with a 12-channel head coil (Siemens, Erlangen, Germany) in Gunma University Hospital. High-resolution T1-weighted anatomical images [magnetization-prepared rapid acquisition with gradient echo (MP-RAGE) sequence] were acquired as follows: repetition time = 2000 ms; echo time = 2 ms; inversion time = 990 ms; flip angle = 9 degrees; field of view = 256 × 256 mm2; matrix size = 256 × 256; voxel size = 1 × 1 × 1 mm3. Functional images in the resting and task states were collected using an echo- planar imaging (EPI) sequence (TR = 2500 ms, TE = 25 ms, flip angle = 90˚, FOV = 220 mm, matrix size = 64 × 64 with pixel dimensions of 3 × 3 mm2 for 256 times). Thirty-five axial slices with a thickness of 4 mm and an interslice gap of zero were acquired.

2.4. Measurements of GM Volumes in Left and Right HPCs by T1-Weighted Imaging

For measurements of GM volumes in the left and right HPCs in each subject, T1-weighted volumetric images were analyzed using SPM8 (Wellcome Department of Cognitive Neurology, London, UK) implemented in Matlab 2015a (MathWorks, Natick, MA, USA) to carry out VBM8 (http://dbm.neuro.uni-jena.de/vbm8/) using default parameters. Each image was inspected for reconstruction artifacts. The VBM8 method combines spatial normalization, segmentation, and volumetric modulation. The toolbox employs a hidden Markov random field (HMRF) model in the procedure to segment each image into GM, white matter, and cerebrospinal fluid. The toolbox then normalizes the GM segment of each image to the International Consortium for Brain Mapping (ICBM) 152 template (Montreal Neurological Institute; MNI) and performs a modulation step to scale each voxel value according to the subject’s total intracranial volume, as well as the regional GM volume expansion/contraction that occurs during nonlinear transformation. The aim of the modulation step was to render the final VBM statistical data reflective of the “volume” change rather than the “concentration” change in GM. GM image segments were inspected for segmentation artifacts, then smoothed using an isotropic Gaussian kernel of 12 mm full width at half maximum (FWHM), to accommodate individual differences in the sulcal and gyral anatomy, and to meet the distributional assumptions of the general linear models necessary for statistical analysis. An absolute threshold mask of 0.10 was used to avoid possible edge effects around the border between GM and the white matter. Finally, we determined the GM volumes in the left and right HPCs, respectively, as follows: the mean signal intensities on T1-weighted images were extracted from the left or right HPC using the Harvard-Oxford Subcortical Structural Atlas from Version 6.0, which is part of the FMRIB’s Software Library (FSL, http://www.fmrib.ox.ac.uk/fsl), using a probability threshold mask of 60 (Figure 1). In addition, the left and right cerebral hemispheres were also estimated as control ROIs, using the atlas of AAL (http://www.gin.cnrs.fr/AAL).

2.5. Functional Connectivity between Left and Right HPCs in Terms of BOLD Signal Changes with Time

The obtained resting-state functional MRI data were analyzed using a temporal band-pass filter from 0.008 to 0.09 Hz with the AFNI program

(http://afni.nimh.nih.gov/afni) ( Cox, 1996 ): 3 dB and pass (Figure 2). Then, we extracted the mean BOLD signal changes with time within each ROI in the left or right HPC in each subject using the AFNI program: 3dROIstats. Pearson’s product-moment correlation coefficient was used as an index of connectivity between the left and right HPCs in each subject, which was computed in terms of the mean BOLD signal changes with time within the above-mentioned ROIs, i.e., the left and right hemispheres of the cerebrum and HPC using the AFNI

Figure 1. ROI masks (yellow) of left and right hippocampi (HPCs) determined on the basis of Harvard-Oxford Subcortical Structural Atlas.

Figure 2. Methods for calculating correlation coefficient “r” between left and right hippocampi (HPCs) in terms of blood-oxygenation-level-dependent (BOLD) signals with time.

program: 3 dfim.

2.6. Statistical Analyses

Pearson’s correlation analysis was conducted to estimate the association between demographic characteristics and MRI data. The unpaired Student t-test was used for comparison between the ECT and non-ECT groups.

3. Results

No significant differences between the ECT and non-ECT groups were observed in age, years of education, and scores of HAM-D and MMSE. Moreover, morphometry showed no significant differences between the ECT and non-ECT groups in terms of the GM volume of all ROIs, i.e., the left and right hemispheres of the cerebrum and HPC. In both the ECT and non-ECT groups, age, years of education, and HAM-D and YMRS scores showed no significant associations with GM volume or the correlation coefficient between time courses of BOLD signal changes in left and right HPC activities.

By computing the correlation coefficient between the left and right cerebral hemi-spheres in terms of BOLD signal changes using the resting-state functional MRI data of each subject, we found that all the subjects of both the ECT and non-ECT groups showed significant correlations between those brain areas (p < 0.001). Also, regarding the correlation between the left and right HPC activities, all the subjects of both the ECT and non-ECT groups showed significant correlations between those brain areas (p < 0.001). Pearson’s correlation analysis showed that the correlation between the left and right HPC activities no significantly correlated with age in both groups or each group of ECT and non-ECT (p > 0.05).

The ECT group showed a significantly smaller correlation coefficient between the left and right HPCs than the non-ECT group (p < 0.01) (Figure 3: correla- tion coefficient r mean ± SD and range of ECT group and non-ECT group; 0.60 ± 0.10, 0.48 to 0.72 and 0.78 ± 0.09, 0.65 to 0.90, respectively). No significant differences in the correlation coefficient between the left and right cerebral hemispheres were observed between the ECT and non-ECT groups.

Figure 3. Comparison between ECT and non-ECT groups in correlation coefficient “r” between left and right hemispheres of cerebrum and hippocampus (HPC) in terms of blood-oxygenation-level-dependent (BOLD) signals with time. NS, not significant. The paired t-test was used. *p < 0.01.

4. Discussion

Despite the high effectiveness of ECT, the mechanism of ECT is still unknown. However, in recent years, the findings regarding the effects of ECT on brain volume and neurofunction have gathered using Voxel-based morphometry and functional MRI. Although several previous research studies have shown the increase in the GM volume of HPC after ECT (Dukart et al., 2014; Jorgensen et al., 2016; Ota et al., 2015), the morphometry carried out in this study showed no significant differences between the ECT and non-ECT groups in the GM volume of all ROIs, i.e., the left and right hemi-spheres of the cerebrum and HPC. The observational durations after ECT in those previous studies [within 3 months (Dukart et al., 2014), 1 - 4 weeks (Jorgensen et al., 2016), and 9.0 ± 14.6 days (Ota et al., 2015)] were shorter than that in our study (0.86 ± 1.59 years). This might be a reason for the absence of a significant difference in GM volume between the ECT and non-ECT groups in our study. Nordanskog’s study showed no significant difference between the GM volume of HPC before ECT and that 6 months after ECT (Nordanskog, Larsson, Larsson, & Johanson, 2014), which might support our finding.

In this study, the ECT group showed a significantly lower connectivity between the left and right HPCs than the non-ECT group. On the other hand, no significant difference in the connectivity between the left and right cerebral hemispheres was found between the ECT and non-ECT groups, which were measured as control ROIs. These findings suggest that the depressive subjects with a history of ECT show sustained neurofunctional alterations, and that is greater asymmetry between bilateral hippocampal activities than those without a history of ECT.

The left and right HPCs are directly connected by the hippocampal commissure ( Wilson, Culver, Waddington, & Gazzaniga, 1975 ), and their activities show definite synchronization ( Goto et al., 2010 ). Consistently, all the subjects enrolled in this study showed significant correlations between the left and right HPCs in terms of the BOLD signal changes with time in resting-state functional MRI. On the other hand, the above-mentioned difference between the ECT and non-ECT groups in the connectivity between the left and right HPCs was observed in this study. The pathophysiological mechanisms underlying our findings are difficult to explain, because to the best of our knowledge there are no studies in which the connectivity between the left and right HPCs have been investigated in subjects with a history of ECT. It can be speculated that the functional difference between the left and right HPCs affected our results, as shown by the laterality of NMDA receptor distribution in the HPCs in a previous study ( Goto et al., 2010 ). Since ECT is suggested to alter NMDA receptor ( Park et al., 2014 ) and glutamate/glutamine expression levels ( Michael et al., 2003 ), the asymmetric effects of ECT on neuronal HPC circuits associated with NMDA might occur after ECT. Further studies to reveal the pathophysiological mechanisms underlying the long-term effects of ECT on brain function are necessary.

There were several limitations in this study. The power value was 0.08 in this study that was not enough to avoid type II error, owing to its small sample size. Furthermore, we were unable to analyze the causal relationship, that is, the effect of an ECT series on functional brain activities over a natural course, because of the cross-sectional design of this study. The ECT methods also lacked uniformity. Furthermore, the associations of psychological and cognitive functions with connectivity between the left and right HPCs should be investigated in detail, because HPC plays significant roles in the functions. Future studies using a longitudinal design having a large number of subjects and psychological and cognitive assessments will be required. Moreover, previous studies showed that cognitive functions were declined immediately after ECT than baseline ( Bodnar et al., 2016 ), but did not change significantly 6 months or 12 months after ECT ( Nordanskog, Larsson, Larsson, & Johanson, 2014 ). Future longitudinal design research with a longer observation period over one year is needed.

In summary, the results of our study suggest that long-term neurofunctional alterations of connectivity between the left and right HPCs may occur. Whether an ECT series can cause neurofunctional alteration in the human brain is a very important issue clinically. Therefore, it is necessary that knowledge gained from studies such as this continue to be accumulated in the future.

Acknowledgements

KN acknowledges the support from JSPS KAKENHI grants (Numbers JP 23591695 and 15KK0295).

Cite this paper

Saito, R., Fujihara, K., Kasagi, M., Motegi, T., Suzuki, Y., Narita, K., Ujita, K., & Fukuda, M. (2017). Can We Find Any Sustained Neurofunctional Alteration in Remitted Depressive Patients with a History of Modified Electroconvulsive Therapy? Open Journal of Depression, 6, 89-99. https://doi.org/10.4236/ojd.2017.63007

References

  1. 1. Abbott, C. C., Jones, T., Lemke, N. T., Gallegos, P., McClintock, S. M., Mayer, A. R., & Calhoun, V. D. (2014). Hippocampal Structural and Functional Changes Associated with Electroconvulsive Therapy Response. Translational Psychiatry, 4, e483.
    https://doi.org/10.1038/tp.2014.124

  2. 2. American Psychiatric Association (2000). Diagnostic and Statistical Manual of Mental Disorders (4th Editon, text rev.). Washington, DC: Author. [Paper reference 1]

  3. 3. Bell-McGinty, S., Butters, M. A., Meltzer, C. C., Greer, P. J., Reynolds, C. F., & Becker, J. T. (2002). Brain Morphometric Abnormalities in Geriatric Depression: Long-Term Neurobiological Effects of Illness Duration. American Journal of Psychiatry, 159, 1424-1427.
    https://doi.org/10.1176/appi.ajp.159.8.1424

  4. 4. Bodnar, A., Krzywotulski, M., Lewandowska, A., Chlopocka-Wozniak, M., Bartkowska-Sniatkowska, A., Michalak, M., & Rybakowski, J. K. (2016). Electroconvulsive Therapy and Cognitive Functions in Treatment-Resistant Depression. The World Journal of Biological Psychiatry, 17, 159-164.
    https://doi.org/10.3109/15622975.2015.1091501 [Paper reference 1]

  5. 5. Cox, R. W. (1996). AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance Neuroimages. Computers and Biomedical Research, 29, 162-173.
    https://doi.org/10.1006/cbmr.1996.0014 [Paper reference 1]

  6. 6. Dukart, J., Regen, F., Kherif, F., Colla, M., Bajbouj, M., Heuser, I., & Draganski, B. (2014). Electroconvulsive Therapy-Induced Brain Plasticity Determines Therapeutic Outcome in Mood Disorders. Proceedings of the National Academy of Sciences of the United States of America, 111, 1156-1161.
    https://doi.org/10.1073/pnas.1321399111

  7. 7. Duman, R. S. (2004). Depression: A Case of Neuronal Life and Death? Biological Psychiatry, 56, 140-145.
    https://doi.org/10.1016/j.biopsych.2004.02.033

  8. 8. Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). Mini-Mental State: A Practical Method for Grading the Cognitive State of Patients for the Clinician. Journal of Psychiatric Research, 12, 189-198.
    https://doi.org/10.1016/0022-3956(75)90026-6 [Paper reference 1]

  9. 9. Goto, K., Kurashima, R., Gokan, H., Inoue, N., Ito, I., & Watanabe, S. (2010). Left-Right Asymmetry Defect in the Hippocampal Circuitry Impairs Spatial Learning and Working Memory in iv Mice. PLoS ONE, 5, e15468.
    https://doi.org/10.1371/journal.pone.0015468 [Paper reference 2]

  10. 10. Hamilton, M. (1967). Development of a Rating Scale for Primary Depressive Illness. British Journal of Social and Clinical Psychology, 6, 278-296.
    https://doi.org/10.1111/j.2044-8260.1967.tb00530.x [Paper reference 1]

  11. 11. Ideno, Y., Takayama, M., Hayashi, K., Takagi, H., & Sugai, Y. (2012). Evaluation of a Japanese Version of the Mini-Mental State Examination in Elderly Persons. Geriatrics & Gerontology International, 12, 310-316.
    https://doi.org/10.1111/j.1447-0594.2011.00772.x [Paper reference 1]

  12. 12. Jorgensen, A., Magnusson, P., Hanson, L. G., Kirkegaard, T., Benveniste, H., Lee, H., Jorgensen, M. B. et al. (2016). Regional Brain Volumes, Diffusivity, and Metabolite Changes after Electroconvulsive Therapy for Severe Depression. Acta Psychiatrica Scandinavica, 133, 154-164.
    https://doi.org/10.1111/acps.12462

  13. 13. Kim, J. J., & Diamond, D. M. (2002). The Stressed Hippocampus, Synaptic Plasticity and Lost Memories. Nature Reviews. Neuroscience, 3, 453-462.

  14. 14. Mervaala, E., Föhr, J., Könönen, M., Valkonen-Korhonen, M., Vainio, P., Partanen, K., Lehtonen, J. et al. (2000). Quantitative MRI of the Hippocampus and Amygdala in Severe Depression. Psychological Medicine, 30, 117-125.
    https://doi.org/10.1017/S0033291799001567

  15. 15. Michael, N., Erfurth, A., Ohrmann, P., Arolt, V., Heindel, W., & Pfleiderer, B. (2003). Metabolic Changes within the Left Dorsolateral Prefrontal Cortex Occurring with Electroconvulsive Therapy in Patients with Treatment Resistant Unipolar Depression. Psychological Medicine, 33, 1277-1284.
    https://doi.org/10.1017/S0033291703007931 [Paper reference 1]

  16. 16. Nordanskog, P., Larsson, M. R., Larsson, E. M., & Johanson, A. (2014). Hippocampal Volume in Relation to Clinical and Cognitive Outcome after Electroconvulsive Therapy in Depression. Acta Psychiatrica Scandinavica, 129, 303-311.
    https://doi.org/10.1111/acps.12150 [Paper reference 1]

  17. 17. Oldfield, R. C. (1971). The Assessment and Analysis of Handedness: The Edinburgh Inventory. Neuropsychologia, 9, 97-113. [Paper reference 1]

  18. 18. Ota, M., Noda, T., Sato, N., Okazaki, M., Ishikawa, M., Hattori, K., Kunugi, H. et al. (2015). Effect of Electroconvulsive Therapy on Gray Matter Volume in Major Depressive Disorder. Journal of Affective Disorders, 186, 186-191.

  19. 19. Park, H. G., Yu, H. S., Park, S., Ahn, Y. M., Kim, Y. S., & Kim, S. H. (2014). Repeated Treatment with Electroconvulsive Seizures Induces HDAC2 Expression and Down-Regulation of NMDA Receptor-Related Genes through Histone Deacetylation in the Rat Frontal Cortex. The International Journal of Neuropsychopharmacology, 17, 1487-1500.
    https://doi.org/10.1017/S1461145714000248 [Paper reference 1]

  20. 20. Sapolsky, R. M. (2004). Is Impaired Neurogenesis Relevant to the Affective Symptoms of Depression? Biological Psychiatry, 56, 137-139.

  21. 21. Snyder, J. S., Soumier, A., Brewer, M., Pickel, J., & Cameron, H. A. (2011). Adult Hippocampal Neurogenesis Buffers Stress Responses and Depressive Behaviour. Nature, 476, 458-461.
    https://doi.org/10.1038/nature10287

  22. 22. Steffens, D. C., Byrum, C. E., McQuoid, D. R., Greenberg, D. L., Payne, M. E., Blitchington, T. F., Krishnan, K. R. R. et al. (2000). Hippocampal Volume in Geriatric Depression. Biological Psychiatry, 48, 301-309.

  23. 23. Videbech, P., & Ravnkilde, B. (2004). Hippocampal Volume and Depression: A Meta-Analysis of MRI Studies. The American Journal of Psychiatry, 161, 1957-1966.
    https://doi.org/10.1176/appi.ajp.161.11.1957

  24. 24. Wilson, D. H., Culver, C., Waddington, M., & Gazzaniga, M. (1975). Disconnection of the Cerebral Hemispheres. An Alternative to Hemispherectomy for the Control of Intractable Seizures. Neurology, 25, 1149-1153.
    https://doi.org/10.1212/WNL.25.12.1149 [Paper reference 1]