Laparoscopic surgery is a difficult surgical procedure compared with laparotomy. In particular, considerable skills and care are required for thread knotting in laparoscopic surgery. In this paper, a method for automatic identification of a laparoscopic surgical procedure for ligation and online distinction of an abnormality, defined as any unusual manipulation, in the identified surgical procedure is proposed. Ligation is divided into several individual surgical procedures, and on the basis of the threshold criteria, each surgical procedure is identified. Next, the identified surgical procedure, thread knotting, is classified as either normal or abnormal using a self-organizing map. Finally, to reduce surgical error, an abnormality warning system which warns detection of an unusual manipulation in the surgical procedure to the operator is constructed.
Recently, minimally invasive surgical procedures, such as laparoscopic surgery, have been taking the place of open procedures. To perform laparoscopic surgery, a surgeon must have considerable skills. A less-experienced surgeon has tendency to perform unusual manipulation in surgical procedure as compared with a skilled surgeon. Such surgeries may result in serious surgical error. Therefore, to reduce surgical error, development of a system that alerts less-experienced surgeons to the unusual manipulation is in demand.
To the best of our knowledge, a system that identifies any abnormality in a surgical procedure, defined as an unusual manipulation usually caused by the inexpertness of a less-experienced surgeon or careless mistake of a skilled surgeon, has not yet been considered. It is a great challenge to identify any abnormality in a surgical procedure and to alert the surgeon to the detection of the unusual manipulation during the surgery. Objectives of this study are to develop a method that identifies laparoscopic surgical procedure and distinguishes any abnormality for the identified surgical procedure automatically during the laparoscopic surgical procedure, and to construct an abnormality warning system which warns detection of the abnormality in the surgical procedure to the operator. To this end, haptic device “PHANTOM Omni”, strain gauge, and surface electromyography (SEMG) are employed to detect any abnormality in the surgical procedure.
In the case where the sensor like a motion capturing system is used, only an appearance of the motion can be measured. On the other hand, by using the SEMG, a muscular activity of surgeon is measured. Therefore, the features such as straining of the muscle can be obtained, which cannot be measured by the motion capturing system. Use of SEMG has attracted the attention of researchers as a method of interaction between humans and machines. The amplitude property of a waveform and the power spectrum based on a frequency analysis are typical information that can be extracted from the SEMG signal. Recognition of 25 kinds of hand gesture, consisting of various motions of wrist and fingers, has been performed in [
As for surgical procedures, it is reported in [
In this paper, a novel method for an automatic identification of a surgical procedure for ligation and an online distinction of any abnormality in the identified surgical procedure on the basis of SEMG measurements of an operator and movements of the forceps is proposed. Ligation is divided into seven distinct procedures. The features of the procedure are extracted from the measurements of the movement of the forceps, and then, on the basis of the threshold criteria for the seven procedures, the surgical procedure is identified as one of the seven procedures. Next, the characteristics of any abnormality in the surgical procedure are extracted from the measurement of the movement of the forceps and operator’s SEMG signals, and the identified surgical procedure is classified as either normal manipulation or abnormal manipulation using a self-organizing map [
Using a simulation box built for laparoscopic surgery incorporating two forceps, the identification of each surgical procedure in ligation and distinction of the abnormality of the identified surgical procedure were carried out for a specific surgical procedure; namely, thread knotting by pulling the suture during ligation. Each surgical procedure in ligation could be identified with greater than 88% accuracy on average, and the abnormality of the surgical procedure of thread knotting could be distinguished with greater than 97% accuracy on average. Thus, the experimental results showed the effectiveness of the proposed method. Therefore, prototype of a novel system to warn detection of the abnormal manipulation to the operator was constructed in application with the proposed method for distinction of the abnormality of the surgical procedure. By using this system, it is considered that serious surgical error can be reduced, since the surgeon can notice unusual manipulation during the surgery.
A preliminary version of this paper is presented in [
The simulation box for laparoscopic surgery is shown in
In order to identify operating characteristics of the forceps, as shown in
The operating characteristics of the forceps as a mean absolute value are defined as follows:
where Strainch(n)
In order to distinguish an abnormality caused by straining of the muscle of the operator, the operator’s SEMG signals were also assessed. The SEMG signals were measured using six electrodes pasted on each forearm of the operator, as shown in
The SEMG signals were measured using a sampling frequency Fs = 1 kHz, and fast Fourier transform (FFT) was performed to each SEMG signal for every N = 512 sampled data, which is equivalent to performing FFT every 0.512 s. After filtering the SEMG signals using the fourth-order Butterworth-type high-pass filter from 1 Hz to 1 kHz range, a full wave rectification was carried out. In addition, for normalization, the measured SEMG signal of each electrode was divided by the maximum value of the premeasured SEMG for each procedure. The characteristics of the SEMG are defined as follows:
1) Mean absolute value: In order to perform pattern recognition, the mean absolute value of each electrode is typically used, which is given as follows:
where EMGch(n)
2) Center-of-gravity: In the case where an unusual manipulation caused by straining of the operator is performed, it is expected that a change in the waveform can be observed in the SEMG signal. Therefore, the value of the center-of-gravity is used to represent the change of the waveform of the SEMG signal, which is defined as follows:
3) Spectrum ratio: In the case where an unusual manipulation caused by straining of the operator is performed, it is also expected that a change of distribution of the power spectrum can be observed in the SEMG signal. Therefore, the ratio of distribution of the power spectrum of the SEMG signal was also employed. It is well known that the SEMG signal is distributed in the frequency band between 5 and 500 Hz. Therefore, to record the ratio of the spectrum, the frequency band is divided into two bands; from 5 to 250 Hz and from 250 to 500 Hz. Thus, the value of the spectrum ratio is defined as follows:
where
and
Now, consider to divide a series of ligation procedures into several procedures from a viewpoint of the forceps manipulation. With taking a significance of the manipulation into consideration, the surgical procedure for ligation is divided into seven procedures, as shown in
1) Grasping (left): Grasping the suture with the left forceps;
2) Overwrapping (left) + Translation: Twisting around the suture held by left forceps on the axis of the right forceps, and moving the right forceps closer to the opposite side of the suture;
3) Grasping (left + right): Grasping the suture with both forceps;
4) Thread knotting: Thread knotting by pulling the suture;
5) Grasping (right): Grasping the suture with the right forceps;
6) Overwrapping (right) + Translation: Twisting around the suture held by right forceps on the axis of the left forceps, and moving the left forceps closer to the opposite side of the suture;
7) Neutral: The state where nothing is operating.
In order to identify the surgical procedure for ligation, the following new features are defined using the features defined by (1) and (2).
Then, on the basis of the threshold criteria, the surgical procedure for ligation is identified as one of the seven procedures. The threshold value THi (I = 1, 2, 3, 4, 5) for each new feature was determined by trial and error through repetition of the experiments. The experiments were performed by an intermediate-level subject who is not engaged in medical treatment but can perform the surgical procedure for ligation smoothly through training. To determine threshold values which are applicable to persons with various skill levels, the intermediate-level subject was chosen. The determined threshold values are shown in
To distinguish the surgical procedure, the following values Ti (I = 1, 2, 3, 4, 5), termed identification value, are defined.
Thus, the surgical procedure for ligation is identified according to the logical criteria for the seven procedures shown in
A SOM is a kind of artificial neural network that can be trained without using teacher signals. Using the SOM, high-order input data can be classified visually on the map. SOM is suitable for this study because it is easy to change the number of classifications without teacher signals and the learning time is comparatively short as compared with other traditional methods.
After the surgical procedure was identified through automatic identification, online distinction of the abnormality of the identified surgical procedure is performed. In this paper, taking the manipulations which a less-expe- rienced surgeon tends to do into consideration, a normal manipulation and abnormal manipulations are defined as follows:
1) Normal: The surgical procedure that is performed in normal manner;
2) Abnormal: The surgical procedures that are assumed to be performed in the following unusual manner:
a) Posture: The surgical procedure that is performed in a posture in which the operator’s elbow is raised;
b) Grasp: The surgical procedure that is performed in a state in which the operator grasps the forceps too
TH1 | TH2 | TH3 | TH4 | TH5 |
---|---|---|---|---|
0.1 | 0.2 | 0.15 | 2.2 | 2.2 |
Identification Value/Procedures | T1 | T2 | T3 | T4 | T5 | |
---|---|---|---|---|---|---|
1 | Grasping (left) | 0 | 1 | 0 | 0 | 0 |
2 | Overwrapping (left) + Translation | 0 | 1 | 0 | 1 | 0 or 1 |
3 | Grasping (right + left) | 1 | 1 | 0 | 0 or 1 | 0 or 1 |
4 | Thread knotting | 1 | 1 | 1 | 0 or 1 | 0 or 1 |
5 | Grasping (right) | 1 | 0 | 0 | 0 | 0 |
6 | Overwrapping (right) + Translation | 1 | 0 | 0 | 0 or 1 | 1 |
Neutral | Else |
Firmly;
c) Pull: The surgical procedure that is performed in a state in which the operator pulls the suture too forcefully;
d) Sudden: A rough surgical procedure that is performed suddenly by the operator;
e) Straining: The surgical procedure that is performed in a state in which the operator is straining.
These different abnormal scenarios are illustrated in
A questionnaire about the degree of danger for each state of abnormal manipulation was conducted to three skilled surgeons of laparoscopic surgery. The results are shown in
From these results, the Grasp, Pull and Sudden states are regarded as states which are easy to lead to danger compared with other states of abnormal manipulation. Therefore, these three states are categorized as high-risk states, since the strong load imposed on the suture may lead to serious surgical error.
The identified surgical procedure through automatic identification is classified either as normal manipulation or abnormal manipulation using two kinds of SOM. Although this method is applicable to any identified surgical procedure, to save the space of the paper, in this study, the thread knotting of ligation was chosen for the distinction of a possible abnormality. This is because thread knotting is the procedure where the surgeon must perform most carefully, according to the surgeon’s opinion in the questionnaire. Therefore, hereafter the thread knotting is classified either as normal or abnormal using two kinds of SOM. Online distinction of the abnormality is
Abnormal Manipulation | Degree of Danger [5 Point System] | |||
---|---|---|---|---|
Surgeon A | Surgeon B | Surgeon C | Total | |
Posture | 3 | 3 | 3 | 9 |
Grasp | 3 | 5 | 5 | 13 |
Pull | 5 | 4 | 5 | 14 |
Sudden | 3 | 5 | 5 | 13 |
Straining | 3 | 5 | 3 | 11 |
performed in the following two-phase process.
First, states with greater risk to the patient must be distinguished. Therefore, the distinction of five states including three high-risk states, namely Normal, Posture, Grasp, Pull and Sudden, is executed based on only the feature of the operation of the forceps using SOM1. Next, for a state of the procedure which has been recognized as Normal by SOM1, the distinction of two states, namely Normal and Straining, is executed based on the feature of operator’s SEMG using SOM2.
Then, SOM1 and SOM2 are built for the left hand side and right hand side, respectively. Therefore, four SOMs are built in total to distinguish the abnormal manipulation. Thus, an accurate distinction of the abnormality is expected by hierarchy classification on the basis of the forceps operation and biological signals. The feature vector for each SOM, which is also an input of the SOM, is defined as follows:
1) SOM1: When configuring the feature vectors, it is desirable to normalize each component. To make the component of the feature vector fall within the range 0 - 1, the following new variables are defined.
where Rphantomch(n) (ch = 1, 2) is the rotation angle at the attached position of the forceps measured from the haptic device.
Then, using Equations (7), (8), (11), and (12), the following feature vectors are defined, which is as input to SOM1 (Right) and SOM1 (Left), respectively.
2) SOM2: Using Equations (3), (4), and (5), the following feature vectors are defined, which is as input to SOM2 (Right) and SOM2 (Left), respectively.
22 - 27 year-old’s six subjects who are not engaged in medical treatment but fully trained the surgical procedure for ligation, and one skilled surgeon who is an authorized surgeon for assessment of skill in laparoscopic surgery were chosen as the operator for the surgical procedure for ligation, and automatic identification of the surgical procedure and online distinction of the abnormality of the surgical procedure identified as thread knotting were performed.
1) Experimental method: In accordance with the algorithm shown in
2) Experimental results: The identification accuracy for each individual procedure was calculated. The results are shown in
Surgical Procedures for Ligation | 1. Grasping (Left) | 2. Overwrapping (Left) + Translation | 3. Grasping (Right + Left) | 4. Thread Knotting | ||||
---|---|---|---|---|---|---|---|---|
Subject | Count | Rate [%] | Count | Rate [%] | Count | Rate [%] | Count | Rate [%] |
A | 15/15 | 100 | 12/15 | 80.0 | 15/15 | 100 | 15/15 | 100 |
B | 15/15 | 100 | 14/15 | 93.3 | 15/15 | 100 | 15/15 | 100 |
C | 15/15 | 100 | 12/15 | 80.0 | 14/15 | 93.3 | 15/15 | 100 |
D | 15/15 | 100 | 13/15 | 86.7 | 14/15 | 93.3 | 15/15 | 100 |
E | 15/15 | 100 | 15/15 | 100 | 11/15 | 73.3 | 15/15 | 100 |
F | 15/15 | 100 | 14/15 | 93.3 | 12/15 | 80.0 | 15/15 | 100 |
Skilled surgeon | 13/13 | 100 | 11/13 | 84.6 | 11/13 | 84.6 | 13/13 | 100 |
Average | 100 | 88.3 | 89.2 | 100 |
1) Experimental method: Using the simulation box, the thread knotting in the surgical procedure for ligation was performed for the rubber sheet in the mannequin under the six states, namely Normal, Posture, Grasp, Pull, Sudden and Straining, repeatedly. Then, feature vectors for SOM1 and SOM2 were constituted, respectively. The SOM toolbox was used to build SOMs, and the size of the SOM was determined as 10 × 10 with hexagon lattice. The SOM1 and SOM2 were constructed by batch learning using the premeasured 20 feature vectors for each state (a total of 120 feature vectors).
In addition, the k-means method was employed to divide the domain of SOM1 into five fields and domain of SOM2 into two fields. The thread knotting was performed 60 times for each state (360 times in total). A feature vector extracted from the online surgical procedure was mapped on the map of the learned SOMs, and an abnormal manipulation was distinguished by viewing the distribution on the map.
For distinction of the abnormality, common SOM1 was used for multiple subjects, but SOM2 was built for each subject and the distinction was conducted individually. This is because SOM2 uses the SEMG as the features and there exists individual difference in the SEMG. It should be described that authors have tried to combine SOM1 and SOM2 and use solo SOM trained with multiple subjects. However, the trained SOM did not work well at all.
2) Experimental results:
The distinction accuracy using SOM1 and SOM2 based on two-handed manipulation of the forceps and SEMG of both hands for the Subject A-D and the skilled surgeon is shown in
From
3) Consideration: In the states Grasp, Pull and Sudden, an incorrect distinction can be seen mutually. This is because the characteristic of the feature vector in these states is similar, and the difference does not appear clearly in the feature vector. However, this is not a serious problem in terms of an abnormality warning. Because, for the states Grasp, Pull and Sudden, a common acoustic warning is executed in an abnormality warning system described in the next section.
Subject | Hand | Distinction Rate [%] | ||||||
---|---|---|---|---|---|---|---|---|
Normal | Posture | Grasp | Pull | Sudden | Straining | Abnormal | ||
A | R | 96.7 | 100 | 100 | 96.7 | 100 | 91.7 | 97.7 |
L | 91.7 | 100 | 100 | 83.3 | 100 | 95.0 | 96.3 | |
B | R | 83.3 | 100 | 93.3 | 83.3 | 83.3 | 100 | 94.7 |
L | 76.7 | 100 | 81.7 | 81.7 | 80.0 | 96.7 | 95.7 | |
C | R | 91.7 | 100 | 93.3 | 98.3 | 90.0 | 83.3 | 95.0 |
L | 88.3 | 100 | 81.7 | 86.7 | 91.7 | 83.3 | 95.0 | |
D | R | 98.3 | 100 | 100 | 98.3 | 98.3 | 100 | 99.3 |
L | 98.3 | 100 | 100 | 95.0 | 98.3 | 98.3 | 98.3 | |
Skilled surgeon | R | 100 | 66.7 | 100 | 100 | 66.7 | 83.3 | 83.3 |
L | 83.3 | 83.3 | 83.3 | 83.3 | 83.3 | 100 | 86.7 |
SOM Area | State of manipulation (492 times for each manipulation) | Abnormal | |||||
---|---|---|---|---|---|---|---|
Normal | Posture | Grasp | Pull | Sudden | Straining | ||
Normal | 446 | 2 | 16 | 13 | 16 | 23 | 70 |
Posture | 5 | 489 | 5 | 0 | 0 | 2 | 496 |
Grasp | 21 | 1 | 461 | 23 | 4 | 6 | 495 |
Pull | 2 | 0 | 5 | 445 | 18 | 4 | 472 |
Sudden | 5 | 0 | 5 | 11 | 454 | 1 | 471 |
Straining | 13 | 0 | 0 | 0 | 0 | 456 | 456 |
Count | 446/492 | 489/492 | 461/492 | 445/492 | 454/492 | 456/492 | 2390/2460 |
Rate [%] | 90.7 | 99.4 | 93.7 | 90.4 | 92.3 | 92.7 | 97.2 |
An abnormality warning system to reduce medical errors is constructed in application with the above results, in which a series of surgical procedures for ligation is automatically identified as one of the seven procedures. When the procedure is identified as thread knotting, distinction of the abnormality is performed. In addition, when the procedure is identified as an abnormal manipulation, the detection of the abnormal manipulation is communicated to the operator as a warning.
The outline of the system is shown in
The abnormal manipulations are classified into Posture and Straining as comparatively low-risk states, and Grasp, Pull and Sudden as high-risk states. For the former, the operator is warned of the detection of the abnormal manipulation by a change in the color of the box at the bottom of the monitor (
Using the proposed system, the thread knotting was performed online for the simulation box. The system worked well as expected. In online distinction of the abnormality, approximately 0.2 s of time delay occurred due to the signal processing. However, it should be mentioned that this was not a cause for concern when performing a surgical procedure.
In this paper, a novel method for the automatic identification of a surgical procedure and online distinction of the abnormality of the identified surgical procedure was proposed. The surgical procedure for ligation was divided into seven procedures. First, on the basis of the threshold criteria, a surgical procedure was identified as one of the seven procedures automatically. Next, when the surgical procedure was identified as thread knotting, the procedure was automatically classified as either normal manipulation or abnormal manipulation using SOMs.
In order to evaluate the proposed method, using the simulation box built for laparoscopic surgery, the experimental works were executed. In the experiments, for both automatic identification of the surgical procedure and online distinction of abnormality for the identified surgical procedure, high recognition accuracy was obtained. Thus, it was concluded that the effectiveness of the proposed method was demonstrated.
In addition, an abnormality warning system to reduce medical errors was constructed. In this system, for comparatively low-risk states, the operator was warned of an abnormal manipulation by visually changing the color of the box at the bottom of the monitor and acoustically by a chime sound. For high-risk states, in addition to the visual and acoustic warnings, a resistance force was provided to the operator by the PHANTOM Omni. The validity of the system was also verified through a series of experiments for the simulation box.
This study was partially supported by Grants-in-Aid for Scientific Research (C) 25330312. The authors would like to thank for the grants.
ChiharuIshii,TakanoriSato,KaitoMurano,HidekiKawamura, (2015) Automatic Identification for Laparoscopic Surgical Procedure for Ligation and Online Distinction of Abnormal Manipulation for Thread Knotting. International Journal of Clinical Medicine,06,887-898. doi: 10.4236/ijcm.2015.612116