Design patterns are object oriented software design practices for solving common design problems and they affect software quality. In this study, we investigate the relationship of design patterns and software defects in a number of open source software projects. Design pattern instances are extracted from the source code repository of these open source software projects. Software defect metrics are extracted from the bug tracking systems of these projects. Using correlation and regression analysis on extracted data, we examine the relationship between design patterns and software defects. Our findings indicate that there is little correlation between the total number of design pattern instances and the number of defects. However, our regression analysis reveals that individual design pattern instances as a group have strong influences on the number of defects. Furthermore, we find that the number of design pattern instances is positively correlated to defect priority. Individual design pattern instances may have positive or negative impacts on defect priority.
Design patterns are object oriented software design practices for solving common design problems. Design patterns provide the reuse of proven designs and architectures rather than the reuse of code. The most well-known design pattern literature in software engineering is the book published by Gang of Four (GoF) in 1995 [
Design patterns have many benefits like improving software quality, understandability, flexibility, reusability, extensibility, maintainability, and reducing development time [
Design patterns have many benefits to software quality, but they also have disadvantages as mentioned by GoF, so they should be applied with care. They bring expert knowledge but the incorrect integration and use of the chosen pattern can overcomplicate the design and make maintenance harder. Bieman et al. [
Some empirical studies also found that the use of design patterns may correlate with higher defect rate and more extensive changes. The previous studies by Vokáč [
In this research, we study the impact of design patterns on software defects by mining the repositories of open source software projects. Mining software repositories have recently emerged as a promising means to understand software engineering practices [
In our study, we first select 26 open source Java software projects. We then extracted metrics of these projects from their repositories including source code repositories and bug tracking systems. The metrics include data about design patterns and defects. The metrics are then put into a metric database for analysis using correlation and regression.
Our study extends previous studies by including many more software projects and using more comprehensive metrics. Previous studies [
The rest of the paper is organized as follows. Section 2 discusses related work. Section 3 introduces research problems and our proposed approach, along with its implementation including the system components and the process. Section 4 presents our study’s results with its design and analysis. Section 5 discusses threats to the validity of our study. Section 6 concludes our study along with discussion on future work.
In this section, we first mention design pattern related surveys and mapping studies. Second, we discuss related work on design patterns and software quality in general. We then describe previous studies on design patterns and software defects.
The following surveys and mapping studies examine the use of design patterns.
Ampatzoglou et al. [
Zhang and Budgen [
In another study, Zhang and Budgen [
Bafandeh Mayvan et al. [
In these studies, the effect of design patterns on software quality, such as maintainability, reusability, testability, extendibility, and so on, are examined. The software quality is measured by code metrics which include many metrics for object-oriented software, such as the number of classes, the depth of inheritance tree, average number of method in a class, and so on.
Ampatzoglou et al. [
Ampatzoglou et al. [
Elish [
Di Penta et al. [
Huston [
Hsueh et al. [
Posnett et al. [
Feitosa et al. [
Izurieta and Bieman [
In another study, Izurieta and Bieman [
Ampatzoglou et al. [
Ampatzoglou et al. [
Aversano et al. [
Some studies used controlled experiments to evaluate the quality of software solutions using design patterns.
Prechelt et al. [
Vokáč et al. [
Ng et al. [
Feitosa et al. [
Sahin et al. [
Bunse and Stiemer [
Litke et al. [
In the following studies, the use of design patterns and their effects on software defects is investigated.
In an early paper, Vokáč [
Aversano et al. [
Gatrell et al. [
Elish and Mohammed [
Ampatzoglou et al. [
In summary, there have been many studies on effect of design pattern on software quality. Some of these previous studies investigated the relationship between design patterns and software defects. They focused on defect rate related to design pattern classes. In their studies, software repositories were examined to identify defects related design pattern classes. Like previous studies on design pattern and software defects, we also extract metrics from software repositories. Unlike previous studies, we use a lot more open source software projects except [
The goal of this study is to understand the effect of design patterns on software quality. As software quality is directly related to software defects, we use metrics related to software defects as measurements of software quality. In this study, we investigate the relationship between design patterns and software defects.
The specific research problems we try to tackle are explained in Section 3.1. Section 3.2 describes our research approach to answer these research questions. The software metrics used in our research are summarized in Section 3.3.
To limit the scope of this study, we examine two categories of defect metrics: the first related to the number of defects and the second related to the priority of defects. As a consequence, we design two groups of investigations. In the first group, we investigate if design pattern instances are related to the number of defects. In the second group, we investigate if design pattern instances are related to the priority of defects.
In the first group of investigations, we examine the relationship between design pattern instance metrics and defect number metrics. We first examine if the
Reference | # of Projects | # of Design Patterns | Defect Data Source |
---|---|---|---|
Vokáč [ | 1 | 5 | CVS |
Aversano et al. [ | 3 | 11 | CVS and SVN |
Ampatzoglou et al. [ | 97 | 11 | Not specified |
Gatrell et al. [ | 1 | 10 + 3 non GoF | Source code commits |
Elish and Mohammed [ | 5 | 17 | CVS |
This paper | 26 | 12 | Bug tracking systems |
number of design pattern instances in a project is correlated with the number of software defects in the project. In other words, does a software project with more design patterns instances have more defects? We perform various correlation analyses of design pattern metrics and defect number metrics. The design patterns metrics include individual design pattern instances as well as all design pattern instances together. The defect number metrics include the number of defects and defect rates (number of defects divided by line of code, number of defects divided by number of classes).
Next, we use regression to further examine how much design patterns instance metrics affect the defect number and defect rates. We investigate the effect of individual design pattern instances on the defect number and defect rates. We perform regression analysis on design pattern metrics and defect number metrics to see how much variations in defect number and defect rates are explained by the design pattern instance metrics.
In the second group of investigations, we examine design pattern instance metrics and defect priority metrics. We first investigate if the number of design pattern instances in a project is correlated to the priority of software defects in the project. In other words, does a software project with more design patterns have defects of higher priority? We also examine the individual design pattern instances and their correlation with defect priority. Also, we use defect rates where the number of defects is divided by line of code and by number of classes, respectively.
As in the first group, we use regression analysis to further examine how much design pattern instances affect the defect priority. We perform regression analysis on design pattern instance metrics and defect priority metrics to see how individual design pattern instances affect the defect priority. Again, the analysis is repeated using defect rates where the number of defects is divided by line of code and by number of classes, respectively.
To answer the research questions listed in the previous section, we collect data from two kinds of software repositories: bug tracking system and source code repository. The metrics are then calculated from the data and are analyzed to understand the relationships between design patterns and defects.
In this section, the system we build to extract and analyze metrics data is explained. We first introduce the system architecture and the process. We then explain each component of the system and the tools that are used in the components.
and defect metrics from the data extracted by the design pattern detector and the bug report examiner and store them into a database. The metrics database is used by the metrics analyzer for various analyses.
Our study is carried out in four steps. In the first step, we select open source software projects and collected source codes and bug reports from the software repositories of these projects. In the second step, design pattern data and defect data are extracted from the codes and the reports by the design pattern detector and the bug report examiner, respectively. In step three, metrics are calculated from the data collected in the previous step and are compiled into a database, to be analyzed by the data analyzer in step four.
The components of the system in
・ Design Pattern Detector
The design pattern detector identifies design pattern instances in source codes. In our approach, we use the design pattern detector developed by Tsantalis [
・ Bug Report Examiner
A bug report contains information related to a software defect, such as severity, priority, type, status, comment, and so on. The bug report examiner scans the bug reports in the bug track systems and collects defect data from these reports.
The bug report examiner is implemented using a tool called Bicho [
・ Metrics Calculator
The metrics calculator computes various metrics related to design patterns and defects from the data extracted by the design pattern detector and the bug report examiner. The metrics are described in Section 3.3.
・ Data Analyzer
The data analyzer examines the metrics extracted by the other three components. We use correlation analysis and regression analysis in examining the metric data. The data analyzer is implemented using IBM’s SPSS software package.
The metrics we calculate from the software repositories include design pattern instance metrics and defect metrics.
Design pattern instance metrics are software metrics related to design pattern instances and software size. There are two groups of metrics, the first related to the total number of design pattern instances in a software project, and the second related to the number of individual design pattern instances in a project.
There are two versions of implementation for the Proxy design pattern. They are denoted as Prx and Prx2. Instances of Adapter and Command design patterns have the same structure so they are hard to distinguish. Their instances are combined as AC. Similarly, the State and Strategy design patterns are similar in implementation and their instances are combined into StSt.
The metrics in
Metric | Description |
---|---|
NDPI | Number of design pattern instances |
NDPI/LOC | Number of design pattern instances divided by line of code |
NDPI/NOC | Number of design pattern instances divided by number of classes |
Metric (Number of Instances) | Design Pattern |
---|---|
FM | Factory Method |
Pro | Prototype |
Sin | Singleton |
AC | Adapter/Command |
Comp | Composite |
Dec | Decorator |
Prx | Proxy version 1 |
Prx2 | Proxy version 2 |
Ob | Observer |
StSt | State/Strategy |
TM | Template Method |
V | Visitor |
For example, FM/LOC and FM/NOC represents the number of Factory Method instances divided by line of code and by number of classes, respectively.
Bug tracking systems usually include information about defects, such as priority, type, severity, status, resolution, comments, description, submitter, and assigned to someone or not, etc. In our study, we use the number of defects and defect priority.
・ Number of Defects, Defect Rate by LOC, and Defect Rate by NOC: the total number of defects detected in a software project (Nbugs), Nbugs divided by the lines of code, and Nbug divided by the number of classes.
・ Defect priority: a categorization of software defect to signify the degree of urgency to be fixed. A lower number indicates a higher priority to fix a defect.
We selected 26 open sources software projects for our study. The projects are all written in Java. Moreover, to ensure we have enough data for a project, all selected project must have been around for more than 3 years.
They should be active which means they have been updated in the last 3 months. Bug tracking system of the projects should be available to public and they should provide bug related information like priority, description, and comments. Also the tool (Bicho) [
The projects are summarized in
In this section, we investigate the relationship between the number of design pattern instances (DPIs) and the number of defects in a project. We first look at total DPIs and the number of defects, followed by individual DPIs and the number of defects.
We first compute the correlation between the total number of DPIs and the total number of defects. The Pearson correlation shows that at project level, the total number of DPIs in a project is not correlated to the number of defects in
Metric | Description |
---|---|
Nbugs | Number of defects |
Nbugs/LOC | Number of defects divided by line of code |
Nbugs/NOC | Number of defects divided by number of classes |
AP | Average priority of defects |
Project | LOC | NOC | Date Established |
---|---|---|---|
AOI | 107,489 | 567 | 11/20/2000 |
DrJava | 90,851 | 795 | 01/15/2002 |
EXist | 339,734 | 2601 | 01/24/2003 |
FreeCol | 101,043 | 768 | 01/02/2002 |
Gate | 216,822 | 1611 | 07/14/2005 |
JBPM | 134,623 | 1215 | 01/02/2003 |
JDT.Core | 327,283 | 1357 | 06/01/2001 |
JEdit | 117,174 | 969 | 12/06/1999 |
JFreeChart | 146,176 | 953 | 11/27/2000 |
JHotDraw | 81,700 | 685 | 10/10/2000 |
JMol | 189,793 | 795 | 03/25/2001 |
JMRI | 479,457 | 4190 | 05/06/2001 |
Joda-Time | 79,585 | 412 | 12/15/2003 |
JTDS | 41,606 | 136 | 08/09/2001 |
Jump | 143,803 | 1448 | 08/30/2004 |
JXplorer | 47,032 | 269 | 06/09/2002 |
LibrePlan | 182,214 | 1422 | 01/04/2009 |
MegaMek | 284,317 | 1924 | 02/19/2002 |
Mobac | 34,499 | 534 | 08/27/2008 |
OmegaT | 55,117 | 608 | 11/28/2002 |
PMD | 40,145 | 751 | 06/21/2002 |
RadipMiner | 319,035 | 3395 | 08/25/2004 |
SubSonic | 26,534 | 315 | 12/10/2004 |
Vassal | 144,168 | 1459 | 09/19/2003 |
XProcess | 334,056 | 2913 | 01/23/2009 |
Yaoqiang | 71,929 | 512 | 05/26/2010 |
that project, with a correlation coefficient of 0.103 and a p-value of 0.618.
Since the number of DPIs and the number of defects may be related to project size, we then normalize the number of DPIs and the number of defects by line of code (LOC) and number of classes (NOC). The number of DPIs and the number of defects are divided by LOC, respectively, and their correlation is calculated. The analysis is repeated using NOC as the divisor. The results are similar to these without normalization. Using LOC as the divisor, the correlation coefficient is −0.138 and the p-value is 0.501. Using NOC as the divisor, the correlation coefficient is −0.115 and the p-value is 0.574.
It is obvious that there is no correlation between the total number of DPIs and the number of defects. Normalized number of DPIs and normalized number of defects does not correlation either. The correlation analysis results show that, as a total, there are no correlation between the number of DPIs and the number of defects.
To further investigate the correlation between the number of DPIs and the number of defects, we look at the number of instances of individual design patterns. We perform correlation analysis between the number of defects and the number of instances of individual design patterns, as listed in
We repeat the correlation analysis with normalized number of defects and normalized number of instances of individual design patterns, i.e., they are both divided by LOC and NOC, respectively.
As shown in
Defect Metrics | DPI Metric | Correlation Coefficient | p-value |
---|---|---|---|
NBugs | NDPI | 0.103 | 0.618 |
Nbugs/LOC | NDPI/LOC | −0.138 | 0.501 |
Nbugs/NOC | NDPI/NOC | −0.115 | 0.574 |
Instances of Individual Design Pattern | Correlation Coefficient | p-value |
---|---|---|
FM | 0.256 | 0.206 |
Pro | 0.079 | 0.703 |
Sin | −0.042 | 0.840 |
AC | 0.021 | 0.918 |
Comp | 0.098 | 0.636 |
Dec | 0.192 | 0.348 |
Ob | −0.017 | 0.933 |
StSt | 0.093 | 0.650 |
TM | 0.195 | 0.339 |
V | 0.175 | 0.393 |
Prx | 0.679 | 0.000 |
Prx2 | 0.140 | 0.494 |
Instances of Individual Design Pattern normalized with LOC | Correlation Coefficient | p-value |
---|---|---|
FM/LOC | 0.108 | 0.601 |
Pro/LOC | −0.177 | 0.386 |
Sin/LOC | 0.011 | 0.958 |
AC/LOC | −0.185 | 0.365 |
Comp/LOC | −0.133 | 0.517 |
Dec/LOC | 0.039 | 0.851 |
Ob/LOC | −0.159 | 0.438 |
StSt/LOC | −0.168 | 0.412 |
TM/LOC | 0.259 | 0.201 |
V/LOC | 0.354 | 0.076 |
Prx/LOC | 0.204 | 0.317 |
Prx2/LOC | 0.195 | 0.340 |
Instances of Individual Design Pattern normalized with NOC | Pearson Correlation | p-value |
---|---|---|
FM/NOC | 0.071 | 0.730 |
Pro/NOC | −0.051 | 0.803 |
Sin/NOC | −0.134 | 0.513 |
AC/NOC | −0.158 | 0.442 |
Comp/NOC | −0.102 | 0.620 |
Dec/NOC | −0.026 | 0.898 |
Ob/NOC | −0.188 | 0.357 |
StSt/NOC | −0.109 | 0.595 |
TM/NOC | 0.220 | 0.279 |
V/NOC | 0.192 | 0.348 |
Prx/NOC | 0.444 | 0.023 |
Prx2/NOC | 0.001 | 0.996 |
The results are similar to original numbers presented in
However, even though there is little correlation between the number of instances of individual design patterns and the number of defects, it is possible that a combination of them may affect significantly the number of defects. We perform linear regression analysis using the number of defect as dependent variable and the number of instances of individual design pattern as independent variables. The results show a strong relationship with R2 value of 0.846 and p-value of 0.002.
It shows that six design patterns have a p-value below 0.05, AC (Adapter/Command), Ob (Observer), StSt (State/Strategy), TM (Template Method), Proxy, and Proxy 2. The number of instances of these six design patterns significantly influences the number of defects. Especially, Adapter/Command and Observer instances have a negative impact the number of defects. This means that as the number of Adapter/Command and Observer instances increases, the number of defects decreases.
One possible explanation is that the use of Adapter/Command and Observer design patterns improves design and code such that there are fewer defects. Our finding on Observer pattern similar to [
Independent Variable | Standardized Coefficients | p-value |
---|---|---|
(Constant) | 0.021 | |
FM | 0.116 | 0.770 |
Pro | −0.393 | 0.258 |
Sin | −0.458 | 0.114 |
AC | −1.340 | 0.002 |
Comp | −0.105 | 0.618 |
Dec | −0.502 | 0.124 |
Ob | −0.581 | 0.007 |
StSt | 0.923 | 0.014 |
TM | 1.370 | 0.015 |
V | −0.181 | 0.361 |
Prx | 0.841 | 0.000 |
Prx2 | 0.445 | 0.024 |
In this section, we investigate the relationship between DPIs and defect priority by analyzing DPI metrics and defect priority metric. Of the 26 projects listed in
The projects use two different scales for their priority value. Some use a 1 to 5 scale for their priority value. Others use a 1 to 9 scale. We first use the min-max normalization to linearly convert those projects using the 1 to 9 scale in to a scale of 1 to 5. For example, a priority of 4 in a 1 to 9 scale is converted into (4 − 1)/(9 − 1) * (5 − 1) + 1 = 2.5. The average priority of defects in every project is then calculated.
We use average priority, denoted as AP, in the reminder of this section.
First, we perform correlation analysis between the number of total DPIs and AP. The Pearson correlation between the number of total DPIs and AP is 0.645 with a p-value of 0.000. It is obvious that there is at least moderate positive correlation between the number of total DPIs and AP. In other words, as the number of total DPIs in a project increases, its defects tend to have larger priority values, which mean low priority of the defects.
We also calculated correlation between AP and normalized number of total DPIs, using LOC and NOC, respectively. The results are similar to these without normalization. The results are summarized in
Next, we perform correlation analysis between the number of instances of individual design patterns and AP.
From
We then perform linear regression with average priority as dependent variable and the number of instances of individual design pattern as independent variables.
DPI Metric | Defect Priority | Correlation Coefficient | p-value |
---|---|---|---|
NDPI | AP | 0.645 | 0.000 |
NDPI/LOC | AP | 0.583 | 0.002 |
NDPI/NOC | AP | 0.605 | 0.001 |
Instances of Individual Design Pattern | Correlation Coefficient | p-value |
---|---|---|
FM | 0.430 | 0.032 |
Pro | 0.545 | 0.005 |
Sin | 0.298 | 0.149 |
AC | 0.572 | 0.003 |
Comp | 0.033 | 0.874 |
Dec | 0.160 | 0.445 |
Ob | 0.315 | 0.124 |
StSt | 0.593 | 0.002 |
TM | 0.635 | 0.001 |
V | −0.022 | 0.915 |
Prx | −0.155 | 0.459 |
Prx2 | −0.241 | 0.245 |
Independent Variables | Standardized Coefficients | p-value |
---|---|---|
(Constant) | 0.000 | |
FM | 0.058 | 0.847 |
Pro | 0.438 | 0.106 |
Sin | −0.368 | 0.096 |
AC | 0.768 | 0.011 |
Comp | −0.193 | 0.265 |
Dec | −0.037 | 0.875 |
Ob | −0.088 | 0.546 |
StSt | 0.278 | 0.303 |
TM | 0.080 | 0.832 |
V | 0.139 | 0.348 |
Prx | −0.313 | 0.016 |
Prx2 | −0.444 | 0.009 |
number of instances of individual design patterns affect average priority.
Form
We also perform linear regression using number of design pattern instances normalized by LOC and NOC, respectively.
Form
The regression has an R2 value of 0.906 and a p-value of 0.000. It is obvious from
For the three cases of linear regression analysis on instances of individual design patterns, using number of instances, number of instances divided by LOC, and number of instances divided by NOC, respectively, we see some similarities and some differences. All three shows that instances of Adapter/Command have positive effect on average priority and instances of Proxy 2 have negative effect
Independent Variables | Standardized Coefficients | p-value |
---|---|---|
(Constant) | 0.000 | |
FM/LOC | −0.021 | 0.888 |
Pro/LOC | 0.603 | 0.001 |
Sin/LOC | −0.345 | 0.003 |
AC/LOC | 0.507 | 0.009 |
Comp/LOC | −0.571 | 0.001 |
Dec/LOC | 0.035 | 0.810 |
Ob/LOC | −0.091 | 0.380 |
StSt/LOC | 0.380 | 0.040 |
TM/LOC | 0.078 | 0.596 |
V/LOC | −0.003 | 0.982 |
Prx/LOC | −0.187 | 0.147 |
Prx2/LOC | −0.348 | 0.018 |
Independent Variables | Standardized Coefficients | p-value |
---|---|---|
(Constant) | 0.000 | |
FM/NOC | 0.011 | 0.953 |
Pro/NOC | 0.670 | 0.000 |
Sin/NOC | −0.167 | 0.259 |
AC/NOC | 0.628 | 0.004 |
Comp/NOC | −0.556 | 0.001 |
Dec/NOC | 0.048 | 0.743 |
Ob/NOC | 0.003 | 0.978 |
StSt/NOC | 0.241 | 0.198 |
TM/NOC | 0.081 | 0.577 |
V/NOC | 0.103 | 0.376 |
Prx/NOC | −0.374 | 0.013 |
Prx2/NOC | −0.311 | 0.022 |
on average priority. Also, instances of Prototype show positive impact on average priority when using normalized number of instances. Similarly, instances of Composite show negative impact on average priority when using normalized number of instances.
Based on above investigations, we conclude that the number of instances of several design patterns has significant effect on the number of defects and their priority.
There are several threats to the validity of our study. We discuss the serious threats in the following.
1) Not all design patterns are detected.
The design pattern detector used in our study [
One way to solve the problem is to improve the design pattern detector so that it can find more design patterns. We are actively looking for a more powerful design pattern detector.
2) Defects are accumulated over time.
The defect data are extracted from the projects’ bug tracking repositories. A project’s bug tracking repository contains all defects reported since the initiation of the project. Since the projects have different initiation date as presented in
3) Priority values may not be consistent among the projects.
As we described in Section 4.2, the projects in our study uses two different scales for defect priority. Some use a scale of 1 to 5 and others use a scale of 1 to 9. We use min-max normalization to transform 1 to 9 scale to 1 to 5 scale. This would introduce some inaccuracy since priority values are not necessarily assigned linearly, e.g., a priority value of 4 does not necessarily mean twice more urgent compared to a priority value of 8. Even for the projects using the 1 to 5 scale, they may not assign priority values consistently, e.g., a priority value 5 in a project may be assigned a priority value 4 in another project. A universal guideline for assigning priority values would help to eliminate this kind of discrepancy.
There are some other threats such as incomplete data in repositories, different application domains of projects, and different skill levels and capabilities of developers. We think these threats are minor and we do not discuss them in detail.
In this study, we investigate the relationship of design patterns and software defects in a number of open source software projects. In particular, we analyze relationships between design pattern metrics and software defect metrics using correlation and regression analysis. In our first group of investigations, we find there is little correlation between the total number of design pattern instances and the number of defects. The number of instances of individual design pattern also does not correlate with the number of defects, except for the Proxy pattern. However, the number of instances of individual design patterns as a group has strong influences on the number of defects. Especially, the number of instances of Adapter/Command, Observer, State/Strategy, Template Method, and Proxy patterns has significant impact on the number of defects.
In our second group of investigations, we found moderate positive correlation between the number of design pattern instances and average defect priority. Moreover, the number of instances of Prototype, Adapter/Command, State/Strategy, and Template Method, is positively correlated with average priority. When considered as a group, the number of instances of Adapter/Command and Proxy 2 has been found to have significant effect on average priority. Prototype and Composite instances, when divided by LOC or NOC, also show significant effect on average priority.
Design patterns have been widely used in software development. Our research extends previous studies on design pattern and software defects by using software defect metrics from bug tracking repositories, in particular, defect priority. There are many ways to extend our research in the future. We discuss some possible future work below.
In the future studies, we can focus finer level of granularity, i.e., role level of design pattern instances and classes.
It is obvious that our study can be extended to use other defect metrics, such as defect fixing time. It will be interesting to investigate how the number of design pattern instances affects other defect metrics.
The design pattern detector used in our study can only find 12 design patterns. One worthwhile research direction is to develop more powerful pattern detectors that can find more design patterns accurately and efficiently.
Since defects are related to complexity, another interesting research topic is to investigate the impacts of design pattern instances on software complexity. We are investigating design pattern instances and their impacts on software complexity metrics.
Onarcan, M.O. and Fu, Y. (2018) A Case Study on Design Patterns and Software Defects in Open Source Software. Journal of Software Engineering and Applications, 11, 249-273. https://doi.org/10.4236/jsea.2018.115016