^{1}

^{1}

^{*}

^{2}

^{3}

In this paper, we consider a cost-based extension of intrusion detection capability (
*C _{ID}*). An objective metric motivated by information theory is presented and based on this formulation; a package for computing the intrusion detection capability of intrusion detection system (IDS), given certain input parameters is developed using Java. In order to determine the expected cost at each IDS operating point, the decision tree method of analysis is employed, and plots of expected cost and intrusion detection capability against false positive rate were generated. The point of intersection between the maximum intrusion detection capability and the expected cost is selected as the optimal operating point. Considering an IDS in the context of its intrinsic ability to detect intrusions at the least expected cost, findings revealed that the optimal operating point is the most suitable for the given IDS. The cost-based extension is used to select optimal operating point, calculate expected cost, and compare two actual intrusion detectors. The proposed cost-based extension of intrusion detection capability will be very useful to information technology (IT), telecommunication firms, and financial institutions, for making proper decisions in evaluating the suitability of an IDS for a specific operational environment.

In recent times, the ease of application of computer systems and availability of internet services has dramatically changed the way businesses are transacted on the global scene. This has led to rapid developments in the field of computing and e-business. Consequently, the risk of unwarranted access to computer systems has increased in proportionate measures. There is no denying the fact that several cases of computer security attacks are reported daily across the globe. This calls for a serious concern for organizations and corporate bodies to decisively step up the game of securing computer systems from intrusion. In order to ameliorate this ugly incident, individuals and organizations are currently deploying passphrases, antivirus applications, and firewall to protect networks and sensitive data. Unfortunately, these algorithms have limited capabilities to secure information. For example, passwords of such algorithms can be compromised [

Although there have been many research and development efforts in IDS, appropriate evaluation of IDS is still a major problem. Some of the problems include 1) no standard benchmark, which makes comparison of IDS difficult, 2) dynamic changing environment, making it difficult to establish a fully descriptive baseline, 3) issues with empirical evaluations (using data-set to test IDS) as there will always be a difference between data-set and real scenario.

However, a key problem in intrusion detection is how to determine the essential metrics to appropriately evaluate IDS in objective terms, especially how to ascertain the capabilities of the IDS to categorise events as normal or intrusive [

In practice, a unifying metric could possibly be deployed to assist the administrator of a particular network in the choice of an appropriate detector from a pool of systems or enhance an existing configuration settings of a known intrusion detector system for a defined network environment [_{ID} is a single unified metric proposed by Gu et al. [_{ID}, it becomes very easy to ascertain or determine the particular operating point that gives the minimum level of uncertainty about a defined input event that occurred due to intrusion or not is determined. However, the C_{ID} metric does not take into consideration the expected cost associated with that operating point. In addition, it could be quite expensive to quantify in practical terms of interest like false alarm and detection rates, how to minimize the uncertainty of an attack.

Thus, this study presents a cost-based extension of the intrusion detection capability (C_{ID}). Determining the corresponding costs complements and increases the scope of C_{ID} as an evaluation metric rather than just diminishing the uncertainty of the intrusions as proposed in [_{ID} for the optimal operating point. This will provide an explanation for the IDS optimal point in terms of the least expected cost. Thus, the cost of tuning the detector to the optimal point will be determined. Another objective is to determine the optimal operating point of an IDS in terms of cost. This defines the ability of the IDS to classify events at the least expected cost. We then demonstrate how the proposed metric facilitates the comparison of IDSs.

In particular, our contributions include the following: 1) a mathematical formulation is presented using information theory and based on this formulation; 2) a package for computing the intrusion detection capability of IDS, given certain input parameters is developed; 3) to include cost function in C_{ID}, a decision tree approach is used as a method of analysis for evaluation; 4) the cost-based extension is used to select optimal operating point, calculate expected cost and compare two actual intrusion detectors. Finally, the results in this paper are compared with the results of related works reported in [

The remainder of this paper is described as follows. Section II summarizes related works on intrusion detection. Section III discusses the theoretical background of intrusion detection as it relates to information theory and the associated cost. Section IV presents the system architecture for software intrusion detection evaluation scheme (SIDES). Section V presents the results with some discussions on changing some of the parameters used in the evaluation. Section VI concludes the paper and states useful contributions as well as recommendations for future studies.

Recently, there has been an unprecedented growth in technologies involving the use of computer applications. Consequently, this has given birth to rapid cases of denial of service attacks, proliferation of worms and virus attack, and increased activities of hackers have led to increased security concern at all levels of public and private-sector organizations. This has encouraged useful researches on IDS in recent years. In the existing literature, various models for IDS have been proposed based on architecture, fault tolerance, and mobile agent platforms. Some authors compared the distributed model architecture with the traditional centralized models and demonstrated that the future of IDS is pointing towards distributed or hybrid architecture [

In 1998, a study sponsored by DARPA was carried out at the Lincoln Laboratory of Massachusetts Institute of Technology. Prior to this study, not much information on intrusion detection systems is available in the open literature. The 1998 DARPA offline project actually opened up this interesting area of research following a detailed and elaborate report on the test of IDSs in a real world environment [

Gu et al. [

In a similar study [_{ID}), the expected cost, and the Bayesian detection rate were reviewed. The strengths and drawbacks of the individual performance metrics were investigated and analyzed in a closed form. In addition, a new IDS performance trade-off referred to as intrusion detection operating characteristics (IDOC) curves is introduced, and real world data were used to test the validity of the practical and simulated results.

In the same vein, Sallay et al. [

Authors in [

In [

On performance metric scorecard-based approach to the evaluation of IDS associated with wireless networks [

Authors in [

In a related study, Verma and Ranga [

In view of the foregoing, this study is aimed at developing a cost-based extension of the intrusion detection capability which has not been given a fair treatment in the existing literature.

Essentially, a quality IDS should be able to distinguish the events monitored (input data) as either intrusive or normal. Here, the IDS provide output information usually in form of alarms, that should give a true picture of the events being monitored. This means that the IDS should be able to detect whether there is actually an intrusion or not at any given time. Therefore, the task of a well designed IDS is to accept and analyze input data stream and give output alerts to show the presence of intrusion. On a careful analysis, each unit of an input data stream could be intrusive or normal and an IDS should be able to know and record these information for the attention of the administrator. This implies that the input of an IDS can be carefully modeled as a random variable X. For instance, if the value of X is high (X = 1), there is an intrusion and if X is low (X = 0), there is no intrusion and the traffic is normal.

Similarly, the output information of a typical IDS can be modeled as a random variable Y. Here, when Y = 1, it means that there is an alert of an intrusion, and when Y = 0, there is no alert information from the IDS. In a situation where it is assumed that an IDS output is available, and this corresponds to each input information to the IDS [

The probability that an intrusion event can be regarded as normal is represented by p ( Y = 0 | X = 1 ) . This is the false negative rate (FN), denoted as γ. Similarly, the probability that a normal event being misclassified as an intrusion is represented by p ( Y = 1 | X = 0 ) . This is the false positive rate (FP), denoted as α. From the foregoing, it can be assumed that X is the random variable depicting the IDS input and Y represents the random variable depicting the IDS output. Therefore, intrusion detection capability can be defined as:

C I D = I ( X ; Y ) H ( X ) (1)

Given what we know from our knowledge of information theory about mutual information, we can rewrite C_{ID} as Equation (2).

C I D = H ( X ) − H ( X | y ) H ( X ) (2)

Ideally, mutual information captures the decrease in the level of uncertainty of the input by evaluating the IDS output. From (2), it can be deduced that C_{ID} gives the ratio of the reduction of uncertainty of the IDS input given the IDS output. In practice, the value of C_{ID} is in the range of [0; 1]. Here, a large value of C_{ID} implies that the IDS is more capable of accurate classification of events.

The mutual information H(X) is defined as given in Equation (3), and the corresponding mutual information that an event has occurred H(X |Y) is given in Equation (4).

H ( X ) = − ∑ x p ( x ) log p ( x ) = − B log B − ( 1 − B ) log ( 1 − B ) (3)

H ( X | Y ) = − ∑ x ∑ y p ( x ) p ( y | x ) log [ p ( x ) p ( y | x ) ] p ( y ) = − B ( 1 − γ ) log P P V − B γ log ( 1 − N P V ) − ( 1 − B ) ( 1 − α ) log N P V − ( 1 − B ) α log ( 1 − P P V ) (4)

Substituting the equations, C_{ID} we obtain Equation (5).

C I D = − B log B − ( 1 − B ) log ( 1 − B ) − B ( 1 − γ ) log P P V − B γ log ( 1 − N P V ) − ( 1 − B ) ( 1 − α ) log N P V − ( 1 − B ) α log ( 1 − P P V ) (5)

In Equation (5), C_{ID} is intrusion detection capability, B is base rate, γ is false negative (FN) rate, α is false positive (FP) rate, PPV is positive predictive value and NPV is negative predicative value.

• Base rate (B): This is a measure of the environment in which IDS operates. When B = 0 or B = 1 (the input is 100% normal or 100% intrusion). In practice, it can be quite difficult to measure or control the base rate in an IDS. This is because the base rate is often seen as an operation parameter partly due to the fact that it is used to measure the IDS environment. The estimation of prior probabilities and base rate B has been presented in [

• False Positive (FP) Rate: This is the probability that the IDS outputs an alarm when there is no intrusion;

• False Negative (FN) Rate: This is the probability that an IDS does not output an alarm when there is an intrusion;

• Positive Predictive Value (PPV): This is the probability that there is an intrusion when the IDS output an alarm. That is, given IDS alarms, how many of them are real intrusions. It is mathematically expressed in Equation (6) [

P P V = B ( 1 − γ ) B ( 1 − γ ) + ( 1 − B ) α (6)

• Negative Predictive Value (NPV): This is the probability that there is no intrusion when the IDS does not output an alarm. That is given that there are no IDS alerts; does it mean that there are really no intrusions? Mathematically, it can be expressed in Equation (7) [

N P V = ( 1 − B ) ( 1 − α ) ( 1 − B ) ( 1 − α ) + B γ (7)

The receiver operating characteristics (ROC) curve shows a graphical illustration of the detection probability against false alarm rate. This means that the curve is capable of showing the probability of detection as seen by the detector at a defined false alarm rate. Alternatively, the curve shows the detector’s captured false rate at a stated probability of detection [

For a given operating point of a particular detector, it is possible to determine the expected cost by analyzing the outputs of the decision tree as illustrated in

As shown in the decision tree of

Conventionally, the decision tree is read from left to right [

C = C γ C α (8)

where C_{γ} refers to the cost of responding to the presence of intrusion and C_{α} is the cost of responding to an intrusion where there is actually no intrusion. In most practical scenarios, it can be assumed that the cost of correct responses to intrusion is negligibly small or zero [

1) Expected Cost Calculation: The formulae depicting the total probability as shown in (9) and (10) can be used to evaluate the probabilities of the detector’s reports [

p 1 = P ( N A ) = P ( N A | N I ) P ( N I ) + P ( N A | I ) = ( 1 − α ) ( 1 − p ) + γ p (9)

1 − p 1 = P ( A ) = P ( A N | I ) P ( N I ) + P ( A | I ) P ( I ) = α ( 1 − p ) + ( 1 − γ ) p (10)

The Bayes Theorem as reported in [

p 2 = P ( N I | N A ) = P ( N A | N I ) P ( N I ) P ( N A ) = ( 1 − α ) ( 1 − p ) p 1 = ( 1 − α ) ( 1 − p ) ( 1 − α ) ( 1 − p ) + γ p (11)

1 − p 2 = P ( I | N A ) = P ( N A | I ) P ( I ) P ( N A ) = γ p p 1 = γ p ( 1 − α ) ( 1 − p ) + γ p (12)

p 3 = P ( N I | A ) = P ( A | N I ) P ( N I ) P ( A ) = α ( 1 − p ) 1 − p 1 = α ( 1 − p ) α ( 1 − p ) + ( 1 − γ ) p (13)

1 − p 3 = P ( I | A ) = P ( A | I ) P ( I ) P ( A ) = ( 1 − γ ) p 1 − p 1 = ( 1 − γ ) p α ( 1 − p ) + ( 1 − γ ) p (14)

As shown in

At any operating point, the expected cost of operating the IDS is given in Equations (15) and (16):

C E X = p 1 min { C γ p , ( 1 − α ) ( 1 − p ) } p 1 + ( 1 − p 1 ) min { C ( 1 − γ ) p , α ( 1 − p ) } 1 − p 1 (15)

C E X = min { C γ p , ( 1 − α ) ( 1 − p ) } + min { C ( 1 − γ ) p , α ( 1 − p ) } (16)

In practice, the optimal operating point is described as the most suitable point achievable by the given IDS in terms of its intrusion detection capabilities, and minimization of the expected cost. Therefore, choosing an optimal operating point would be equivalent to the best choice of values for the parameters α and γ that can provide the desired least expected cost.

On the concept of base-rate fallacy, there seems to be a very large difference between the amounts of events seen as normal and the amount of intrusion events, which are very few. This huge difference can results in the generation of multitudes of false alarms. Here, fallacy maintains that due to the low probability of a real attack, especially when an IDS triggers an alarm, the probability of intrusion occurring could be very minimal. Furthermore, Gu et al. [^{−5}, unless stated otherwise.

Introducing the cost-based extensions on C_{ID} metric makes it achieve similar capability as ROC which integrates cost analysis and more practically beneficial, because the various operating points for the IDS will have an associated cost function [_{ID} at the least expected cost.

A mathematical formula as shown in Equation (5) is derived from an information theoretic point of view. To ease computation, a software intrusion detection evaluation system (SIDES) package is developed. The application provides a tool for calculating the intrusion detection capability C_{ID} of IDS using values from the

RESPONSE | ||
---|---|---|

Detectors Report | No Response (NR) | Response (R) |

No alarm (NA) | 1 − p 2 = C γ p ( 1 − α ) ( 1 − p ) + γ p | p 2 = ( 1 − α ) ( 1 − p ) ( 1 − α ) ( 1 − p ) + γ p |

Alarm (A) | 1 − p 3 = C ( 1 − γ ) p α ( 1 − p ) + ( 1 − γ ) p | p 3 = α ( 1 − p ) α ( 1 − p ) + ( 1 − γ ) p |

Receiver Operating Characteristics (ROC) reported in [_{ID}.

The algorithm for the SIDES package is as shown in Algorithm 1.

Using this application and the Receiver Operating Characteristics (ROC) values reported in [_{ID} is regarded as the best ID capability of the system and gives the most optimized operating point for the IDS. This is without recourse to the cost implication of operating at this optimal point. It is therefore necessary to attach a corresponding cost to this point.

To introduce cost function into C_{ID}, we adopt the decision tree analysis method [_{ID}. To have an acceptable trade-off between cost and capability, C_{ID} and C_{EX} values are plotted against α. The lowest point on the C_{EX} curve is matched with the highest point on the C_{ID} curve to determine the optimal operating point. More specifically, the observable deviations in the values of the expected cost could be very useful metric suitable for the comparison of two intrusion detectors.

Text fields were used to receive input; False Positive rate (α), False Negative rate (γ) and Base rate (B). “Reset values” button was designed to clear the input values. Calculate PPV and calculate NPV buttons were designed to calculate PPV and NPV respectively. Calculate C_{ID} button was designed to calculate the intrusion detection capability of the IDS given the initial inputs received. The results panel is designed to display the calculated values PPV, NPV and C_{ID}. Back home button was designed to take the user back to initial information window. Exit button was designed to close the package.

Results of C_{ID} values were computed using data extracted from two ROC curves reported in [_{1} and IDS_{2}, respectively. As in [_{1} ROC curve can be approximated as given in Equations (17) and (18).

1 − γ = 0.6909 × ( 1 − exp ( − 65625.64 α 1.19 ) ) (17)

1 − γ = 0.4909 × ( 1 − exp ( − 11932.6 α 1.19 ) ) (18)

Initial findings revealed that in 666,000 network session over a typical day, about 43 intrusion attempts were detected. Based on the assumption that the intrusion responses are achieved per session each time intrusion detectors are applied, the base-rate of intrusion is given as in (19).

B = Total number of intrusion attempts Total number of network sessions = 43 660000 = 6.52 × 10 − 5 (19)

Hence, we can estimate the probability of intrusion by the base-rate p = 6.52 × 10 − 5 . The results obtained from estimating the probability of intrusion are as depicted in

In practice, the point at which the highest intrusion detection capability and its threshold yields the most suitable threshold is referred to as the optimal operating point. Here, the optimal operating point for IDS_{1} occurs at α = 0.003, 1 − γ = 0.6807 corresponding to C_{ID} of 0.45567, while that of IDS_{2} occurs at α = 0.001, 1 − γ = 0.47112, and C_{ID} of 0.2403. From the foregoing, IDS_{2} achieves a better ID capability than IDS_{1}. By extension, comparing the two detectors based on the above analysis, we can conclude that IDS_{2} is better than IDS_{1}. However, this is without recourse to the cost of operating at the selected optimal point.

For the derivation of minimum expected-cost operating point, the decision tree as shown in ^{−}^{5} as in [

From _{ID} for IDS_{1} occurs at α = 0.0003, with a C_{ID} value of 0.4557. The minimum corresponding cost occurs at α = 0.0003, with an expected cost of 0.0211. Hence, the optimal operating point for IDS_{1} is 0.4557, 0.0211.

From _{ID} for IDS_{1} occurs at α = 0.0010, with a C_{ID} value of 0.2403. The minimum corresponding cost occurs at α = 0.0010, with an expected cost of 0.0355. Thus, the optimal operating point for IDS_{2} is 0.2403, 0.0355.

A comparative analysis of IDS_{1} and IDS_{2} is as shown in

IDS_{1} is a better detector with a C_{ID} of 0.2154 per session higher than the C_{ID} of IDS_{2} and an expected cost of 0.0144 per session less than that of IDS_{2}. The effect of the various input parameters on C_{ID} and C_{EX} is examined.

Ideally, an IDS may not be able to effectively control the base rate but it is a very important factor to be considered when presenting reports on intrusion detection capability because the base rate defines the environment of operation [_{ID} values were computed for different base rate values. The impact of different base rates on C_{ID} is as shown in

From ^{−4}, FP = 0.1 and FN = 0.1, In a case where the value of FP is decreased from 0.1 to 0.01, correspondingly, C_{ID} changes from 0.17 to 0.36. However, if FN is decreased by the same magnitude, the C_{ID} only changes from about 0.17 to 0.20. This shows that C_{ID} is more responsive to variations in false positive (FP) than false negative (FN). Hence, for low base rates, reducing FP will improve C_{ID} more than the same reduction in FN.

IDS_{1} | IDS_{2} | |
---|---|---|

α | 0.0003 | 0.0010 |

1 − γ | 0.3699 | 0.4711 |

C_{ID} | 0.4557 | 0.2403 |

C_{EX} | 0.0211 | 0.0355 |

The base rate B was fixed and for each value of FP (α), the FN (γ) values were varied and the corresponding C_{ID} calculated. A plot of False Positives rates against C_{ID} is shown in

From _{ID} changes from 0.44 to 0.37 (a difference of 0.07). However, when FP changes from 0.01 to 0.03 (a difference of 0.02), C_{ID} changes from 0.44 to 0.33 (a difference of 0.11). Hence, for low base rate B, little changes in False Positive result in large changes in C_{ID} as shown in

The base rate B is fixed while for each value of FN, the FP values are varied and the corresponding C_{ID} calculated. A plot of False Positive rates on against C_{ID} is as shown in

From _{ID} changes from 0.58 to 0.49 (a difference of 0.09). However, when FN changes from 0.1 to 0.15 (a difference of 0.05), C_{ID} changes from 0.58 to 0.54 (a difference of 0.04). Only large changes in FN will significantly affect C_{ID}. Hence, for low base rate B, only a large variation of FN (γ) have a significant effect on C_{ID} as shown in

As pointed out in [

From

Gu et al. [

In this paper, the concept of cost analysis in intrusion detection capability (C_{ID}) in a typical IDS environment with a low base rate is presented. Information theoretic analysis is used to model IDS and determine the intrusion detection capability of the detector. The decision tree method was introduced to compute the expected cost of operation for each operating point. Findings revealed that the optimal operating point is the point of intersection between the maximum C_{ID} and the expected cost curve. Cost-based extension of C_{ID} can be a very useful method to appropriately evaluate IDS to determine the type and capabilities of an IDS to be deployed in a particular network. This is of great importance in determining the suitability of an IDS in a given environment regarding the ability of the detector to classify events appropriately at the least expected cost. Future work could include investigating the impact of cost ratio on the expected cost. In addition, future studies can compare the results of this study with other functional forms of the ROC curves (power, polynomial and exponential curves). Furthermore, future studies could be directed towards a single metric mathematical model that combines cost analysis with C_{ID}.

The authors declare no conflicts of interest regarding the publication of this paper.

Imoize, A.L., Oyedare, T., Otuokere, M.E. and Shetty, S. (2018) Software Intrusion Detection Evaluation System: A Cost-Based Evaluation of Intrusion Detection Capability. Communications and Network, 10, 211-229. https://doi.org/10.4236/cn.2018.104017