To convert invisible, unstructured and time-sensitive machine data into information for decision making is a challenge. Tools available today handle only structured data. All the transaction data are getting captured without understanding its future relevance and usage. It leads to other big data analytics related issue in storing, archiving, processing, not bringing in relevant business insights to the business user. In this paper, we are proposing a context aware pattern methodology to filter relevant transaction data based on the preference of business.
There are varieties of diversified portfolio of applications getting deployed in the Enterprise Infrastructure space, and each application has a different trend of arrival patterns which generates machine data that need to be captured and processed to gain business insights. The problem starts from collection and filtering and processing of the data becomes difficult due to the rate in which the data getting generated is huge [
The purpose of this Context Aware Transactional framework is to categorize the patterns and filter the machine data based on the relevance of each transaction getting performed. Based on the filtered data, the enterprise can concentrate on how to effectively bring out the business insights within the application by not spending too much on the cost aspects with respect to the data storage and processing [
The objective of this paper is to propose a methodological approach to implement a non-intrusive component which can be plugged into the existing enterprise infrastructure layer to bring out all the insights business wants by capturing only the relevant business oriented machine data [
Context aware filtering is the process of recognizing the machine data based on the pattern. The patterns are application specific based on specifics like business rules, database access related, and external interfaces. The methodology is built in such a way that this component can be deployed as non-intrusive into any of the enterprise layer to generate data insights. The high level logical steps involved in context aware filtering are shown in
In this phase preserving of existing business and technical knowledge are captured will be utilized and key characteristics of the existing application are captured and stored in the master Meta data in the repository.
In this phase pattern matching will be applied on the machine data, and the recognized data are retrieved and stored in the desired repositories. The generation of machine data is in multiple phases, so the pattern recognizer will be a logical independent component which can be made as non-intrusive deployment whenever any transaction happens.
In this phase the pattern filter gets applied on to the pattern recognized machine data which properly filters the relevant and store it in the database for further processing.
In this phase the pattern extractor and visualizer helps the enterprise to devise the strategy based on the business rule to extract data.
The high level proposed architecture is explained in
This component will act as a listener component to the channels. The channel sends the request based on the request the listener component intelligently forms the triggering point for the Pattern builder to trigger its operations. It acts as a signal sender for the next component to act upon.
This component will also act as intelligent interpreter and filters out the rules present across applications.
This component retrieves the composed pattern from pattern composer and builds a searchable pattern format
which can be directly applied on to the enterprise contextual data getting captured by the Channel Listener component. This component also deals with the intelligent interpretation of the contextual data from the Enterprise with multiple dimensions and variety. The smartness is built into the component itself and different scoring algorithms based implementation is leveraged to achieve the same.
This component applies the filter rules and it has to interact much with the infrastructure component. The filtered data after the appliance of rules will be streamed to Pattern processor component
This component retrieves the filtered contextual data and parses efficiently to aggregate and assemble the data as per the requirements
This component provides Authorization, Authentication, Logging, Security etc. and it’s visible to all the other components in this framework. This leverages most of the open source libraries for its operation.
A portion of transaction log file used for the experimentation has been shown in
The API and the corresponding functions are explained in
Properties details and context aware details to filter the relevant data to connect to twitter and also the extraction based on filtering option given in context Data Pattern parameter is defined in
The report analysis for visualization of contextual relevant data and also the percentage of savings before and after context aware filtering is shown in
Input: Transaction log file |
---|
a) Listen to TCP/IP channels; b) New pattern repository builder pb; c) pb. build pattern (); d) pb. persist patterns (); e) Data: machine data; f) Getconnection (); g) List patterns list: = readall patterns (); h) New pattern composer pc; i) List redundant patterns = pc. get redundent patterns (patterns list); j) New pattern filter pf; k) List filtered pattern list = pf. filter patterns (patterns list, redundant patterns); l) Data = pf. apply filter (data, filtered pattern list); m) Data=aggregate & assemble (data); n) Persist data (data); o) Release connection (); p) New pattern extractor pe; q) pe. show log report (); |
Output: Contextual relevant data consolidated for gaining business insights |
Most of the analytics application captures all the data for future analytics, but the key aspect is to bring in the context aware filtering on the data getting generated from multiple sources of applications. This eases out the analytics complexity on the enterprise and brings in better prospects towards visualization of data. The complexity and cost factor involved in data management like storing, archiving, backup, recovery, etc. can be reduced by this framework.