The network shown in this image is monitored using monitoring probes. The Probe is connected via a TAP network and aggregation devices (network packet brokers) to the different interfaces of the network. These interfaces are logically and physically different which allows us to get a full view of what’s going on in the network. All these different interfaces are analyzed by probes.
A Probe is a device that can decode the traffic from the network and produce meta data records (XDR extended data records). The records are sent to a database. The database must be very powerful in terms of processing and storage to handle the huge amount of data. Such a system produces, even on a mid-size network, terabyte of data and billion of records per day.
Typically such a monitoring probe covers the L5 – L7 in the OSI stack.
Why is this done?
Monitoring is important to networks and this data can be used for several applications:
SLA against customers
SLA against other providers
SLA against network vendors
A good working monitoring system can save a lot of money and help to improve the performance.
What is the challenge and how big data and AI can be useful?
Typically such monitoring systems provide KPIs (key performance indicators). The KPI is a formula, which calculates some information on this XDR from the probes and provides these results in different graphs. The big issue with the KPIs is that they are predefined in a LAB and are not flexible. They are neither adopted on the customer’s network nor on the network changes.
Big networks behave like a living organism, which can be influenced by customers’ behavior, external factors like weather, and the transported contend. Therefore, a KPI is not an accurate way to show network behavior because it is unidimensional – too strict and not flexible at all.
Typically KPIs do not take account of knowing existing issues in networks like updates, weather and other external impacts. This is the reason why KPIs often produce a lot of false positive results. And most importantly, KPIs can show by definition only known issues, for instant dynamic correlated events could never be detected with KPIs.
The idea with big data is now to add intelligent, flexible and a multidimensional view to a network. With the help of databases like Hadoop, MongoDB it is now possible to add data from multiple sources to produce more useful reports.
Cubro now invests in AI and machine learning project to prove that Big Data in combination with AI is a useful approach to improve the mentioned issue.