SevOne. Storage Solution for NetFlow Data (FlowDB).
6-20x faster generation of raw non-aggregated reports
Challenges
Due to the amount of data collected by the client’s solution, they were either having issues executing reports based on raw NetFlow data, or it took a fair amount of time to generate such. The inability to create raw NetFlow reports resulted in a lack of granularity, which was critical, especially for implementing billing solutions. Without these reports, SevOne’s clients could not obtain an accurate global overview of all devices in their network. The existing solution could not offer drill-down capabilities in large data sets.
Network Operations teams were under increasing pressure to keep the network performing at its best. The lack of an alerting system at their disposal prevented them from detecting and responding promptly to current and potential issues. They needed to reduce the Mean time to repair (MTTR) and solve performance problems faster.
Our R&D team was asked to find a way to tackle these challenges.
Solution
We developed a custom database engine for NetFlow data to improve raw data storage and facilitate faster report generation. We achieved this by creating a structure that organizes the data most efficiently and conveniently for reporting purposes. We also utilized reports requests parallelism and data compression and increased memory efficiency. Using raw data for analytics brings a new level of accuracy, including real-time reporting.
To further enhance the performance of the SevOne network monitoring solution, we developed a NetFlow Alerting system that uses thresholds defining acceptable network performance. This allowed SevOne users to get threshold breach alerts based on a predefined set of rules during data processing. With this information, the operation teams could monitor for conditions that warrant investigation, such as backup traffic occurring during business hours, multicast traffic rates that drop below the defined bit rate, etc.