Networking data visualisation is still too poorly understood and stigmatised as a "dumbing down" of data for Raffael...
Marty's taste. Marty, chief security strategist for Splunk, recently authored Applied Security Visualization, published by Addison-Wesley Professional.
"If you have a huge amount of data, you need to get a feel for what's really in there," Marty said. "If you have to go through 100,000 log files, by the time you get to line 100, you've forgotten what's on line 1."
Visualisations, by contrast, let networking professionals skim large amounts of data at once and quickly home in on outliers or other hard-to-detect trends.
Once people start thinking about the problem differently, they quickly reap the benefits, Marty said. The most challenging learning curve is at the beginning as IT professionals become comfortable moving from numerical data to graphical representations in the form of curves and color.
Part of the problem has also been a lack of tools to dive into data visualisation for the interested professional not quite ready to plunk down between $US2,000 and $US20,000 on specialised reporting and visualisation software with vendors like netForensics or CrossTec. Marty said that cost or complexity should not be a barrier in getting at least some of the benefits of data visualisation, and he's created an open source tool called AfterGlow and the website SecViz.org to help users get started.
So what is needed to begin getting useful work done with data visualisation? As when approaching almost all networking problems, good logs are critical, Marty said.
"I think a good starting point is collecting the logs [in one place]," he said. "A lot of people don't keep the correct logs or keep them around long enough, depending on what their use cases are."
Getting good logging data to start with should not be a challenge: Firewalls, applications, and intrusion detection platforms all can or are generating voluminous data, and it is just a matter of organising it in an easily accessible manner so that one set of logs can be properly correlated to another.
The next step is to develop a clear purpose.
"A lot of people say, 'I have these NetFlow logs, and I want to analyze them,' " Marty said. "Do you want to verify traffic against usage policy, though? Or look for attacks?" The better understanding networking pros have of what they are looking for, the more likely they are to find it, rather than just having some potentially interesting wallpaper, with little to show.
For inspiration, several SecViz users have posted their own graphs, used to detect everything from Worm attack patterns to their current IP table configuration.
Marty also suggests that networkers read his book or search online for tutorials to fit their needs.
"It's great grounds for exploring what's there, or even asking: 'I have this dataset; how do I go about analyzing it?' " he said.
To truly tap into visualisation's power, however, some professionals will want to consider tapping into a variety of scripting languages that can help them pull data in a more automated, particular way than Excel's user-friendly but finite controls allow. Tapping into a framework like ChartDirector means that a bit more technical learning is required, but more precise graphs can be scripted to update themselves as new data comes in.
And once a networking pro has his charts cooked up, how best to use them?
Marty outlined three major use case areas:
- Discover and explore: Internally, sifting through thousands of log files is inefficient and, worse, it's easy for critical elements to be overlooked. A good graph can help spot trends and correlate them to other parts of the network, helping to spot and diagnose problems in one fell swoop.
- Communicate: Whether it's with the storage group on how their backups are tanking your network or your CIO on why the wireless controller upgrade is critical this budget cycle, charts can help quickly explain trends and demands to people who don't have a networking pro's specialised knowledge. Marty said one IT group had regularly sent another group log data under the assumption that they were self-explanatory. But the second IT group had no clue what they were looking at until they saw a chart.
- Strategise: Logs are great for tackling problem spots, but graphs take networkers up a level and let them see the trends, giving them a chance to look at the longer-range picture and decide how to survive not only the next 12 hours but the next 12 months.
Marty did have one warning for those getting ready to dive into visualisation's benefits: garbage in, garbage out.
"One of the dangerous things is if you don't understand the log file itself, don't assume you'll understand the visualisation of it or even generate a visualisation that makes sense," he said. "If I have a firewall log file, and I have no idea about the IP addresses that are used or the role of internal machines, it gets very hard to analyze that."