IBM TeaLeaf + Splunk = Powerful fraud investigation and security analytics platform for financial firms. Part 3: Visualizing Trends and Patterns of Cyberattacks

Once you got all the beautiful and rich traffic data exported from Tealeaf and imported in Splunk – the possibilities are virtually endless to create very powerful search and cross referencing analytics and security investigation tool.

Within my consulting work for a major financial services firm I’ve built a specialized Splunk App that allows using single dashboard to execute multiple searches and visualize results and trends by leveraging Tealeaf data.

In addition to that a number of custom searches and alerts were created to create summary indexes and automatically detect and alert on possible malware infections, notify about suspicious activity patterns and out of bound activities.

Before sharing more details about visualizing trends and malicious traffic patterns – here are few tips on general design of custom Splunk security analytic apps and dashboards with Splunk.

While I cannot offer image snapshots of the actual dashboard’s visuals used within financial firm due to obvious client’s security concerns, I can describe the general approach we took to overcome Tealeaf’s limitations with Splunk as well as number of important points on how to take the most advantage of Splunk as a security research tool.

1. Minimize number of dashboards created and put an effort to place search results into the single screen. Security teams are overloaded with multitude of tools one crappier than other and need to switch between different tools and screens is frustrating.
   Ability to do and see a lot on a single screen will be appreciated.
2. Make it easy to search for IP addresses as well as for a group of IP addresses in a single search.
   Here are few tips:
   • Splunk allows for easy and direct searching and matching for a CIDR subnets.
     Instead of (older): ... | where cidrmatch("87.238.80.0/21", clientip) ..., consider using simpler: ... | where clientip=12.3.45.0/24
   • Use ‘TERM’ to search for IP within _raw event data.
     Quite often searches like: ... clientip=12.3.45.67 ... or ..."12.3.45.67"... are much slower than: ... TERM(12.3.45.67)
     Note: TERM() doesn’t work with CIDR subnets.
3. Use javascript snippets for dashboard developed in Simple XML to enhance and customize functionality.
   For example with the help of a small javascript snippet I’ve implemented a special input field that allows pasting of arbitrary text. Then with the help of small javascript function – it automatically finds all IP addresses and IP subnets within text, converts these to SPL query and runs search fully automatically.
   This “magic” search field saved lots of time for security investigators who often receive arbitrary reports with a number of suspicious IP addresses and want to quickly correlate these with recent traffic data.
   This field avoids need of multiple repeated copy/pastings and need of running multiple searches.
4. Do not hardcode your app or dashboard’s abilities to limited set of input fields.
   Consider adding “raw search” input field to Splunk security/search dashboard. This will make dashboard very powerful and opened to run more complex queries down the road without putting an extra development work.
   It’s hard to predict what kind of combo search needs to be run – and having an ability to type:
   ... username=johnsmith1 AND (ip=12.34.5.67 OR ip=45.6.78.0/24) ... directly within the dashboard’s input directly will come handy every day.
5. Take advantage of drilldowns – make as many fields or rows clickable as possible.
   It makes searches and investigation work much easier when clicking on IP address or username field opens new dashboard tab with more specific search results data.
6. Come up with extra fields allowing for drilldowns to outside/external services. For example it’s very easy to add a simple field named “Whois” next to the IP address that links directly to “whois” service allowing for quick check of ownership of IP address.
   Another field that came handy is Tealeaf session ID. Because we use Tealeaf as one of our data sources – clicking on Tealeaf ID opens new tab with Tealeaf interface allowing for visual replay of sessions.
7. Have easy access to summary of Browser user agents and referers. Quite often malicious activity and automated scripting attacks can be detected by the content of these fields – so having easy access to these will help.

One of the important tasks to protect client against cyber attacks is an ability to see the bigger picture behind traffic activity. The questions are:

• is there a relationship between different traffic sources?
• is there a common actor or actors behind the seemingly isolated hacks trying to accomplish the same task of compromising client’s service?
• Is the activity driven by the humans or by scripts and bots?
• Is there detectable timing pattern behind activity that might shed the light on attack specifics to provide timely protection?

This and many other questions were answered by the addition to Advanced Security Analytics Dashboard that I built for the client.

I used Simple XML for the dashboard that utilized sparklines within the table that grouped search results by IP addresses.

But instead of traditional sparklines use as an actual line I’ve configured it to be shown as colors bars by customizing it’s configuration via Simple XML settings. The advantage of that approach is that it makes it easier to correlate activity across different IP addresses as well as incorporate color coding for each bar depending on intensity of traffic.

Actual search query to generate sparklines is simple and looks like this (fragment):

... | stats count, sparkline(count,6h) as "Activity Trends and Patterns per IP address" by IP, Country | sort - count | table ...
With some custom CSS and minor configuration effort – here are the results we came up with while investigating bash shellshock and other vulnerability attacks detected on client’s banking portal:

Simple XML settings allows for color coding of sparkline bars depending on the value. In essence I coded low volume request with green, average volume with yellow and higher volume with orange, red.

In this specific case the raw search was run over the period of 30 days to find all hits with signatures of bash shellshock attack pattern.

What’s interesting here is that single screen clearly shows timing correlation between different malicious traffic sources. Sparklines represented with bars are an excellent tool to expose correlation between multiple traffic sources, volumes, activities over time as well as visually show typical patterns of automated tools and script an attackers would use.

The image shows that completely different IP addresses, originating from multiple geographical regions: Italy, USA, Germany, South Korea and Guernsey – are all engaging in some sort of automated scripting attacks to probe enterprise banking portal for shellshock vulnerabilities.

Single screen also shows the variations of referers and user agent strings in attempt to exploit possible weaknesses within the portal.

Upon further studies of traffic forensics we were able to discover specific scripts the attackers used as well as certain patterns they use to change IP addresses and choose dedicated server providers in attempt to hide their origins and to launch subsequent attacks.

Splunk Sparklines shows that with a bit of creativity an enterprise can gain lots of visibility into underlying traffic patterns that gives security analysts an ability of quick visual-based detection of malicious activity across multiple data metrics.