Traffic Ray: New Splunk App to visualize your HTTP WEB traffic for security and analytics needs

Update: Splunk approved and published Traffic Ray within official Splunk app repository here.

Traffic Ray is a real time Web traffic analytics Splunk App I built for web server administrators and web hosting service providers.

Traffic Ray leverages raw Apache log files to visualize incoming Web server traffic allowing to discover incoming IP activity patterns, detect malicious activity, view bandwidth consumption trends and gain insights into Web visitor’s origins and behaviors on a single dashboard. Ok, on two :).

Being a webmaster as well as Web hosting server administrator myself – I often wanted to get unobstructed, quick, visual, real time view into incoming Web traffic stats and patterns. While working with many different reporting and analytic solutions I found most of them to be either too convoluted, overly generic, suspiciously intrusive or unacceptably restrictive. I needed an easy ability to gain comprehensive server-wide Web traffic insights as well as ways to do quick drilldowns into specific IP address behavior patterns or specific Web site bandwidth consumption trends.
Also, quite often ill-behaving or outright malicious incoming Web traffic source causes server to send automated, generic, non-descriptive system alerts about excessive server loads, suspicious running processes and alike that would require further root cause analysis.

Before I had to rely on multiple tools to put together a big picture of events as well as login to system shell and manually search and grep through raw logs to discover culprits of suspicious activity – and that was time consuming and unpleasant process.

Now with Traffic Ray it is essentially one click step to grasp all the necessary and specific information about root causes and origins of suspicious activity on a single screen view.

I always wanted to find a simple way to leverage mountain of data that is generated and kept within Apache web server logs and with the help of Splunk+Simple XML toolkit it became possible to build working and aesthetically pleasing solution in a relatively short period of time.

Extra bits of excitement

• Within the first few hours of deploying test version of Traffic Ray analytics to the real Web Hosting server I was able to discover nuggets of data that directly helped three of my clients to improve their businesses. Two of these clients were able to boost their marketing efforts by fixing static image delivery issues and third client discovered that the root cause of slow website performance was a single ill-designed, well hidden stylesheet.
• I particularly excited about security aspect of Traffic Ray. With only 1-2 clicks I can now see all ongoing malicious activity, suspicious incoming traffic patterns, coordinated cyberattacks along with the offending IP addresses and their origins on a single visual dashboard.

Target Audience

With petabytes of raw Apache Web server traffic logs data generated worldwide daily – millions of webmasters, web server administrators and web hosting service providers can utilize Traffic Ray to tap into the treasure trove of raw logs data to gain quick insights into incoming Web traffic events to better secure and optimize their web portals, services and offerings.

With Splunk’s ability to scale large amounts of data it becomes much easier for hosting service providers to deploy robust Web traffic analytic and visualization solution leveraging hundreds of gigabytes of raw web server traffic log data files to deliver better and more secure services.

Operational Value and Benefits

• Security With Traffic Ray’s ability to visualize different security threats in real time and historically – from occasional malicious activity to coordinated cyberattacks spawning multiple geographic regions – hosting services and data centers will have powerful tool to collect enough actionable information to remedy attacks and make their service offerings more secure and robust.
• Investigations Traffic Ray allows for additional content filtering via number of input controls (by IP address, content type, traffic type, website name) as well as by entering simple fragments of raw Splunk queries directly into the dashboard inputs. This opens new potentials for sophisticated intelligence work in the fields of security, traffic analysis and even for marketing and promotional work.
• Reducing total cost of ownership and boosting performance of Web Hosting services Bandwidth consumption stats and analytics provided by Traffic Ray – from overall summary, to specific hosted Website and to specific incoming IP address provides enough information to discover and remedy causes of excessive data bandwidth usage. Traffic Ray helps to detect and remove bottlenecks from web hosting services, improve overall quality of offering, boosting customer satisfaction and reducing costs.
• Marketing Given into the hands of webmasters – Traffic Ray pinpoints sources of Website’s content delivery errors, helps to discover missing images and URLs, misbehaving pages and provide information about the most active sources and volumes of organic and paid traffic for each website. Traffic Ray helping webmasters to have a clear insight into the actual traffic data instead of relying on third party providers and thus better spending their budget marketing dollars.

Portability of Traffic Ray

Traffic Ray currently supports any Apache-based platform – all flavors of Linux as well as Windows servers. Apache server is currently used to power vast majority of worldwide websites (source: Netcraft). Immediate future plans include addition of support for Microsoft IIS servers.

Future plans and sustainability of Traffic Ray

• Support for Microsoft IIS server logs data.
• Support for security applications to allow Traffic Ray alerts to be used as a trigger source to ban malicious traffic sources from accessing server resources in real time automatically.
• Support for Webmasters marketing needs, such as delivering easier ways to differentiate between paid and organic traffic, showing better summaries for referrer data, integration with third party e-commerce systems to allow webmaster to run analytics for paid clients as well as to track marketing campaigns.
• Deploying dedicated live demo, support and blogging portal for Traffic Ray.

Contact me to get a copy of Traffic Ray Splunk App. Soon is will be downloadable directly from Splunk.

Get Traffic Ray App from official Splunkbase here.