12 / 07 / 2015
Adding alert on file system modification events
Let’s setup alert that will send email to administrator when some executable script will be modified on Web server under user’s file system space. We will run scheduled search every 5 minutes to scan last 5 minutes worth of modifications.
- On a Splunk indexer, login into Splunk.
- Go to: Settings -> Searches, Reports and Alerts. Press [New].
- Destination app: “Search & Reporting (search)”.
- Search Name: “Scripts Added/Modified”.
- Search, paste this:
index=fs _index_earliest=-5m@m | eval indextime=strftime(_indextime, "%Y-%m-%d %H:%M:%S") | dedup source | table _time, indextime, source
- Schedule type: “Cron”
- Cron Schedule: */5 * * * *
- Run as (*) Owner
- Alert, Condition: “if number of events”, “is greater than”, 0
- Alert mode: “Once per search”
- Expiration: “After 24 hours”
- Send email: Enable
- To: – specify your email address. Also – make sure to configure email settings here (set Splunk server name or IP to yours):
- List in triggered alerts: [x] Enable
12 / 07 / 2015
In this article I’ll demonstrate step by step how to setup Splunk analytics to detect successful known and unknown malware attacks on web hosting systems in real time.
In addition the same solution will include instructions to deploy fully automated investigative analytics to discover the origins of attackers (IP addresses) as well as any modifications within the file system.
This information is essential to discover and immediately eliminate all possible backdoors and exploits that attacker tried to plant.
Real time alerts will be delivered via email to system administrator as soon as attack occurs. The same information will be available via Splunk web interface for further analysis.
The information presented will help system administrators as well as hosting service providers to devise measures to detect close to 100% of possibly successful cyber attacks and take immediate actions before malware tries to propagate further and cause significant damage or loss to business.
Deployment of such system not only can help to prevent significant monetary and business losses for the enterprise but also can assist in avoiding an embarrassment and negative publicity where customers and news outlets learns about successful hack before the actual business does.
While the steps described are somewhat specific to CentOS Linux hosting based on WHM/Cpanels – the same approach can be adjusted to any type of hosting and operating environments due to Splunk’s multi-platform support capabilities.
Wouldn’t it be nice if your SIEM would send a “possible insider threat!” alert when it detects that employee is in fearful, paranoid or even panicky emotional state while trying to access secure, confidential corporate documents repository?
Or receive real time “possible account takeover!” alert when it detects that currently logged in user is in deep anxiety or experiencing severely agitated emotional conditions while trying to initiate money transfer to an outside bank account?
This approach is used very successfully to detect potential threats by the world’s most secure airlines.
Trained security officers are able to see if passenger feels nervous or agitated or otherwise exhibits emotionally unusual behaviors and then follow up with further checks and investigation. On one occasion by interrogating the nervous passenger the actual bomb was found inside his luggage while the passenger mistakenly thought he had been hired to smuggle diamonds.
The Step Up from User Behavior Analytics
With some creativity, knowledge of human psychology and analytics approach we can apply the same methods to today’s machine data generated by users, clients and employees of financial institutions, banks, governments facilities and corporations to prevent known and unknown attacks from outside as well as from inside of enterprise.
A while ago analyzing an account takeover cyber attack I’ve isolated a complete data set belonging to the attacker who’ve accessed another user account with legitimate credentials.
Attacker’s session activity was almost identical to legitimate user’s activity across all metrics:
Pages accessed, session duration, session hit length, browser user agent used, geographical region of originated IP address, order in which pages were accessed, approximately the same time of the date as legitimate user would use, etc…
10 / 05 / 2015
One of my enterprise clients observed that certain class of attacks having a number of distinctive characteristics: attacker who possessed correct user account credentials won’t try to engage into malicious behavior right away.
Initial activity would involve certain degree of reconnaissance and gathering of future victim’s specific data, such as account balances, trade histories and other. So normal “red flags” and risk scoring metrics won’t generate any alerts.
However in many cases such pre-fraudulent activity was still carrying an unusual behavior marks: either session velocity (average number of seconds per hit) or session density (number of hits within the session) or both exceeded normal, baseline session patterns typical for the average client application user’s behavior.
Abnormally high session velocity is also a typical pattern of an automated script-driven session that both fraudsters and competition were using to siphon data from the client’s servers.
One of the possible solutions to detect these activities would be to calculate average session velocity and density and then apply these values to trigger alerts when session metrics exceeded thresholds.
The issue here is that due to the client’s business specific these averages vary greatly depending on the time of the day, time of the week and also the month of the year.
10 / 05 / 2015
Back in my days at IBM T.J. Watson Research Center where we were working on techniques to detect known and unknown malware, the fast growing challenge was the rising threat of malware’s abilities to become polymorphic.
Malicious snippets of code encrypted themselves and made it very difficult to apply conventional signature based detection techniques.
We’ve developed a tiny virtual machine in C language that was able to load malware code in real time and analyze it’s behavior without need to figure out how to decrypt it. Certain score metrics were assigned to keypoints and function calls and logic was put in place to trigger an alert if “risk” score exceeded certain heuristic threshold.
That technique allowed us to deliver top quality enterprise security solution (purchased by Symantec later on) that was capable of detecting previously unknown threats. That was more than 15 years ago.
While working with financial clients and technology companies today I can see that old behavior pattern analysis stays as strong as ever helping enterprises to discover new types of suspicious behaviors and investigate malicious activities with previously unknown patterns from previously unknown sources.
29 / 03 / 2015
To remind – this is the challenge – what we want to accomplish:
Detect and alert when C-class IP subnet tries to access at least 5 different accounts within an hour and at least 75% of total accounts touched has never been accessed from this subnet *and* from this USER_AGENT within the last 45 days.
And, as you may remember from Part 1, here’s the basic logic that we need to implement to make it happen:
- Scan last hour of access log data and find the list of subnets that tried to access multiple accounts within that hour.
- For each of these accounts – take username, IP, USER_AGENT and scan the previous 45 days of traffic history to find usernames that has never been touched by this IP/USER_AGENT combo.
- Alert if number of found accounts is above threshold.
I’ve spent quite a bit of effort to come up with a single query that does all of the above and in a pretty efficient manner.
The biggest part of challenge is that the query needs to find events (#1 above) but then it needs to run very custom search for each event against summary index that we’ve created (#2 above). And added icing on this cake is that the query needs to return results only if there are *no matches* found for the second part of search.
This quickly gets mind-boggling and it is a rather interesting puzzle to solve with SPL.
The way I solved it – is with a combination of macros + advanced subsearch. But instead of returning traditional results – the subsearch will return new, custom crafted Splunk search query to be executed by the outer search.
I named this approach Advanced Negative Look Behind (ANLB) query.
29 / 03 / 2015
Summary indexing is a great way to speedup Splunk searches by pre-creating a subset of only necessary data for specific purpose. In our case we need to filter out of all available WEB traffic data only login events. This will allow us to have very fast, much smaller data subset with all the information we need to reference against when matching with new, suspicious login events.
To proceed with building summary index we need to make a set of assumptions. These assumptions are needed to build the query and all other elements of the solution. You’ll be able to substitute names to your specifics later on if wanted to.
- Lets assume you have your WEB logs with all the event data indexed in Splunk already.
All web events are located within index named: logs.
- Field names (or aliases):
- HTTP request method (GET, POST, HEAD, etc..): method
- URL of page accessed: page
- Username field: username
- IP address of visitor: ip
- USER_AGENT value: ua
To generate summary index of login data – we need to create index itself first.
29 / 03 / 2015
- Detecting Bank Accounts Takeover Fraud Cyberattacks with Splunk. Part 2: Building Reference Summary Index of Logins Data.
- Detecting Bank Accounts Takeover Fraud Cyberattacks with Splunk. Part3: The Advanced Negative Look Behind Query.
In these series of posts I’ll cover the complete strategy of utilizing Splunk Enterprise in detecting customer account takeover fraud as well as setting up an automated alerts when such activity is detected.
While I’ve helped to implement these measures for large financial firm – the same approach can be applied to any online enterprise where it is essential to protect online customer accounts, quickly detect suspicious activity and to act and prevent monetary and business losses.
The techniques I am going to describe generate pretty low level of false positives and contain efficient ways to adjust trigger thresholds within multiple metrics for specific business needs. In addition – it is tested and works really well for portals that generate up to 3,000,000-5,000,000 hits a day.
Specific use case that is covered in these posts applies to situation where credentials of multiple clients (sometimes thousands or more) got in the hands of attackers who will try to take advantage of these for monetary, competitive or political gains. With the help of Splunk, enterprise will be able to quickly and automatically detect such situation and take necessary measures to protect business and clients.
Account takeover fraud comes into play when fraudster gains access to customer account credentials via any means: phishing campaigns, malware, spyware or by buying sets of stolen customer credential data on darknets or black online markets and forums.
I won’t get into the details of multiple possible ways customer credentials may be compromised but the end result is an ability of unauthorized person(s) to access multiple customer accounts and cause significant damages to customers and to business, including large monetary losses.
The worst way the enterprise can learn about cyberattack on their own customers is from CNN.
24 / 02 / 2015
Once you got all the beautiful and rich traffic data exported from Tealeaf and imported in Splunk – the possibilities are virtually endless to create very powerful search and cross referencing analytics and security investigation tool.
Within my consulting work for a major financial services firm I’ve built a specialized Splunk App that allows using single dashboard to execute multiple searches and visualize results and trends by leveraging Tealeaf data.
In addition to that a number of custom searches and alerts were created to create summary indexes and automatically detect and alert on possible malware infections, notify about suspicious activity patterns and out of bound activities.
Before sharing more details about visualizing trends and malicious traffic patterns – here are few tips on general design of custom Splunk security analytic apps and dashboards with Splunk.
While I cannot offer image snapshots of the actual dashboard’s visuals used within financial firm due to obvious client’s security concerns, I can describe the general approach we took to overcome Tealeaf’s limitations with Splunk as well as number of important points on how to take the most advantage of Splunk as a security research tool.
18 / 01 / 2015
Traffic Ray is a real time Web traffic analytics Splunk App I built for web server administrators and web hosting service providers.
Traffic Ray leverages raw Apache log files to visualize incoming Web server traffic allowing to discover incoming IP activity patterns, detect malicious activity, view bandwidth consumption trends and gain insights into Web visitor’s origins and behaviors on a single dashboard. Ok, on two :).
Being a webmaster as well as Web hosting server administrator myself – I often wanted to get unobstructed, quick, visual, real time view into incoming Web traffic stats and patterns. While working with many different reporting and analytic solutions I found most of them to be either too convoluted, overly generic, suspiciously intrusive or unacceptably restrictive. I needed an easy ability to gain comprehensive server-wide Web traffic insights as well as ways to do quick drilldowns into specific IP address behavior patterns or specific Web site bandwidth consumption trends.
Also, quite often ill-behaving or outright malicious incoming Web traffic source causes server to send automated, generic, non-descriptive system alerts about excessive server loads, suspicious running processes and alike that would require further root cause analysis.
Before I had to rely on multiple tools to put together a big picture of events as well as login to system shell and manually search and grep through raw logs to discover culprits of suspicious activity – and that was time consuming and unpleasant process.