Erik Hjelmvik
,?
Monday, 17 August 2015 08:45:00 (UTC/GMT)
I am a long time skeptic when it comes to blacklists and other forms of signature based detection mechanisms.
The information security industry has also declared the signature based anti-virus approach
dead several times during the past 10 years.
Yet, we still rely on anti-virus signatures, IDS rules, IP blacklists, malware domain lists, YARA rules etc.
to detect malware infections and other forms of intrusions in our networks.
This outdated approach puts a high administrative burden on IT and security operations today,
since we need to keep all our signature databases up to date,
both when it comes to end point AV signatures as well as IDS rules and other signature based detection methods
and threat feeds. Many organizations probably spend more time and money on updating all these blacklists
and signature databases than actually investigating the security alerts these detection systems generate.
What can I say; the world is truly upside down...
Image: Shower by Nevit Dilmen.
I would therefore like to use this blog post to briefly describe an effective blacklist-free approach
for detecting malware and intrusions just by analyzing network traffic.
My approach relies on a combination of whitelisting and common sense anomaly detection
(i.e. not the academic statistical anomaly detection algorithms that never seem to work in reality).
I also encourage CERT/CSIRT/SOC/SecOps units to practice Sun Tzu's old ”know yourself”,
or rather ”know your systems and networks” approach.
Know your enemy and know yourself and you can fight a hundred battles without disaster.
- Sun Tzu in The Art of War
Image: Art of War in Bamboo by vlasta2
My method doesn't rely on any dark magic, it is actually just a simple
Rinse-Repeat approach
built on the following steps:
- Look at network traffic
- Define what's normal (whitelist)
- Remove that
- GOTO 1.
After looping through these steps a few times you'll be left with some odd network traffic,
which will have a high ratio of maliciousness.
The key here is, of course, to know what traffic to classify as ”normal”.
This is where ”know your systems and networks” comes in.
What Traffic is Normal?
I recently realized that Mike Poor seems to be thinking
along the same lines, when I read his foreword to Chris Sanders' and Jason Smith's book Applied NSM:
The next time you are at your console, review some logs. You might think... "I don't know what to look for".
Start with what you know, understand, and don't care about.
Discard those. Everything else is of interest.
Following Mike's advice we might, for example, define“normal” traffic as:
- HTTP(S) traffic to popular web servers on the Internet on standard ports (TCP 80 and 443).
- SMB traffic between client networks and file servers.
- DNS queries from clients to your name server on UDP 53, where the servers successfully answers with an A, AAAA, CNAME, MX, NS or SOA record.
- ...any other traffic which is normal in your organization.
Whitelisting IP ranges belonging to Google, Facebook, Microsoft and Akamai as
”popular web servers” will reduce the dataset a great deal, but that's far from enough.
One approach we use is to perform DNS whitelisting
by classifying all servers with a domain name listed in Alexa's Top 1 Million list as ”popular”.
You might argue that such a method just replaces the old blacklist-updating-problem with
a new whitelist-updating-problem. Well yes, you are right to some extent,
but the good part is that the whitelist changes very little over time compared to a blacklist.
So you don't need to update very often. Another great benefit is that the whitelist/rinse-repeat approach
also enables detection of 0-day exploits and C2 traffic of unknown malware,
since we aren't looking for known badness – just odd traffic.
Threat Hunting with Rinse-Repeat
Mike Poor isn't the only well merited incident handler who seems to have adopted a strategy
similar to the Rinse-Repeat method;
Richard Bejtlich (former US Air Force CERT and
GE CIRT member)
reveal some valuable insight in his book
“The Practice of Network Security Monitoring”:
I often use Argus with Racluster to quickly search a large collection of session data via the command line, especially for unexpected entries. Rather than searching for specific data, I tell Argus what to omit, and then I review what’s left.
In his book Richard also mentions that he uses a similar methodology when going on “hunting trips”
(i.e. actively looking for intrusions without having received an IDS alert):
Sometimes I hunt for traffic by telling Wireshark what to ignore so that I can examine what’s left behind. I start with a simple filter, review the results, add another filter, review the results, and so on until I’m left with a small amount of traffic to analyze.
I personally find Rinse-Repeat Intrusion Detection ideal for threat hunting,
especially in situations where you are provided with a big PCAP dataset to answer the classic question
“Have we been hacked?”.
However, unfortunately the “blacklist mentality” is so conditioned among incident responders
that they often choose to crunch these datasets through blacklists and signature databases in order to
then review thousands of alerts, which are full of false positives.
In most situations such approaches are just a huge waste of time and computing power,
and I'm hoping to see a change in the incident responders' mindsets in the future.
I teach this “rinse-repeat” threat hunting method in our
Network Forensics Training.
In this class students get hands-on experience with a dataset of 3.5 GB / 40.000 flows,
which is then reduced to just a fraction through a few iterations in the rinse-repeat loop.
The remaining part of the PCAP dataset has a very high ratio of hacking attacks as well as
command-and-control traffic from RAT's, backdoors and botnets.
UPDATE 2025-08-08
We have now published a blog post detailing how to use dynamic protocol detection
to identify services running on non-standard ports. This is a good example on how to put the Rinse-Repeat methodology into practice.
Posted by Erik Hjelmvik on Monday, 17 August 2015 08:45:00 (UTC/GMT)
Tags:
#Rinse-Repeat?
#PCAP?
#NSM?
#PCAP?
#Intrusion Detection?
#IDS?
#network forensics?
4SICS is the the leading Industrial Control System (ICS) security conference in Europe,
which brings in speakers and attendees from all around the world.
I tought a one-day 
Image Credit: Patrick Nixdorf
Image: Miuref/Boaxxe Trojan C2 traffic to "1.web-counter[.]info" on TCP 443 doesn't use SSL (or HTTPS)
Our class is held the days before the 
