Scraping IOCs from the Internet

Last Updated for Accuracy: September 2019

A Threat Intelligence program, I have found, has multiple stages. The first stage would be manually looking for IOCs and trying to hunt for random artifacts your find online to correlate some kind of attack. The last stage would be where your organization has a dedicated threat hunting team where they have identified their critical assets and properly threat modeled, has dedicated infrastructure for TAXI or STIX threat intelligence to be shared on a platform, such as MISP, to organize and share your intelligence, and so on and so forth.

This post will be what I was looking for when I started in the SOC, where and how to find usable IOCs, study the kind of traffic these IOCs generate on a SIEM and how they interact with a NGFW, and ultimately impress my boss with finding ticket-worthy traffic to escalate to my customers.

This looks like a job for a Tier 1 Analyst…

While working in a SOC, one of your tasks could be looking for IOCs and curating threat lists to proactively hunt malicious activity in your network.

One of the ways you can do this is to pick a malware family, Emotet for example, and scour the internet by putting together lists with various indicators of compromise, like hashes, IPs, and domains.

You would then feed the list into your SIEM or network analyzer tools to automatically detect IOCs in your log files.

Keep in mind, threat actors continually change their tactics and their command and control servers, so the IOCs from yesterday may not be fruitful today. Its the best practice to update your feeds daily and be persistent.

Open Source Threat Intelligence Feeds

Many security enthusiasts create their own websites and repositories to track and analyze malware. There are many open source feeds free to download that give a good starting point for any threat hunting operation. Here are some well known threat feeds;

These feeds are updated daily and have many interesting artifacts in them, usually, with a searchable database that tags the artifacts with malware type, status, and IP and domain lookup information.

Abuse CH Feodo, Ransomeware, SSL Blacklist, and URL threat lists

Twitter

I have found, one of the best tools to dive into threat hunting is Twitter. You can start with with searching for hashtags like; #threathunting #malwaremustdie #malware #malspam #phishing or malware specific tags like #emotet or #azorult and see a ton of interesting information.

I recommend using TweetDeck and set up pre-defined searches. In the below screenshot, I am searching for the latest posts from ‘app.any.run’, ‘virustotal.com’, and ‘urlscan.io/result’, basically, turning Twitter into a threat intelligence feed from various malware researchers and other threat hunters to share information.

Tweetdeck saved searches

On top of monitoring the search feeds, you can find many interesting people follow who provide amazing threat intelligence and information regarding malware. Check out the following profiles;

These users will usually posts pastebin dumps of IOC lists that were harvested from some malware sample they reverse engineered.

If you click on their pastebin links, you will get something like the list of IOCs below;

pb

You will have to export this data using a text editor to get the raw data in a separate file

Putting It All Together

Once you have all of your threat feeds, how do you organize them? A stumbling threat hunting program may spend tedious hours curating manual text lists, tagging them with information, and feeding them into a monitoring tool or database.

Alternatively, a mature threat hunting program will leverage a platform, such as MISP, to input their data and share it with other organizations, using TAXI or STIX standards.

MISP

MISP Project is an open source threat intelligence platform for gathering, sharing, storing and correlating Indicators of Compromise of targeted attacks, threat intelligence, financial fraud information, vulnerability information or even counter-terrorism information.

What makes MISP an excellent tool to incorporate into your security solution is how easy it is to share threat intelligence between different organizations or data feeds.

MISP sharing comes in two flavors, feeds we all know such as abuse.ch and the ability to connect to other MISP instances.

Final Words

Be persistent. Keep a track of what your searching for. When you find these IOCs, they will most likely be blocked by your NGFW (if you have one), but its still useful to look at their activity; what port and service were they trying to exploit, TELNET? SSH? Looking at this traffic will give you an idea of what to search for when crafting SIEM use cases.