Quick and Flexible IOC Hunting in Splunk

By Tony Lee and Arjun Mathew

Imagine that you are battling a known threat actor.  You have gathered indicators of compromise (IOCs) from reversing malware as well as helpful contributions from the rest of the security community.  But how could those IOCs be tasked across your existing data quickly in order to track attacker movement in real-time?  Here is one possible solution:

1. Use a lookup file
2. Clever Splunk search
3. Even more clever dashboard

This article will outline the process and even share an example dashboard (shown in the screenshot below).

Figure 1:  Known IOC Dashboard provided at the end of the article

Lookup File

We used the following process to create a lookup file and definition.  Create a file in excel and save it as a CSV called known_iocs.csv (similar to the file below).

Figure 2:  CSV that we initially populated with our IOCs

Then within Splunk, navigate to the following to create the lookup and the definition:

Settings > Lookups > Lookup table files > Add new

• Destination app:  Select the app
• Upload a lookup file:  known_iocs.csv
• Destination filename:  known_iocs.csv

Settings > Lookups > Lookup definitions > Add new

• Destination app:  Select the app
• Name:  known_iocs.csv
• Type:  File-based
• Lookup file:  known_iocs.csv

Now, here is the problem.  How do you scale this solution to a group effort to update a lookup table with IOCs?  It does not work well to pass the CSV around and then constantly upload.  Enter another graceful solution from Luke Murphey -- The Lookup File Editor Splunk App (https://splunkbase.splunk.com/app/1724/).

Figure 3:  Lookup File Editor App from Luke Murphey

Once the Lookup File Editor Splunk App is installed, navigate to it, search for your known_ioc.csv file.  Open it and right click on the bottom line and "Insert a new row".  You can edit the lookup file right in Splunk.  Once it is saved, the correlation searches will automatically run with the new IOC data.

Figure 4:  Inserting a new line to our known_ioc.csv file

Clever Search

Now that we have a lookup table that has our IOCs in it and a convenient way to edit it, we just need a search that will apply the IOCs to our data.  The example below applies the IOCs to the cylance_protect index, but feel free to change the index name as needed.  Additionally, we show how to search just one column of the IOC data as well as multiple columns.


One type of IOC (Hash):

index=cylance_protect [|inputlookup known_iocs.csv | rename Hash as query | table query] | stats count


Two types of IOCs (Hash & FileName)

index=cylance_protect [|inputlookup known_iocs.csv | rename Hash as query | table query] OR [|inputlookup known_iocs.csv | rename FileName as query | table query] | stats count

Note the OR statement between the two inputlookups -- needed when querying multiple columns.

Figure 5:  What will be our top panels showing a count of the hits

Even More Clever Dashboard

Now that we have functional searches, we need a dashboard to monitor our different data feeds such as:

• Proxy
• Firewalls
• DNS
• Antivirus Hits
• Email Protection
• Windows Event Logs

You can see in the screenshot below that we use Single Value panels on the top row.  Each of these panels contains a dynamic drilldown to populate the panel below it with the contents of the Single Value panel when clicked.

Figure 6:  Dashboard displayed at the start of the article and in the Sample Dashboard section below

The drilldown for each Single Value panel sets a token which is essentially the search, but without the stats count (feel free to table the data as needed):

        <drilldown>
          <set token="alert">index=proxy $wild$ [|inputlookup known_iocs.csv | rename Domain as query | table query] | table _raw</set>
        </drilldown>



Then the bottom panel is just a search of the token set in the drilldown above.

        <search>
          <query>| search $alert$</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>

Conclusion

Using a clever combination of features that already exist within Splunk (for the most part), we were able to create a quick method to update an IOC list and apply it against existing data within Splunk. Simply monitor these dashboards and use it to track the attacker's activities in real-time.

Sample Dashboard

The sample dashboard below uses a number of indexes to search over different data feeds.  Just change these indexes to the ones you are interested in monitoring.


<form>
  <label>Known IOC Hits</label>
  <description>Threat Actor</description>
  <fieldset submitButton="true">
    <input type="time" searchWhenChanged="true" token="time">
      <label>Time Range</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="text" searchWhenChanged="true" token="wild">
      <label>Wildcard Search</label>
      <default>*</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <single>
        <title>Proxy</title>
        <search>
          <query>index=proxy $wild$ [|inputlookup known_iocs.csv | rename Domain as query | table query] | stats count</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="colorMode">none</option>
        <option name="drilldown">all</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="useColors">1</option>
        <drilldown>
          <set token="alert">index=proxy $wild$ [|inputlookup known_iocs.csv | rename Domain as query | table query] | table _raw</set>
        </drilldown>
      </single>
    </panel>
    <panel>
      <single>
        <title>Firewalls</title>
        <search>
          <query>index=firewalls $wild$ [|inputlookup known_iocs.csv | rename IP as query | table query] | stats count</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="drilldown">all</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <drilldown>
          <set token="alert">index=firewalls $wild$ [|inputlookup known_iocs.csv | rename IP as query | table query] | table _raw</set>
        </drilldown>
      </single>
    </panel>
    <panel>
      <single>
        <title>DNS</title>
        <search>
          <query>index=dns $wild$ [|inputlookup known_iocs.csv | rename Domain as query | table query] | stats count</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="drilldown">all</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="useColors">1</option>
        <drilldown>
          <set token="alert">index=dns $wild$ [|inputlookup known_iocs.csv | rename Domain as query | table query] | table _raw</set>
        </drilldown>
      </single>
    </panel>
    <panel>
      <single>
        <title>Antivirus Hits</title>
        <search>
          <query>index=av $wild$ [|inputlookup known_iocs.csv | rename Hash as query | table query] | stats count</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="drilldown">all</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="useColors">1</option>
        <drilldown>
          <set token="alert">index=av $wild$ [|inputlookup known_iocs.csv | rename Hash as query | table query] | table _raw</set>
        </drilldown>
      </single>
    </panel>
    <panel>
      <single>
        <title>Email Protection</title>
        <search>
          <query>index=mail_protection $wild$ [|inputlookup known_iocs.csv | rename Domain as query | table query] | stats count</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="drilldown">all</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="useColors">1</option>
        <drilldown>
          <set token="alert">index=mail_protection $wild$ [|inputlookup known_iocs.csv | rename Domain as query | table query] | table _raw</set>
        </drilldown>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Information Table (Click one of the numbers above to populate this table with Details)</title>
      <table>
        <search>
          <query>| search $alert$</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="drilldown">cell</option>
      </table>
    </panel>
  </row>
</form>

Event acknowlegement using Splunk KV Store

By Tony Lee

Introduction

Whether you use Splunk for operations, security, or any other purpose--it can be helpful to be able to acknowledge events and add notes.  Splunk provides a few different methods to accomplish this task:  using an external database, writing to files, or the App Key Value Store (aka KV Store).  The problem with using an external database is that it requires another system to provision and protect and can add unwanted complexity.  Writing to files can be problematic in a distributed Splunk architecture that may use clustered or non-clustered components.  The last option is the Splunk KV Store which appears to be the current recommendation from Splunk, but this can also appear complex at first--thus we will do our best to break it down in this article.

In the most basic explanation, the KV Store allows users to write information to Splunk and recall it at a later time.  Furthermore, KV Store lookups can be used to augment your event data by mapping event fields to fields assigned in your App Key Value Store collections. KV Store lookups can be invoked through REST endpoints or by using the following SPL search commands: lookup, inputlookup, and outputlookup.  REST commands can require additional permissions, so this article will look at possibilities using the search commands.

References

Before we get started, we will list some references that helped in our understanding of the Splunk KV Store:
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ConfigureKVstorelookups
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Outputlookup
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Inputlookup
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Lookup

Deciding on the fields

For this example, we wanted to add a couple of fields to augment our event data.  Namely an acknowledgement field (we will call this Ack) and a notes field (we will call this Notes).  We will match the unique event id field with a field that is also called id.

So, in summary, we have id, Ack, and Notes.  Splunk also uses an internal _key field, but we will not reference this directly in our efforts.

Getting started

Per our references above on configuring KV Store lookups, we will need two supporting configurations:

1. A collections.conf file specifying our collection name
2. A stanza in transforms.conf to specify kvstore parameters

cat collections.conf 

#

# Splunk app KV Store collection file

#

[acknotescoll]

head transforms.conf 

[acknotes]

external_type = kvstore

collection = acknotescoll

fields_list = _key, id, Ack, Notes

Interacting with KV Store using search

The reference links provide helpful examples, but they do not provide everything necessary.  Some of this was discovered through a bit of trial and error.  Especially the flags and resulting behavior.  We list below the major actions that can be taken and the search commands necessary to perform those actions: 

Write new record:
| localop | stats count | eval id=101 | eval Ack="Y" | eval Notes="These are notes for event 101"| outputlookup acknotes append=True

Note:  Without append=True, the entire KV Store is erased and only this record will be present


Update a record (only works if the record already exists):
| inputlookup acknotes where id="100" | eval Ack="N" | eval Notes="We can choose not to ack event 100" | outputlookup acknotes append=True

Note:  Without append=True, the entire KV Store is erased and only this record will be present

Read all records:
| inputlookup acknotes


Read a record (A new search):
| inputlookup acknotes where id="$id$" | table _key, id, Ack, Notes

Read a record (combined with another search):
<search> | lookup acknotes where id="100" | table _key, id, Ack, Notes

Limitation and work around

Unfortunately, it does not look like Splunk has a single search command/method to update a record, but create the record if it does not already exist.  I may be mistaken about this and hope that I am missing some clever flag, so feel free to leave comments in the feedback section below.  To get around this limitation, we first created a "simple" search command to check for the existence of a record.

Determine if record exists:
| inputlookup acknotes where id="108" | appendpipe [stats count | where count==0] | eval execute=if(isnull(id),"Record Does Not Exist","Record Exists!") | table execute

Example of a record that exists

Example of record that does not exist

Conditional update:
Now that we can determine if a record exists and we know how to create a new record and update an existing record, we can combine all three to modify and/or create entries depending on their existence.

<query>| inputlookup acknotes where id="$id$" | appendpipe [stats count | where count==0] | eval execute=if(isnull(id),"| localop | stats count | eval id=$id$ | eval Ack=\"$Ack$\" | eval Notes=\"$Note$\" | outputlookup acknotes append=True","| inputlookup acknotes where id=\"$id$\" | eval Ack=\"$Ack$\" | eval Notes=\"$Note$\" | outputlookup acknotes append=True") | eval kvid=$id$ | eval kvack="$Ack$" | eval kvnote="$Note$" | eval Submit="Click me to Submit" | table kvid, kvack, kvnote, execute, Submit</query>

Results

These are just some examples of what is possible.

You could create an event acknowledgement page

Event acknowledgement page

Once the fields are filled in at the top with the event id, acknowledgement, and notes, it could create the command to either update or add a new entry to the KV Store.  Clicking the Submit hyperlink will actually run that command and modify the KV Store.

Event acknowledgement page filled out and waiting for click to submit

Once the data is populated in the KV Store, these records can be mapped to the original events to add this data for analysts.

Original event data with KV Store augmentation

Conclusion

Hopefully this helps expose some of the interesting possibilities of using Splunk's KV Store to create an event acknowledgement/ticketing system using search operations.  Feel free to leave feedback below--especially if there is an easier search operation for updating a record and adding a new one if it does not already exist.  Thanks for reading.