Introduction
Whether you use Splunk for operations, security, or any other purpose--it can be helpful to be able to acknowledge events and add notes. Splunk provides a few different methods to accomplish this task: using an external database, writing to files, or the App Key Value Store (aka KV Store). The problem with using an external database is that it requires another system to provision and protect and can add unwanted complexity. Writing to files can be problematic in a distributed Splunk architecture that may use clustered or non-clustered components. The last option is the Splunk KV Store which appears to be the current recommendation from Splunk, but this can also appear complex at first--thus we will do our best to break it down in this article.In the most basic explanation, the KV Store allows users to write information to Splunk and recall it at a later time. Furthermore, KV Store lookups can be used to augment your event data by mapping event fields to fields assigned in your App Key Value Store collections. KV Store lookups can be invoked through REST endpoints or by using the following SPL search commands: lookup, inputlookup, and outputlookup. REST commands can require additional permissions, so this article will look at possibilities using the search commands.
References
Before we get started, we will list some references that helped in our understanding of the Splunk KV Store:
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ConfigureKVstorelookups
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Outputlookup
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Inputlookup
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Lookup
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ConfigureKVstorelookups
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Outputlookup
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Inputlookup
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Lookup
Deciding on the fields
For this example, we wanted to add a couple of fields to augment our event data. Namely an acknowledgement field (we will call this Ack) and a notes field (we will call this Notes). We will match the unique event id field with a field that is also called id.
So, in summary, we have id, Ack, and Notes. Splunk also uses an internal _key field, but we will not reference this directly in our efforts.
Getting started
Per our references above on configuring KV Store lookups, we will need two supporting configurations:- A collections.conf file specifying our collection name
- A stanza in transforms.conf to specify kvstore parameters
cat collections.conf
#
# Splunk app KV Store collection file
#
[acknotescoll]
head transforms.conf
[acknotes]
external_type = kvstore
collection = acknotescoll
fields_list = _key, id, Ack, Notes
Interacting with KV Store using search
The reference links provide helpful examples, but they do not provide everything necessary. Some of this was discovered through a bit of trial and error. Especially the flags and resulting behavior. We list below the major actions that can be taken and the search commands necessary to perform those actions:
Write new record:
| localop | stats count | eval id=101 | eval Ack="Y" | eval Notes="These are notes for event 101"| outputlookup acknotes append=True
Note: Without append=True, the entire KV Store is erased and only this record will be present
Update a record (only works if the record already exists):
| inputlookup acknotes where id="100" | eval Ack="N" | eval Notes="We can choose not to ack event 100" | outputlookup acknotes append=True
Note: Without append=True, the entire KV Store is erased and only this record will be present
Read all records:
| inputlookup acknotes
Read a record (A new search):
| inputlookup acknotes where id="$id$" | table _key, id, Ack, Notes
Read a record (combined with another search):
<search> | lookup acknotes where id="100" | table _key, id, Ack, Notes
Determine if record exists:
| inputlookup acknotes where id="108" | appendpipe [stats count | where count==0] | eval execute=if(isnull(id),"Record Does Not Exist","Record Exists!") | table execute
Conditional update:
Now that we can determine if a record exists and we know how to create a new record and update an existing record, we can combine all three to modify and/or create entries depending on their existence.
<query>| inputlookup acknotes where id="$id$" | appendpipe [stats count | where count==0] | eval execute=if(isnull(id),"| localop | stats count | eval id=$id$ | eval Ack=\"$Ack$\" | eval Notes=\"$Note$\" | outputlookup acknotes append=True","| inputlookup acknotes where id=\"$id$\" | eval Ack=\"$Ack$\" | eval Notes=\"$Note$\" | outputlookup acknotes append=True") | eval kvid=$id$ | eval kvack="$Ack$" | eval kvnote="$Note$" | eval Submit="Click me to Submit" | table kvid, kvack, kvnote, execute, Submit</query>
Write new record:
| localop | stats count | eval id=101 | eval Ack="Y" | eval Notes="These are notes for event 101"| outputlookup acknotes append=True
Note: Without append=True, the entire KV Store is erased and only this record will be present
Update a record (only works if the record already exists):
| inputlookup acknotes where id="100" | eval Ack="N" | eval Notes="We can choose not to ack event 100" | outputlookup acknotes append=True
Note: Without append=True, the entire KV Store is erased and only this record will be present
| inputlookup acknotes
Read a record (A new search):
| inputlookup acknotes where id="$id$" | table _key, id, Ack, Notes
<search> | lookup acknotes where id="100" | table _key, id, Ack, Notes
Limitation and work around
Unfortunately, it does not look like Splunk has a single search command/method to update a record, but create the record if it does not already exist. I may be mistaken about this and hope that I am missing some clever flag, so feel free to leave comments in the feedback section below. To get around this limitation, we first created a "simple" search command to check for the existence of a record.
Determine if record exists:
| inputlookup acknotes where id="108" | appendpipe [stats count | where count==0] | eval execute=if(isnull(id),"Record Does Not Exist","Record Exists!") | table execute
Example of a record that exists
Example of record that does not exist
Now that we can determine if a record exists and we know how to create a new record and update an existing record, we can combine all three to modify and/or create entries depending on their existence.
<query>| inputlookup acknotes where id="$id$" | appendpipe [stats count | where count==0] | eval execute=if(isnull(id),"| localop | stats count | eval id=$id$ | eval Ack=\"$Ack$\" | eval Notes=\"$Note$\" | outputlookup acknotes append=True","| inputlookup acknotes where id=\"$id$\" | eval Ack=\"$Ack$\" | eval Notes=\"$Note$\" | outputlookup acknotes append=True") | eval kvid=$id$ | eval kvack="$Ack$" | eval kvnote="$Note$" | eval Submit="Click me to Submit" | table kvid, kvack, kvnote, execute, Submit</query>
Results
These are just some examples of what is possible.
You could create an event acknowledgement page
You could create an event acknowledgement page
Event acknowledgement page
Once the fields are filled in at the top with the event id, acknowledgement, and notes, it could create the command to either update or add a new entry to the KV Store. Clicking the Submit hyperlink will actually run that command and modify the KV Store.
Event acknowledgement page filled out and waiting for click to submit
Once the data is populated in the KV Store, these records can be mapped to the original events to add this data for analysts.
Original event data with KV Store augmentation
Conclusion
Hopefully this helps expose some of the interesting possibilities of using Splunk's KV Store to create an event acknowledgement/ticketing system using search operations. Feel free to leave feedback below--especially if there is an easier search operation for updating a record and adding a new one if it does not already exist. Thanks for reading.
