Parsing and Displaying Windows Firewall data in Splunk

By Tony Lee

Have you ever wondered what it would be like to harness all of the Windows firewall data in your environment and crunch that activity using big data analytics?  What you find might shock you.

Fortunately for us Splunk heads, Andreas Roth has already completed most of the work to send the logs to Splunk and even parse them.  https://splunkbase.splunk.com/app/3300/#/details

Figure 1:  TA to collect and parse the logs already exists!

Big shout out to Andreas for the jump start.  However, we added some parsing to get the layer 4 transport protocol and then created a dashboard (shown below) that we are going to share here.

Figure 2:  Dashboard provided in this article

Prerequisites

There are some things you will need to do before we can make use of the Windows Firewall logs:
1)  Enable Windows Firewall Logging 
   Tip:  Use the link in the Log Location section below to enable and configure the firewall via GPO

2)  Forward the logs (written to disk) to Splunk via a Splunk UF, beats agent, etc.
   Tip:  This is made "easy" by installing the TA mentioned above on your forwarders

3)  Parse the logs
   Tip:  This is made easy by installing the TA mentioned above on your indexers

4)  Display the logs
   Tip:  This is made easy by using our dashboard code found at the end of this article

Log Location

The first thing we need to do is discover where those logs are located. After a bit of research, you will find that by default they should be located here:

%systemroot%\Windows\system32\LogFiles\Firewall\pfirewall.log

However, when I tried searching my machine for any sign of the logs, I discovered that they were turned off.  In fact, Windows does not log these to disk by default.

Figure 1:  Windows Firewall Logs off by default

To enable Windows Firewall logging, see the following article:  https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-firewall/configure-the-windows-firewall-log#to-configure-the-windows-firewall-log

Raw Log

What do these logs look like once they are written to disk? Well, they are short sweet and to the point.  See the example log below:

2018-07-03 14:19:55 DROP UDP 192.168.2.1 224.0.0.252 50859 5355 56 - - - - - - - RECEIVE

There are a lot of fields there, but Microsoft kindly places a header in the log file to indicate the field names:

date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsym tcpack tcpwin icmptype icmpcode info path

As mentioned before, if you installed the TA-winfw on your search head and universal forwarders most of the parsing will be performed for you.  Although, there was one field which did not seem to be parsed for us.  No biggie, we don't have to do the real heavy lifting.

Parsing

Using our example event from earlier, the transport field did not seem to be parsed for us which typically indicates if the packet was UDP or TCP, etc..  Instead the protocol field just displayed "ip".

2018-07-03 14:19:55 DROP UDP 192.168.2.1 224.0.0.252 50859 5355 56 - - - - - - - RECEIVE


To correct this, we added the following regex generated from the Splunk field extractor for our sourcetype of winfw:

^(?:[^ \n]* ){3}(?P<transport>\w+)

Search String

After parsing out the "transport" field, we can now form our search string:

index=winfw | table _time, dvc, direction, action, transport, src_ip, src_port, dest_ip, dest_port

Taking this a step farther, we created a dashboard which is provided at the bottom of the article.

Conclusion

Using the dashboard code below I bet you can find some interesting events in your network. Even if you don't find something malicious, you can probably find a misconfiguration or two. Correcting these issues will not only save on host performance and network performance, but now Splunk performance too.  Happy Splunking!

Great  resource:
https://www.howtogeek.com/220204/how-to-track-firewall-activity-with-the-windows-firewall-log/

Dashboard Code

The following dashboard assumes that the appropriate logs are being collected and sent to Splunk. Additionally, the dashboard code assumes an index of winfw and a sourcetype of winfw. Feel free to adjust as necessary. Splunk dashboard code provided below:


<form>
  <label>Windows Firewall</label>
  <fieldset submitButton="true" autoRun="true">
    <input type="time" searchWhenChanged="false" token="time">
      <label>Time Range</label>
      <default>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="text" searchWhenChanged="false" token="wild">
      <label>Wildcard Search</label>
      <default>*</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <title>Event Count</title>
        <search>
          <query>| tstats count where index=winfw by host</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="drilldown">cell</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
    <panel>
      <chart>
        <title>Top Action</title>
        <search>
          <query>index=winfw $wild$ | table _time, action | top action</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="charting.chart">pie</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
    <panel>
      <table>
        <title>Top Source IP</title>
        <search>
          <query>index=winfw $wild$ | table _time, src_ip | top limit=0 src_ip</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="count">10</option>
        <option name="drilldown">cell</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
    <panel>
      <table>
        <title>Top Dest IP</title>
        <search>
          <query>index=winfw  $wild$ | table _time, dest_ip | top limit=0 dest_ip</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="count">10</option>
        <option name="drilldown">cell</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
    <panel>
      <table>
        <title>Top Dest Port</title>
        <search>
          <query>index=winfw $wild$ | table _time, dest_port | top limit=0 dest_port</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="count">10</option>
        <option name="drilldown">cell</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>Details</title>
        <search>
          <query>index=winfw $wild$ | table _time, dvc, direction, action, transport, src_ip, src_port, dest_ip, dest_port</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">20</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
</form>

Splunk and ELK – Impartial Comparison Part I - Similarities

By Tony Lee

This series is not intended to start a “Big Data” holy war, but instead hopefully offer some unbiased insight for those looking to implement Splunk, ELK or even both platforms.  After all both platforms are highly regarded in their abilities to collect, parse, analyze, and display log data.  In fact, the first article in this series will show how the two competing technologies are similar in the following areas:

• Purpose
• Architecture
• Cost

Caveat

Most articles on this subject seem to have some sort of agenda to push folks in one direction or another—so we will do our absolute best to keep it unbiased. We admit that we know Splunk better than we know the ELK stack, so we are banking on ELK (and even Splunk) colleagues and readers to help keep us honest. Lastly, our hope is to update this article as we learn or receive more information and the two products continue to mature.

Similar Purpose

Both Splunk and ELK stack are designed to be highly efficient in log collection and search while allowing users to create visualizations and dashboards.  The similar goal and purpose of the two platforms naturally means that many of the concepts are also similar.  One minor annoyance is that the concepts are referred to by different names.  Thus, the table below should help those that are familiar with one platform map ideas and concepts to the other.

Splunk	ELK Stack
Search Head	Kibana
Indexer	Elastic Search
Forwarder	Logstash
Universal Forwarder	Beats (Filebeat, Metricbeat, Packetbeat, Winlogbeat, Auditbeat, Heartbeat, etc.)
Search Processing Language (SPL)	Lucene query syntax
Panel	Panel
Index	Index

Similar Architecture

In many ways, even the architecture between Splunk and ELK are very similar.  The diagram below highlights the key components along with the names of each component in both platforms.

Figure 1:  Architectural similarities

Cost

This is also an area where there are more similarities than most would imagine due to a misconception that ELK (with comparable features to Splunk) is free.  While the core components may be free, the extensions that make ELK an enterprise-scalable log collection platform are not free—and this is by design.  According to Shay Banon, Founder, CEO and Director of Elasticsearch:

“We are a business. And part of being a business is the belief that those businesses who can pay us, should. And those who cannot, should not be paying us. In return, our responsibility is to ensure that we continue to add features valuable to all our users and ensure a commercial relationship with us is beneficial to our customers. This is the balance required to be a healthy company.”

Elastic does this by identifying “high-value features and to offer them as commercial extensions to the core software. This model, sometimes called ‘open core’, is what culminated in our creation of X-Pack. To build and integrate features and capabilities that we maintain the Intellectual Property (IP) of and offer either on a subscription or a free basis. Maintaining this control of our IP has been what has allowed us to invest the vast majority of our engineering time and resources in continuing to improve our core, open source offerings.”

Source:  https://www.elastic.co/blog/doubling-down-on-open

That said, which enterprise-critical features aren’t included in the open source or even basic free license?  The subscription comparison screenshot found below shows that one extension not included for free is Security (formerly Shields).  This includes Encrypted communications, Role-based Access Control (RBAC), and even authentication.  Most would argue that an enterprise needs a login page and the ability to control who can edit vs. view searches, visualizations, and dashboards, thus it is not a fair comparison to say that Splunk costs money while ELK is free.  There are alternatives to X-PACK, but we will leave that to another article since it is not officially developed and maintained as part of the ELK stack.

Figure 2:  Encryption, RBAC, and even authentication is not free

In terms of host much Splunk costs vs. ELK, there are also many arguments there--some of which include the cost of build time, maintenance, etc.  It mostly depends on your skills to negotiate with each vendor.

Conclusion

Splunk and ELK stack are similar in many ways.  In fact, knowing one platform can help a security practitioner learn the other because many of the concepts are close enough to transfer.  The reduction in the learning curve is a huge advantage for those that need to convert from one platform to the other.  That said, there are differences, however we will discuss those in the next article.  In the meantime, we hope that this article was useful for you and we are open to feedback and corrections, so feel free to leave your comments below.  Please note that any inappropriate comments will not be posted—thanks in advance.  😊

Efficient Blue Coat (and other) Splunk Log Parsing

By Tony Lee

Special Notes

1)  This blog post does not only pertain to Blue Coat logs, but possibly other data sources as well.
2)  This is not a knock on Blue Coat, the app, TA, or any of that, it is just one example of many where we might want to change the way we send data to Splunk.  Fortunately Blue Coat provides the means to do so.  (hat tip)

Background info

A little while back, we were working on a custom Splunk app that included ingesting Blue Coat logs into a SOC's single pane of glass, but we were getting an error message of:

Field extractor name=custom_client_events is unusually slow (max single event time=1146ms)

The Splunk architecture was more than sufficient.  The Blue Coat TA worked great on small instances, but we found that it did not scale to a Blue Coat deployment of this magnitude.  The main reason for this error was the parsing in transforms.conf looked like this:

[custom_client_events]
REGEX = (?<date>[^\s]+)\s+(?<time>[^\s]+)\s+(?<duration>[^\s]+)\s+(?<src_ip>[^\s]+)\s+(?<user>[^\s]+)\s+(?<cs_auth_group>[^\s]+)\s+(?<x_exception_id>[^\s]+)\s+(?<filter_result>[^\s]+)\s+\"(?<category>[^\"]+)\"\s+(?<http_referrer>[^\s]+)\s+(?<status>[^\s]+)\s+(?<action>[^\s]+)\s+(?<http_method>[^\s]+)\s+(?<http_content_type>[^\s]+)\s+(?<cs_uri_scheme>[^\s]+)\s+(?<dest>[^\s]+)\s+(?<uri_port>[^\s]+)\s+(?<uri_path>[^\s]+)\s+(?<uri_query>[^\s]+)\s+(?<uri_extension>[^\s]+)\s+\"(?<http_user_agent>[^\"]+)\"\s+(?<dest_ip>[^\s]+)\s+(?<bytes_in>[^\s]+)\s+(?<bytes_out>[^\s]+)\s+\"*(?<x_virus_id>[^\"]+)\"*\s+\"*(?<x_bluecoat_application_name>[^\"]+)\"*\s+\"*(?<x_bluecoat_application_operation>[^\"]+)

The robustness and volume of data was simply too much for this type of extraction.

Solution

The solution is not to make Splunk adapt, but instead change the way data is sent to it. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format. Then the Blue Coat app and TA try to parse this space separated data using the complex regex seen above. Instead of doing that, fortunately you can instruct Blue Coat to send the data in a different format such as key value pair--which Splunk likes and natively parses.

In this case, have the Blue Coat admins define a custom log format with the following fields:

Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

Since this data comes into Splunk as key=value pair now, Splunk parses it natively.

We just removed the TAs from the indexer and replaced it with a simpler props.conf file of this:

[bluecoat:proxysg:customclient]
SHOULD_LINEMERGE = false

This just turns off line merging which is on by default and makes the parsing even faster. Also remember to rename the props.conf and transforms.conf (ex: .bak files) included in the app if you have it installed on your search head--that contains the same complicated regex which will slow down data ingestion. Lastly, by defining your own format, you can add other fields you care about--such as the target IP (cs-ip) which is not included in the default bcreportermain_v1 format for some reason. We hope this helps others that run into this situation.

Conclusion

Again, this issue is not isolated to Blue Coat, but to any data source that has the ability to change the way it sends data. We were quite happy to find that Blue Coat provides that ability and it certainly reduced the load on the entire system and gave back those resources for adding other data.  Hat tip to Blue Coat for providing the flexibility of custom log formats.  Happy Splunking!