Wednesday, November 27, 2019

Making Splunk Dashboards Available Outside of Splunk

By Tony Lee

Have you ever built a beautiful Splunk dashboard that was not only aesthetically pleasing, but also incredibly insightful? If so, maybe others learned of its value and now want that data shared--even to users who may not have Splunk accounts. We had such a case in which the Chief Information Officer (CIO) wanted to add a particular Splunk panel to his intranet site for all employees to see. In this article we will recreate the scenario and show you how this can be accomplished. The two screenshots below show two possibilities of embedding Splunk panels into external sites.

Figure 1:  Example of  a single panel embedded into a page outside of Splunk

Figure 2:  Example of two panels embedded into a page outside of Splunk

Problem

There are quite a few issues that we need to solve, such as:
  • Splunk does not make it easy to share panels and especially entire dashboards outside of their platform
  • Some data may be sensitive in nature, so just remember the potential audience
    • Fortunately, this sharing can be disabled if needed
  • This solution needs to be long-term low maintenance which means no manual updates
  • The new website must be able to reach the Splunk search head via HTTPS

Potential Solution

The potential solution we are going to show uses scheduled saved reports to share out a panel. Here are the steps below:

1) Generate your insightful panel using the proper search.  Click Save As > Report

Figure 3:  Creating a report

2)  Select the content and time range selector

Figure 4:  Saving the report

3) After saving the report, let's schedule it

Figure 5: Schedule the report

4) Specify the schedule - this example updates the report every hour with the last hour of data

Figure 6:  Schedule parameters

5) Generate the embedded link by clicking Edit > Embed

Figure 7: Generating the embedded link

6)  Copy the embedded iframe link into the external site in question (Example site shown in Demo Page Code section below)

Figure 8:  Embedded iframe link

Conclusion

This is one possible solution that creates a low maintenance panel shared outside of Splunk. If you want to share an entire dashboard, this can be repeated for every panel in the dashboard. Just be cautious of the sensitivity of the data. If it is later determined that this data is no longer needed or should not be shared, it can be disabled. If you have a different method of sharing panels and especially entire dashboards, we would love to hear it.  Feel free to post it in the comment section below and as always happy Splunking.

Demo Page Code

This is just a demo page that contains two embedded saved reports.  The panel on the left is a single value number and the one on the right is a timechart.  Just remember to replace the two locations of:  "YOUR_EMBEDDED_LINK_HERE"


<HTML>
<HEAD>
<TITLE>This is an embedded demo</TITLE>
<style type="text/css">
<!--
td {
  height: 300px;
  vertical-align: center;
  text-align: center;
}
iframe {
  vertical-align: center;
  text-align: center;
}
-->
</style>
</HEAD>
<BODY>

<center><h2>CIO's Corner</h2></center>

<table style="width:100%" border=1>
  <tr>
    <th>Total Count</th>
    <th>Count over Time</th> 
  </tr>
  <tr>
    <td width="25%" height=300><iframe frameborder="0" scrolling="no" src="YOUR_EMBEDDED_LINK_HERE"></iframe></td>
    <td width="75%" height=300><iframe height="100%" width="100%" frameborder="0" scrolling="no" src="YOUR_EMBEDDED_LINK_HERE"></iframe></td>
  </tr>
</table>

</BODY>
</HTML>


Tuesday, November 12, 2019

Building a DirtyNet - The Ultimate Malware Playground

By Tony Lee


Do you have a safe place to handle and investigate potential malware? Maybe you are you doing this on your home PC or in a VM to try to protect the rest of your network?  Whether you have a DirtyNet or not, let’s start an open discussion that covers some safe malware handling possibilities along with the pros and cons of each. Maybe this will help provide justification or a framework for even a minimal amount of funding to build something official and supported.

Figure 1:  DirtyNet Information Flow

 
The Benefits

There are many benefits to building a DirtyNet to take a closer look at potential malware, here are a few:

  • Determine maliciousness of malware
    • Never seen before binaries – Threat intel and malware repositories know nothing about it
    • Low conviction rate – Threat intel and repositories have seen it, but are not convinced of the maliciousness

  • Extract Indicators of Compromise (IOCs)
    • Useful in hunting for presence in the rest of the environment

  • Protect the rest of the network
    • Prevent users from mishandling malware on their corporate PCs by providing a dedicate environment

 
Possible Solutions
There are probably a large number of potential solutions – many of them bad ideas and some good ideas. We will cover the two most common setups here:

1) Stand-alone Malware Analysis Laptops

This is very common because it is easy to implement and somewhat reduces the risk of accidental malware spread.  There are some pros and cons that should be noted though:

Pros

  • Lower risk of mishandled malware than using a VM on your corp machine

Cons
  • Requires sneaker net – USB drives with write-blockers
  • Not real-time feeding from sensors
  • Difficult to connect/automate sensors and analysis stack
  • Difficult to scale to a large amount of malware
  • Difficult to extract and harvest IOC for reuse

 
2) Building a DirtyNet

This is not always the route chosen, but when built correctly, it is a beautiful and graceful solution. Here are the pros and cons:


Pros
  • Can be automated
  • Flexible sensor integration
  • Flexible automated malware analysis stack
  • Scales to higher volume of malware


Cons
  • Higher risk of mishandled malware when not following proper procedures

 
DirtyNet Rules

Notice the single con of the DirtyNet…  But just like fight club, it is important that EVERYONE follow the rules:

  • Write-only file share exposed to larger environment to accept malware and move inside DirtyNet
  • Malware stays within DirtyNet.  No outbound traffic except whiteline for FireEye AX / Malware analysis stack
  • Malware can be taken from the Malware repo and moved to an analysis workstation within DirtyNet
  • Malware is ideally password protected zipped with “infected” before transfer and during storage
  • Analyst connection to dirty net does not compromise the rest of the network. Virtualized desktops through a browser plugin with no file sharing is a decent option.

 
Requirements

So you want to build a DirtyNet now... and you are curious as to what you might need. The info below contains the high-level requirements:
  • Hardware or Virtual (AWS/Azure/GCP) capacity 
  • Whiteline randomized POP for malware to make callouts (ideal but not necessary)
  • Viper or similar framework for the Malware repository
  • Time from a network engineer to set up network and hosts


Conclusion
Now that you have the blueprint, will you be building a DirtyNet of your own? Also, if you already have a DirtyNet, we would be interested in hearing your tips, tricks, or opinions posted in the comment section below. As a bonus, do you have any cool integrations setup?  For example, we have CyBot’s (https://github.com/cylance/CyBot) Cuckoo plugin safely feeding out instance with malware. Thanks for reading!

Figure 2:  CyBot Cuckoo plugin safe submission to DirtyNet


Wednesday, September 18, 2019

Syslog Over TLS Data Feed Splitting via syslog-ng

By Tony Lee

There are times where you may only be able to send data to a single syslog receiver, but all is not lost. Fortunately, many syslog receivers can handle splitting the feed. In this article we will use syslog-ng to show the steps needed to split a syslog feed (possibly for the purpose of taking a single feed and sending it to two SIEMs).  As a bonus, we will give you the steps to also receive and split a TLS encrypted syslog feed.


Figure 1:  Unencrypted syslog feed splitting

Basic Requirements

Since we are not handling much volume in this example, we just need minimum requirements:

Our setup
  • Amazon EC2 instance
  • Ubuntu Server 18.04 LTS (Red Hat is also possible, but requires different commands)
  • 1 vCPU, 1GB RAM, 10GB HD

Basic Setup Steps

The syslog-ng package was not available out of the box for us so we had to run:

sudo apt update

Install and create a basic configuration file

sudo su -
apt install syslog-ng
mv syslog-ng.conf syslog-ng.bak
vim syslog-ng.conf

Note:  We used the same version found in the “Config version” field after running “syslog-ng --version”


@version: 3.13

source s_input { tcp(ip(0.0.0.0) port(10514)); };
destination d_siem{ udp("10.0.0.1" port(7514) template("$MSG") ); };
destination d_siem2{ tcp("10.0.0.2" port(7515) template("$MSG") ); };

log{
source(s_input);
destination(d_siem);
destination(d_siem2);
};


Configuration explanation:

  • template("$MSG") ensures that syslog-ng does not add another header to the message
  • Single input called s_input listening on all interfaces over TCP port 10514
  • Two destinations
1) d_siem sending to 10.0.0.1 on UDP 514
2) d_siem2 sending to 10.0.0.2 on TCP 515

Validate the configuration file
Use the following command to validate and troubleshoot syslog-ng configuration issues:

syslog-ng -s -f syslog-ng.conf


Start/restart the Service
Use the following command to check and restart the syslog-ng service:

service syslog-ng status
service syslog-ng restart


Validate the port is open
Use the following command to validate listening port is open:

netstat -an | grep :10514
tcp        0      0 0.0.0.0:10514           0.0.0.0:*               LISTEN


Test the Splitting
Since we are splitting TCP syslog in our example, we can use a web browser to test it:

http://<Your_syslog-ng_server>:10514


Figure 2:  Showing port details

Seeing the results in on the final server on UDP 7514 and TCP 7514


Figure 3:  Confirmation that the UDP 7514 split is working


Figure 4:  Confirmation that the TCP 7515 split is working

Enabling Syslog over TLS on the Input

We mentioned that there would be a bonus, now let’s enable TLS.
Note: This should be a requirement if your data is traversing a public network such as the Internet.

Creating private Certificate Authority (CA)
If you don’t have your own CA or choose not to use a public CA, you will need to create a CA on your syslog-ng server to sign certificates

sudo su –
cd /root
mkdir CA
cd CA
mkdir certs crl newcerts private
echo "01" > serial
cp /dev/null index.txt
cp /etc/ssl/openssl.cnf .
vi openssl.cnf

Change from:
dir = ./demoCA
to:
dir = .

:wq to save and exit vi

Generate the CA certificate
openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -config openssl.cnf

Creating Server Certificate Request
Whether you use the local CA or an external/public CA, you will need to generate a server certificate request.

openssl req -nodes -new -x509 -keyout serverkey.pem -out serverreq.pem -days 365 -config openssl.cnf
openssl x509 -x509toreq -in serverreq.pem -signkey serverkey.pem -out tmp.pem

Common name should be the IP address or FQDN of your server. Leave email field blank.

Signing Server Certificate with CA
Next you can either get the serverreq.pem signed by your external CA, or sign it locally with the next step:

openssl ca -config openssl.cnf -policy policy_anything -out servercert.pem -infiles tmp.pem
rm tmp.pem

Either way you should end up with servercert.pem and serverkey.pem files, which will be used as
the server certificate and key for TLS encryption.

Copying Certificates to Syslog Directory

mkdir /etc/syslog-ng/cert.d
mkdir /etc/syslog-ng/key.d
mkdir /etc/syslog-ng/ca.d (if using local Certificate Authority)
cp cacert.pem /etc/syslog-ng/ca.d/ (if using local Certificate Authority)
cp servercert.pem /etc/syslog-ng/cert.d/
cp serverkey.pem /etc/syslog-ng/key.d/
cd /etc/syslog-ng/cert.d/
chown root servercert.pem
chmod 600 servercert.pem
cd /etc/syslog-ng/key.d/
chown root serverkey.pem
chmod 600 serverkey.pem
cd /etc/syslog-ng/ca.d/ (if using local Certificate Authority)
chown root cacert.pem (if using local Certificate Authority)
chmod 600 cacert.pem (if using local Certificate Authority)

Modify syslog-ng configuration file to use TLS
vim syslog-ng.conf


@version: 3.13

source s_input { tcp(ip(0.0.0.0) port(10514) tls(key_file("/etc/syslog-ng/key.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") peer_verify(optional-untrusted)) ); };
destination d_siem{ udp("10.0.0.1" port(7514) template("$MSG") ); };
destination d_siem2{ tcp("10.0.0.2" port(7515) template("$MSG") ); };

log{
source(s_input);
destination(d_siem);
destination(d_siem2);
};


Validate the configuration file
Use the following command to validate and troubleshoot syslog-ng configuration issues:

syslog-ng -s -f syslog-ng.conf


Start/restart the Service
Use the following command to check and restart the syslog-ng service:

service syslog-ng status
service syslog-ng restart

Test the Splitting
Since we are splitting TCP over TLS syslog in our example, we can use a web browser to test it:
https://<Your_syslog-ng_server>:10514



Figure 5:  TLS enabled on the Input

Enabling Syslog over TLS on the Input & Output

Let’s go for the double bonus now and enable TLS on the Input and TCP Output
Note: This should be a requirement if your data is traversing multiple public networks such as the Internet.

Modify syslog-ng configuration file to use TLS

vim syslog-ng.conf


@version: 3.13

source s_input { tcp(ip(0.0.0.0) port(10514) tls(key_file("/etc/syslog-ng/key.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") peer_verify(optional-untrusted)) ); };
destination d_siem{ udp("10.0.0.1" port(7514) template("$MSG") ); };
destination d_siem2{ tcp("10.0.0.2" port(7515) tls(key_file("/etc/syslog-ng/key.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") peer_verify(optional-untrusted)) template("$MSG") ); };

log{
source(s_input);
destination(d_siem);
destination(d_siem2);
};


Validate the configuration file
Use the following command to validate and troubleshoot syslog-ng configuration issues:

syslog-ng -s -f syslog-ng.conf


Start/restart the Service
Use the following command to check and restart the syslog-ng service:

service syslog-ng status
service syslog-ng restart

Test the Splitting
Since we are splitting TCP over TLS syslog in our example, we can use a web browser to test it:
https://<Your_syslog-ng_server>:10514


Figure 6:  TLS enabled on input and TCP output
Note:  We cannot TLS encrypt the UDP data stream

Set Service to Autostart on Boot

If all is well, now we can set the service to autostart at boot – just in case we experience a reboot.

sudo update-rc.d syslog-ng defaults

Added Security

Hopefully the source and destination IP addresses are static and known. This allows us to further protect the syslog ports by adding one to one firewall rules. This can be performed at both the network and host layer.

Caching Capabilities

One other option is to add some caching capabilities on this server by adding another destination of a local file. With enough disk space, it could provide some time to collect logs from the syslog-ng server before they rolled in the case of a network issue between the syslog-ng server and the downstream host.

Conclusion

In this article we showed how we can set up an intermediary syslog-ng server to split a single feed into multiple feeds. As a bonus, we also showed how we could secure the syslog traffic by enabling TLS to wrap the data with encryption. We hope this article helps get you up and running quicker when you need to split and secure that next feed.

Thanks

Zack Link for the original write up using Red Hat Linux. Syslog-ng team for the great software and documentation (https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/mutual-authentication-using-tls#TOPIC-956369).