Splunk: The unsung hero of creative mainframe logging

By Tony Lee

The situation

Have you ever, in your life, heard a good sentence that started with: “So, we have this mainframe... that has logging and compliance requirements…” Yeah, me neither. But this was a unique situation that required a quick and creative solution--and it needed to be done yesterday.  Queue the horror music.

In summary:  We needed to quickly log and make sense of mainframe data for reporting and compliance reasons. The mainframe did not support external logging such as syslog. However, the mainframe could produce a CSV file and that file could be scheduled to upload to an FTP server (Not SFTP, FTPS, or SCP).  Yikes!

Possible solutions

We could stand up an FTP server and use the Splunk Universal forwarder to monitor the FTP upload directory, but we did not have extra hardware or virtual capacity readily available. After a quick Google search, we ran across this little gem of an app called the Splunk FTP Reviver app (written by Luke Murphey):  https://splunkbase.splunk.com/app/3318/. This app cleverly creates a python FTP server using Splunk—best of all, it leverages Splunk’s user accounts and role-based access controls.

How it worked

At a high level, here are the steps involved:

1. Install the FTP Receiver app:  https://splunkbase.splunk.com/app/3318/
2. Create an index for the mainframe data (Settings -> Indexes -> New -> Name: mainframe)
3. Create an FTP directory for the uploaded files (mkdir /opt/splunk/ftp)
4. Create FTP Data input (Settings -> Data Inputs -> Local Inputs -> FTP -> New -> name: mainframe, port: 2121, path: ftp, sourcetype: csv, index: mainframe)
5. Create a role with the ftp_write privileges (Settings -> Access Controls -> Roles: Add new -> Name: ftp_write, Capabilities: ftp_write)
6. Create a Splunk user for the FTP Receiver app (Settings -> Access Controls -> Users: Add new -> Name: mainframe, Assign to roles: ftp_write)
7. Configure the mainframe to send to the FTP Receiver app port (on your own for that one)
8. Create a local data input to monitor the FTP upload directory and ingest as CSV (Settings -> Data inputs -> Local inputs -> Files and Directories -> New -> Browse to /opt/splunk/ftp -> Continuously monitor -> Sourcetype: csv, index: mainframe)

Illustrated, the solution looks like this:

Figure 1:  Diagram of functional components

If you run into any issues, troubleshoot and confirm that the FTP server is working via a common web browser.

Figure 2:  Troubleshooting with the web browser

Conclusion

Putting aside concerns that the mainframe may be older than most of the IT staff and the fact that FTP is still a clear-text protocol, this was an interesting solution that was created using the flexibility of Splunk. Add some mitigating controls and a little bit of SPL + dashboard design and it may be the easiest and most powerful mainframe reporter in existence.

Figure 3:  Splunk rocks, the process works

CyBot – Threat Intelligence Chat Bot

By Tony Lee

Introduction

For those who could not attend, this year’s Black Hat security conference did not disappoint.  It was an awesome time to collaborate and share with the security community.  In doing so, we open sourced a new tool at Black Hat Arsenal at aimed at assisting Security Operations Centers (SOCs) and digital first responders.  We affectionately call it:  CyBot – Threat Intelligence Chat Bot.

We understand that typical SOC environments face a number of challenges:

• Many SOCs are overwhelmed with the number of incoming alerts
• Service Level Agreements (SLAs) often define a maximum time to investigate and contain an incident
• Security tools may be plentiful but they are often not centralized
• Collaboration on a large investigation may be challenging

We have even seen cases where the SOC receives so many alerts that all of them may not be properly investigated.

To combat this, CyBot can be your threat intelligence chat bot waiting to do research for you.  For example, instead of going to various websites or dashboards to perform research, you could just ask CyBot simple questions and even share results with other investigators.  All from within one chat window you can do the following and more:

·         Ask about the threat reputation of URLs and hashes

·         Perform WHOIS, nslookup, and geoip lookups

·         Unshorten potentially malicious shortened URLs

·         Extract links from a potentially malicious website

CyBot Menu

Best of all, this capability is now free and being actively developed.  All documentation, slides, and plugins have been made publicly available via github:  https://github.com/CylanceSPEAR/CyBot.

Props

Very few tasks are ever accomplished in complete isolation.  Tools, services, and ideas were combined from awesome places such as:

• Errbot developers for the fantastic tool and customer service
• VirusTotal
• geoip - freegeoip.net
• Google Safebrowsing - https://developers.google.com/safe-browsing/ and Jun C. Valdez Hashid - C0re
• Unshorten - unshorten.me
• Codename - https://mark.biek.org/code-name/ Black Hat Arsenal team for the amazing support and tool release venue

• Non-bots: Bill Hau, Corey White, Dennis Hanzlik, Ian Ahl, Dave Pany, Dan Dumond, Kyle Champlin, Kierian Evans, Andrew Callow, Mark Stevens