rsyslog fun - Basic Splunk Log Collection and Forwarding - Part II

By Tony Lee

Welcome to part II in our series covering how to use rsyslog to route and forward logs to Splunk. Please see Part I of the series (http://securitysynapse.blogspot.com/2019/01/rsyslog-fun-basic-splunk-log-collection-part-i.html) for the basics in opening ports, routing traffic by IP address or hostname, and monitoring files to send the data on to Splunk Indexers. As a reminder, choosing between rsyslog, syslog-ng, or other software is entirely up to the reader and may depend on their environment and approved/available software. We also realize that this is not the only option for log architecture or collection, but it may help those faced with this task—especially if rsyslog is the standard in their environment. That said, let's look at some more advanced scenarios concerning file permissions, routing logs via regex, and routing logs via ports. We will wrap up with some helpful hints on a possible method to synchronize the rsyslog and Splunk configuration files.

File Permissions

There are times where you may need to adjust the file permissions for the files that rsyslog is writing to disk. For example, if following best practice and running the Splunk Universal Forwarder as a lower privileged account, it will need access to the logs files.  Using the following rsyslogd.conf directives at the top of the configuration file will change the permissions on the directories and files created.  The following example creates directories with permissions of 755 and files with a permission of 644:

$umask 0000 $DirCreateMode 0755 $FileCreateMode 0644

Routing logs via Regex

Another more advanced rsyslog option is the ability to drop or route data at the event level via regex. For example, maybe you want to drop certain packets -- such as Cisco Teardown packets generated from ASA's. Note: this rsyslog ability is useful since we are using Splunk Universal Forwarders in our example and not Splunk Heavy Forwarders.

Or maybe you have thousands of hosts and don't want to maintain a giant list of IP addresses in an if-statement. For example, maybe you want to route thousands of Cisco Meraki host packets to a particular file via a regex pattern.

Possibly even more challenging would be devices in a particular CIDR range that end in a specific octet.

These three examples are covered in the rsyslog.conf snippet below:

#Drop Cisco ASA Teardown packets :msg, contains, ": Teardown " ~ & stop #Route Cisco Meraki hosts to specific directory if ($msg contains ' events type=') then ?ciscoMerakiFile & stop #ICS Devices 10.160.0.0/11 (last octet being .150) :fromhost-ip, regex, "10\.\\(1[6-8][0-9]\\|19[0-1]\\)\..*\.150" -?icsDevices & stop

Routing logs via Port

I know we just provided you the ability to route packets via regex, however sometimes that can be inefficient--especially at high events per second. If you are really fortunate, the source sending the data has the ability to send to a different port. Then it may be worth looking into routing data to different files based on port.  The example file below provides port 6517 and 6518 as an example.

#Dynamic template names template(name="file6517" type="string" string="/rsyslog/port6517/%FROMHOST%/%$YEAR%-%$MONTH%-%$DAY%.log") template(name="file6518" type="string" string="/rsyslog/port6518/%FROMHOST%/%$YEAR%-%$MONTH%-%$DAY%.log") #Rulesets  ruleset(name="port6517"){     action(type="omfile" dynafile="file6517") } ruleset(name="port6518"){     action(type="omfile" dynafile="file6518") } input(type="imtcp" port="6517" ruleset="port6517") input(type="imtcp" port="6518" ruleset="port6518")

Synchronizing Multiple Rsyslog Servers

Since our architecture in part I outlined using a load balancer and multiple rsyslog servers, we will eventually need a way to synchronize the configuration files across the multiple rsyslog servers.  The example below provides two bash shell scripts to perform just that task. The first one will synchronize the rsyslog configuration and the second will synchronize the Splunk configuration--both scripts restart the respective service. Note: This is not the only method available for synchronization, but it is one possible method. Remember to replace <other_server> with the actual IP or FQDN of that server.

On the rsyslog server that you make the changes on, create these two bash scripts and modify the <other_server> section. Once you make a change to the rsyslog or Splunk UF configuration, run the necessary script.

sync-rsyslog.sh

scp /etc/rsyslog.conf <other_server>:/etc/rsyslog.conf ssh <other_server> service rsyslog restart

sync-splunk.sh

scp /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local/inputs.conf <other_server>:/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local/inputs.conf ssh <other_server> /opt/splunkforwarder/bin/splunk restart

Conclusion

In this article, we outlined key advanced features within rsyslog that may not be immediately evident. Hopefully this article will save you some Googling time when trying to operationalize log collection and forwarding using rsyslog in your environment. After all, eventually you will probably need to deal with file permissions, routing logs via regex and/or port, and configuration synchronization. We hope you enjoyed the article and found it useful.  Feel free to post your favorite tips and tricks in the comments section below. Happy Splunking!

Basic Troubleshooting

So you did everything above and you are still not seeing data...  Walk through some of these steps:

1. Ensure all network firewalls permit the traffic
2. Ensure iptables allows traffic to the rsyslog server
3. Run tcpdump on the source to ensure it is sending data
4. Run tcpdump on the rsyslog server to ensure it is receiving data
5. Verify permissions when writing the files to disk
6. Attempt to telnet (or even web browse) to the rsyslog server port to see if anything is written to the directories