Fun with Microsoft Power BI - Part II - Reporting

By Tony Lee

Welcome to part II in this series of going from zero to hero using Power BI to ingest, process, and make amazing reports.  If you have read some of our other articles you can probably tell by now that we enjoy making data actionable. Honestly, it doesn't matter what type of data or even where the data ends up. As long as we can make informed decisions using the data -- we love it. Following in this theme we are going to make BlackBerry (formerly known as Cylance) Protect Threat Data Report (TDR) CSVs actionable using Power BI and Power BI Desktop. You can use any data source to follow along in this series, but our example BlackBerry Protect report is shown below which we will happily share the Power BI file at the end of the series for you to load and analyze your own data, so stay tuned for that!

Figure 1:  Our Power BI report using BlackBerry Protect TDR data

In the first article, we covered:

• Getting Started
• Data Ingest
• Adjusting Fields
• Visualizations
• Saving Your Work

In this article, we will cover:

• Tabs
• More Visualizations
• Text box
• Slicer
• Table
• Pie Charts
• Treemap
• Using Reports and Dashboards
• Uploading Reports to Power BI Service (Online)

Tabs

Just a quick note for folks that are familiar with excel tabs. In Power BI, they work much the same. Since the canvas size is somewhat fixed in size, tabs might come in handy for building multiple reports and overcoming canvas limitations.

Figure 2: Shows boundaries and tabs

More Visualizations

In the first article, we showed you how to add an event count, which essentially counted the number of rows in our BlackBerry Protect Threat Data Report. We started off with an easy visualization and will progress to more difficult ones.

Text box

We are fans of being able to quickly glance at a dashboard and know what it is...  thus a simple text box at the top is a great way to accomplish this task.  Insert > Text Box.  Then get fancy fancy with the background color and font. We will get even fancier in a later article using images with hyperlinks, but let's crawl, walk, then fly.

Slicer

We are also fans of being able to interact with the data by carving and drilling down into interesting events. For this, we use a slicer (Power BI term), also known as a filter to the rest of the world. ;-) In our BlackBerry Protect example, we are filtering on multiple fields using a checkbox in most cases. To do this, simply click in the canvas, click on the slicer icon, drag the field you want to filter on, and bob's your uncle. 

Figure 3:  Check box filters (aka slicers)

You may have also noticed that we have a time range selector in our report. This is also a slicer, but we dragged a time field into the field box and Power BI is smart enough to convert it from a check box to a time range selector. For our example, we used the "Create Time" field for the malware. Note that there are additional options that specify how the time range selector behaves. Those are configurable by using the down arrow. Our preference for this scenario is "Relative Date" which allows you to specify a relative time range such as:  Last, Next, or This: X days, weeks, months, and years.

Figure 4:  Configuring how the time range selector works

Table

Tables are fairly straight forward to use as well. To use, simply click on the canvas > click the table icon > Drag the fields you want in the table.  The down arrow allows you to correct the order of appearance if necessary by moving the field up and down the list or you can drag the box up or down in the list.

Doughnut and Pie Charts

We saved doughnut and pie charts to later in this article because it is not as straight forward to create it depending on your data set. We used a doughnut chart in our example for "File Status" and "Detected By". The problem with these fields is that the data source does not provide the statistical breakdown (percentage per file status or what detected the malware). That must be calculated and Power BI may or may not do that automagically. For this, it helps to know a little bit of DAX to create measures. Per Microsoft:

"Data Analysis Expressions (DAX) is a library of functions and operators that can be combined to build formulas and expressions in Power BI, Analysis Services, and Power Pivot in Excel data models."

"Measures are dynamic calculation formulas where the results change depending on context. Measures are used in reporting that support combining and filtering model data by using multiple attributes such as a Power BI report or Excel PivotTable or PivotChart. Measures are created by using the DAX formula bar in the model designer."

Source:  https://docs.microsoft.com/en-us/dax/ 

These formulas are heavily dependent upon functions and there are a lot of them. A good reference is found here: https://docs.microsoft.com/en-us/dax/dax-function-reference

As an example... say you had a column called "Sales Amount" and you wanted to sum that and store it in a measure called "Total Sales".  The DAX for that would look like the following:

Total Sales = SUM([Sales Amount])

Now that we have a bit more understanding, we will create the "File Status" doughnut chart. To do this, we went to the Modeling menu > New measure > and enter:

File Status Stats = DIVIDE(count('ThreatsDataReport'[File Status]),CALCULATE(count('ThreatsDataReport'[File Status]),ALLSELECTED(('ThreatsDataReport'))))

Note:

• ThreatsDataReport is our table, the entries in the brackets [] are the columns
• ALLSELECTED is so our slicers (filters) still work when selected

Once we click the checkmark, we should have a new field called "File Status Stats".  We will click the canvas, click the pie or doughnut chart icon, and drag the "File Status Stats" field to the Values box in visualization and we will drag "File Status" to the Legend box.

Figure 5:  Creating a measure and a doughnut chart in Power BI

Doughnut and Pie Charts - Cont.

Note: After some trial and error, we discovered a potentially easier way to create doughnuts and pie charts showing top values. Select either pie or doughnut chart and drag the field into the Legend box and drag the same field into the Values box and make sure Count is selected for Values. This creates the same pie and doughnut chart as our DAX fanciness above.

Pro-tip:  You can now change this visualization to a table to show an easy to read statistical breakdown (Top or Rare).

Figure 6:  Create a pie or doughnut chart while avoiding DAX

Treemap

Fortunately, creating a treemap is similar to creating a doughnut or pie chart. Once you have the DAX (or logic) created, it is all downhill from there. In our example, the malware classification data made an awesome treemap visualization. For this treemap, you can either drag the same field twice (with Value as a Count) as we showed in the example above, or you can use DAX for the calculation.

Our DAX to calculate the classification stats measure:

Classification Stats = DIVIDE(count('ThreatsDataReport'[Classification]),CALCULATE(count('ThreatsDataReport'[Classification]),ALLSELECTED(('ThreatsDataReport'))))

To create the treemap, click the canvas, then the treemap icon, drag the "Classification" field to the Group box, and the "Classification Stats" field to the Values box.

Figure 7:  Creating a treemap using the malware classifications

Note: Most visualizations apply a filter when clicked. Not all of them are very intuitive in how to clear the filter. For the treemap, either click the box you already clicked or click the top left box to clear the filter.

Using Reports and Dashboards

Now that you have a report built, you can use it as-is in Power BI Desktop. The slicers function to carve up the data as needed, however there is no presentation view or mode. Thus your presentation to others will not look as polished as it could. To get around this problem, we can take the .pbix file and upload it (or publish it) to a personal workspace in Power BI Service where you can use it as individual reports or pin visualizations from multiple reports to make a dashboard.

Uploading Reports to Power BI Service (Online)

Upfront warning:  Be careful publishing the work since the .pbix contains your data set as well and could potentially leak information.

Keep in mind that there are multiple methods to load a local Power BI Desktop file into the online service. You can use the Publish button in Power BI Desktop or you can log in to your Power BI account:  https://powerbi.microsoft.com/en-us/landing/signin/

Click "My Workspace" > "Get Data" > "Local File" > navigate to your .pbix file

Figure 8:  Uploaded .pbix to Power BI Service workspace

This should now be in your personal workspace and you can view the report via your web browser. To enter a full screen presentation mode, click the double arrow in the top right corner of the window.

Figure 9:  Entering full screen presentation mode

Conclusion

In part II of our series we covered how to use tabs and many more visualizations--including some that are not so straight forward. We only scratched the surface of DAX since that could be a blog series unto itself. But we also discovered the presentation limitation in Power BI Desktop and how to over come that presentation by using the Power BI Service. We hope that you are now able to take data, ingest it, create a report, and present that report in a professional setting. Our future articles will focus on improving the usefulness and functionality of carving and displaying data using Power BI. Please feel free to leave feedback and your favorite Power BI features in the comments section below. Stay tuned for more great Power BI info!

rsyslog fun - Basic Splunk Log Collection and Forwarding - Part II

By Tony Lee

Welcome to part II in our series covering how to use rsyslog to route and forward logs to Splunk. Please see Part I of the series (http://securitysynapse.blogspot.com/2019/01/rsyslog-fun-basic-splunk-log-collection-part-i.html) for the basics in opening ports, routing traffic by IP address or hostname, and monitoring files to send the data on to Splunk Indexers. As a reminder, choosing between rsyslog, syslog-ng, or other software is entirely up to the reader and may depend on their environment and approved/available software. We also realize that this is not the only option for log architecture or collection, but it may help those faced with this task—especially if rsyslog is the standard in their environment. That said, let's look at some more advanced scenarios concerning file permissions, routing logs via regex, and routing logs via ports. We will wrap up with some helpful hints on a possible method to synchronize the rsyslog and Splunk configuration files.

File Permissions

There are times where you may need to adjust the file permissions for the files that rsyslog is writing to disk. For example, if following best practice and running the Splunk Universal Forwarder as a lower privileged account, it will need access to the logs files.  Using the following rsyslogd.conf directives at the top of the configuration file will change the permissions on the directories and files created.  The following example creates directories with permissions of 755 and files with a permission of 644:

$umask 0000 $DirCreateMode 0755 $FileCreateMode 0644

Routing logs via Regex

Another more advanced rsyslog option is the ability to drop or route data at the event level via regex. For example, maybe you want to drop certain packets -- such as Cisco Teardown packets generated from ASA's. Note: this rsyslog ability is useful since we are using Splunk Universal Forwarders in our example and not Splunk Heavy Forwarders.

Or maybe you have thousands of hosts and don't want to maintain a giant list of IP addresses in an if-statement. For example, maybe you want to route thousands of Cisco Meraki host packets to a particular file via a regex pattern.

Possibly even more challenging would be devices in a particular CIDR range that end in a specific octet.

These three examples are covered in the rsyslog.conf snippet below:

#Drop Cisco ASA Teardown packets :msg, contains, ": Teardown " ~ & stop #Route Cisco Meraki hosts to specific directory if ($msg contains ' events type=') then ?ciscoMerakiFile & stop #ICS Devices 10.160.0.0/11 (last octet being .150) :fromhost-ip, regex, "10\.\\(1[6-8][0-9]\\|19[0-1]\\)\..*\.150" -?icsDevices & stop

Routing logs via Port

I know we just provided you the ability to route packets via regex, however sometimes that can be inefficient--especially at high events per second. If you are really fortunate, the source sending the data has the ability to send to a different port. Then it may be worth looking into routing data to different files based on port.  The example file below provides port 6517 and 6518 as an example.

#Dynamic template names template(name="file6517" type="string" string="/rsyslog/port6517/%FROMHOST%/%$YEAR%-%$MONTH%-%$DAY%.log") template(name="file6518" type="string" string="/rsyslog/port6518/%FROMHOST%/%$YEAR%-%$MONTH%-%$DAY%.log") #Rulesets  ruleset(name="port6517"){     action(type="omfile" dynafile="file6517") } ruleset(name="port6518"){     action(type="omfile" dynafile="file6518") } input(type="imtcp" port="6517" ruleset="port6517") input(type="imtcp" port="6518" ruleset="port6518")

Synchronizing Multiple Rsyslog Servers

Since our architecture in part I outlined using a load balancer and multiple rsyslog servers, we will eventually need a way to synchronize the configuration files across the multiple rsyslog servers.  The example below provides two bash shell scripts to perform just that task. The first one will synchronize the rsyslog configuration and the second will synchronize the Splunk configuration--both scripts restart the respective service. Note: This is not the only method available for synchronization, but it is one possible method. Remember to replace <other_server> with the actual IP or FQDN of that server.

On the rsyslog server that you make the changes on, create these two bash scripts and modify the <other_server> section. Once you make a change to the rsyslog or Splunk UF configuration, run the necessary script.

sync-rsyslog.sh

scp /etc/rsyslog.conf <other_server>:/etc/rsyslog.conf ssh <other_server> service rsyslog restart

sync-splunk.sh

scp /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local/inputs.conf <other_server>:/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local/inputs.conf ssh <other_server> /opt/splunkforwarder/bin/splunk restart

Conclusion

In this article, we outlined key advanced features within rsyslog that may not be immediately evident. Hopefully this article will save you some Googling time when trying to operationalize log collection and forwarding using rsyslog in your environment. After all, eventually you will probably need to deal with file permissions, routing logs via regex and/or port, and configuration synchronization. We hope you enjoyed the article and found it useful.  Feel free to post your favorite tips and tricks in the comments section below. Happy Splunking!

Basic Troubleshooting

So you did everything above and you are still not seeing data...  Walk through some of these steps:

1. Ensure all network firewalls permit the traffic
2. Ensure iptables allows traffic to the rsyslog server
3. Run tcpdump on the source to ensure it is sending data
4. Run tcpdump on the rsyslog server to ensure it is receiving data
5. Verify permissions when writing the files to disk
6. Attempt to telnet (or even web browse) to the rsyslog server port to see if anything is written to the directories

rsyslog fun - Basic Splunk Log Collection and Forwarding - Part I

By Tony Lee

We found it a bit surprising that there are so few articles on how to use an rsyslog server to forward logs to Splunk. This provided the motivation to write this article and hopefully save others some Googling. Choosing between rsyslog, syslog-ng, or other software is entirely up to the reader and may depend on their environment and approved/available software. We realize that this is not the only option for log architecture or collection, but it may help those faced with this task—especially if rsyslog is the standard in their environment.

Warnings

Before we jump in, we wanted to remind you of three potential gotchas that may thwart your success and give you a troubleshooting migraine.

1. Network firewalls – You may not own this, but make sure the network path is clear
2. iptables – Complex rule sets can throw you for a loop
3. SE Linux – Believe it or not, SE Linux when locked down can prevent the writing of the log files

If something is not working the way you expect it to work, it is most likely due to one of the three items mentioned above. It could be worth temporarily disabling them until you get everything working. Just don’t forget to go back and lock it down.

Note:  We will also be using Splunk Universal Forwarders (UF) in this article.  Universal Forwarders have very little pre-processing or filtering capabilities when compared to Heavy Forwarders.  If significant filtering is necessary, consider using a Splunk Heavy Forwarder in the same fashion as we are using the UFs below.

Architecture

Whether your Splunk instance is on-prem or it is in the cloud, you will most likely need syslog collectors and forwarders at some point. The architecture diagram below shows one potential configuration. The number of each component is configurable and dependent upon the volume of traffic.

Figure 1:  Architecture diagram illustrating traffic flow from data sources to the Index Cluster

Rsyslog configuration

Rsyslog is a flexible service, but in this case rsyslog’s primary role will be to:

• Open the sockets to accept data from the sources
• Properly route traffic to local temporary files that Splunk will forward on to the indexers

If you are fortunate enough to be able to route network traffic to different ports, you may be able to reduce the if-then logic shown below for routing the events to separate files. In this case, we were not able to open separate ports from the load balancer, thus we needed to do the routing on our end. In the next article we will cover more advanced routing to include regex and traffic coming in on different ports.

Note:  Modern rsyslog is designed to run extra config files that exist in the /etc/rsyslog.d/ directory. If that directory exists, place the following 15-splunk-rsyslog.conf file in that directory. Otherwise, the /etc/rsyslog.conf file is interpreted from top to bottom, so make a copy of your current config file (cp /etc/rsyslog.conf /etc/rsyslog.bak) and selectively add the following at the top of the new active rsyslog.conf file. This addition to the rsyslog configuration will do the following (assuming the day is 2018-06-01:

• Open TCP and UDP 514
• Write all data from 192.168.1.1 to:  /rsyslog/cisco/192.168.1.1/2018-06-01.log
• Write all data from 192.168.1.2 to:  /rsyslog/cisco/192.168.1.2/2018-06-01.log
• Write all data from 10.1.1.* to /rsyslog/pan/10.1.1.*/2018-06-01.log (where * is the last octet of the source IP
• Write all remaining data to /rsyslog/unclaimed/<host>/2018-06-01.log (where <host> is the source IP or hostname of the sender)

Note:  If the rsyslog server sees the hosts by their hostname instead of IP address, feel free to use $fromhost == '<hostname>' in the configuration file below.

/etc/rsyslog.d/15-splunk-rsyslog.conf

$ModLoad imtcp $ModLoad imudp $UDPServerRun 514 $InputTCPServerRun 514 # do this in FRONT of the local/regular rules $template ciscoFile,"/rsyslog/cisco/%fromhost%/%$YEAR%-%$MONTH%-%$DAY%.log" $template PANFile,"/rsyslog/pan/%fromhost%/%$YEAR%-%$MONTH%-%$DAY%.log" $template unclaimedFile,"/rsyslog/unclaimed/%fromhost%/%$YEAR%-%$MONTH%-%$DAY%.log" if ($fromhost-ip == '192.168.1.1' or $fromhost-ip == '192.168.1.2') then ?ciscoFile & stop if $fromhost-ip startswith '10.1.1' then ?PANFile & stop else ?unclaimedFile & stop # local/regular rules, like *.* /var/log/syslog.log

Note:  Rsyslog should create directories that don't already exist, but just in case it doesn't, you need to create the directories and make them writable.  For example:

mkdir -p /rsyslog/cisco/ mkdir -p /rsyslog/pan/ mkdir -p /rsyslog/unclaimed/

Pro tip:  After making changes to the rsyslog config file, you can verify that there are no syntax errors BEFORE you restart the rsyslog daemon.  For a simple rsyslog config validation.  Try using the following command: 

rsyslogd -N 1

If there are no errors, then you should be good to restart the rsyslog service so your changes take effect:

service rsyslog restart

Log cleanup

The rsyslog servers in our setup are not intended to store the data permanently. They are intended to act as a caching server for temporary storage before shipping the logs off to the Splunk Indexers for proper long-term storage. Since disk space is not unlimited on these caching servers we will need to implement log rotation and deletion so we do not fill up the hard disk. Our rsyslog config file already takes care of the log rotation with the template parameter specifying the name of the file as “%$YEAR%-%$MONTH%-%$DAY%.log", however, we still need to clean up the files, so they don’t sit there indefinitely. One possible solution is to use a daily cron job to clean up files in the /rsyslog/ directory that are more than x days old (where x is defined by the organization). Once you have some files in the /rsyslog/ directory, try the following command to see what would potentially be deleted. The command below lists files in the rsyslog directory that are older than two days.

find /rsyslog/ -type f -mtime +1 -exec ls -l "{}" \;

If you are happy with a two-day cache period, add it to a daily cron job (as shown below).  Otherwise feel free to play with the +1 until you are comfortable with what it will delete and use that for your cron job.

/etc/cron.daily/logdelete.sh

find /rsyslog/ -type f -mtime +1 -delete

Splunk Universal Forwarder (UF) Configuration

Splunk Forwarders are very flexible in terms of data ingest. For example, they can create listening ports, monitor directories, run scripts, etc. In this case, since rsyslog is writing the information to a directory, we will use a Splunk UF to monitor those directories and send them to the appropriate indexes and label them with the appropriate sourcetypes.  See our example configuration below.

Note:  Make sure the indexes mentioned below exist prior to trying to send data there. These will need to be created within Splunk.  Also ensure that the UF is configured to forward data to indexers (out of the scope of this write up).

/opt/splunkforwarder/etc/apps/SplunkForwarder/local/inputs.conf 

[monitor:///rsyslog/cisco/] whitelist = \.log$ host_segment=3 sourcetype = cisco:ios index = cisco [monitor:///rsyslog/pan/] whitelist = \.log$ host_segment=3 sourcetype = pan:traffic index = pan_logs [monitor:///rsyslog/unclaimed/] whitelist = \.log$ host_segment=3 sourcetype = syslog index = lastchanceindex

Pro tip:  Remember to restart the Splunk UF after modifying files.  

/opt/splunkforwarder/bin/splunk restart

Conclusion

A simple Splunk search of index=cisco, index=pan_logs, or index=lastchanceindex should be able to confirm that you are now receiving data in Splunk. Keep monitoring the lastchanceindex to move hosts to where they need to go as they come on-line. Moving the hosts is accomplished by editing the rsyslog.conf file and possibly adding another monitor stanza within the Splunk UF config. This process can be challenging to create, but once it is going, it just needs a little care from time to time to make sure that all is well.  We hope you found this article helpful.  Happy Splunking!

Basic Troubleshooting

So you did everything above and you are still not seeing data...  Walk through some of these steps:

1. Ensure all network firewalls permit the traffic
2. Ensure iptables allows traffic to the rsyslog server
3. Run tcpdump on the source to ensure it is sending data
4. Run tcpdump on the rsyslog server to ensure it is receiving data
5. Verify permissions when writing the files to disk
6. Attempt to telnet (or even web browse) to the rsyslog server port to see if anything is written to the directories

References

• https://www.rsyslog.com/storing-messages-from-a-remote-system-into-a-specific-file/ 
• https://serverfault.com/questions/400293/syslog-ip-ranges-to-specific-files-using-rsyslog  
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/s1-debugging_rsyslog 

Troubleshooting Data Sources with Incorrect Times using Splunk

By Tony Lee

Have you ever had a data source that you thought was sending data using the wrong time? This can be a problem since Splunk tries to parse and use the event time instead of the ingest time, this can cause issues when trying to find ingested data. If you suspect this is the case you may be experiencing one of the following scenarios:

• Systems not using NTP that experience clock drift
• Systems using broken or faulty NTP
• Systems using the wrong timezone (ex: Sending events in central time, but specifies GMT)

Depending on the time range selected, this can result in data not showing up within Splunk (or any SIEM) because the data may appear to be in the past or the future.  For example, events that are lagging current time by 5 hours will not show up if "Last 4 hours" is selected for the time range.  In a similar fashion, events that are sent with a future date and time will only show up when the time range selector of "All Time" is selected.

Enough about the problems, let's walk through building one possible solution. As a bonus we provide the dashboard shown below at the bottom of the article.

Figure 1:  Last Communicated Calculator

Dashboard Components

To assist in usability, we provide a drop down input at the top that contains a list of the indexes.  This is list of indexes is populated dynamically. This is derived using the dbinspect command which contains data about existing indexes within Splunk. The following creates the drop down input in the dashboard.

| dbinspect index=* | where NOT match(index, "^_") | table index | dedup index

The upper (host detail) panel consists of columns indicating the host, total count, first written time, last written time and so on--perfect information to determine time issues.  This information can be found using the metadata command which can quickly query info about hosts, sources, and sourcetypes. In this case, we care about the hosts.

| metadata index=<index we care about> type=hosts

The lower panel (a time-based area chart), represents the volume of data at a given time for a given host. We used the tstats command that we covered in previous article, but looks like the following:

| tstats prestats=t count where index=<index we care about> AND host=<host we care about> by host, _time | timechart useother=false count by host


It is certainly noteworthy that every search on this dashboard uses metadata and that's why it is so quick to discover these details. As a result, you will probably notice that there is no time wasted waiting for the search to return as the data renders almost instantly.

Conclusion

Splunk provides decent visibility into various features within Monitoring Console / DMC (Distributed management console), but we found this flexible and customizable dashboard to be quite helpful for gaining additional insight into the last time a host communicated. This can be used to identify, troubleshoot, and finally confirm the time being reported by devices. We hope this article helps you troubleshoot these very frustrating issues. Enjoy!

Dashboard XML code

Below is the dashboard code needed to see the Last Communicated Times for hosts by Index.  Feel free to modify the dashboard as needed:

<form>
  <label>Last Communicated Calculator</label>
  <description>Select an Index (or Indexes) - High Number is bad...</description>
  <fieldset submitButton="true">
    <input type="time" token="time">
      <label>Time Range</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="index">
      <label>Index</label>
      <fieldForLabel>Index</fieldForLabel>
      <fieldForValue>index</fieldForValue>
      <search>
        <query>| dbinspect index=* | where NOT match(index, "^_") | table index | dedup index | sort index</query>
        <earliest>-30d@d</earliest>
        <latest>now</latest>
      </search>
      <valuePrefix>index=</valuePrefix>
      <delimiter> OR </delimiter>
    </input>
    <input type="text" token="host">
      <label>Host</label>
      <default>*</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <title>Hosts</title>
        <search>
          <query>| metadata $index$ type=hosts | dedup host | eval currentTime=now() | eval seconds=now()-lastTime | eval minutes=(seconds/60) | eval hours=(minutes/60) | convert ctime(lastTime) ctime(firstTime) ctime(currentTime) | table host, totalCount, firstTime, lastTime, currentTime, hours, minutes, seconds | sort - seconds | rename hours AS "Last Comm (in hrs)", minutes AS "Last Comm (in mins)", seconds AS "Last Comm (in secs)" | search host=$host$</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">20</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">true</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>Visual (Keep in mind your time range.  Anything beyond the time range will not show up)</title>
        <search>
          <query>| tstats prestats=t count where $index$ AND host=$host$ by host, _time | timechart useother=false count by host</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="charting.chart">area</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
</form>