Showing posts with label query. Show all posts
Showing posts with label query. Show all posts

Sunday, February 9, 2020

Detecting DNS Tunneling Using Subdomain Frequency Analysis

By Tony Lee

For those that may not be familiar, DNS tunneling is the process of embedding data inside DNS requests and responses. It is typically used by a threat actor as a last resort command and control (C2) channel due to the size limitations of DNS packets. One trade-off for the limited size and slow speed is that it is one of the stealthiest and most difficult C2 channels to detect--thus it is quite effective at  maintaining persistence even after other C2 is discovered and eliminated. Due to the prevalence of DNS and its generally trusted (not inspected) status in most networks, most organizations have little-to-no visibility, however this article will cover just one of the many methods of detecting this stealthy C2.  The first step is to collect the DNS logs and send them to a central log aggregation platform such as Splunk--which is what our example uses, but feel free to apply it to your platform of choosing.


Figure 1:  Visual explanation of DNS Tunneling


Query Length

After collecting the logs, we will next look for abnormally long DNS queries. Technically the longest allowable domain name per the RFC is 253 characters (including the dots)--thus we can encode enough data in each fully qualified domain name (FQDN) to constitute C2, but it does require creating longer DNS queries. The example search below assumes that we are storing the DNS logs in an index called dns and log format of Microsoft Windows AD DNS (hence the sourcetype), but this will also work for BIND and other DNS data.  Our example checks for a DNS query length greater than 50 characters--but it can be tuned to whatever works best in your environment. Keep in mind that if the number is too low, you will see too many false positives.  If it is too high, you may miss some tunneling or other strangeness that you want to investigate.  We recommend that you start at 50 and adjust as needed.

index=dns sourcetype="MSAD:NT6:DNS" NOT (query="_ldap*" OR query="*.in-addr.arpa." OR query="*.ip6.arpa") | where len(query)>50

Note:  Be sure to check that your DNS query is actually contained in the "query" field

Subdomain Frequency Analysis

If the above search worked and returned some results, we have met the prerequisites.  Taking the search above a step further we are going to parse out the domain portion and use statistical analysis to give us the following columns:
  • The domain controlled by the attacker
  • Potential number of victims querying this domain
  • Number of subdomains
  • Number of times the attacker-controlled domain was queried
  • List of potential victims
  • List of the subdomains that were queried

index=dns sourcetype="MSAD:NT6:DNS" NOT (query="_ldap*" OR query="*.in-addr.arpa." OR query="*.ip6.arpa") | where len(query)>50 | rex field=query ".*?\.(?<domain>.*)\.$" | stats dc(src_ip) AS NumSourceIPs, dc(query) AS NumSubDomains, count(query) AS DomainQueryCount, values(src_ip) AS SourceIPs, values(query) AS Subdomains by domain | sort - NumSubDomains | where NumSubDomains>1

Figure 1:  Fields provided by the search string above

Now look for anything that stands out as being suspicious.  Keep in mind some of this may be legitimate traffic, but you may also find some known bad domains as you research the results.


Defense Strategy

As mentioned in the introduction, DNS tunneling can be tough to defend against.  However, follow these tips and it will increase the difficulty of attacker success.
  • Keep a simple DNS infrastructure.  The more complicated and chained, the more difficult it is to detect this issue
  • Do not allow hosts to go directly to the Internet for DNS (block all but the DNS servers at the firewall and monitor for any deviation)
  • Send all DNS logs (Microsoft AD, BIND, etc.) to a centralized logging platform
  • Run searches (such as the one provided in this article) to continuously monitor and alert on anomalies
  • Use threat intelligence to augment your DNS tunneling searches to alert against known malicious domains

Conclusion

There are many ways of detecting potential DNS tunneling and this is just one of them. Even if you don't catch a threat actor in your network, you may end up discovering something else strange.  Content distribution networks (CDNs), threat actors, marketing software, spyware, and more may turn up in your search results.  Maybe some or all of those are a concern to your team.  You won't know until you take a look though.  Happy Searching!

References

Good explanation of DNS tunneling found at:  https://hackersterminal.com/dns-tunneling/ -- Kudos for their graphic as well--it is one of the clearest we found on-line.  Nice work.

Wednesday, May 29, 2019

osquery - Part V - Integration

By Tony Lee and Matt Kemelhar

This series on osquery will take us on a journey from stand-alone agents, to managing multiple agents with Kolide Fleet, and then finally onto more advanced integrations and analysis.  We already covered the following topics:

Part I - Local Agent Interaction:  http://securitysynapse.blogspot.com/2019/05/osquery-part-i-local-agent-interaction.html
Part II - Kolide Centralized Management:  http://securitysynapse.blogspot.com/2019/05/osquery-part-ii-kolide-centralized.html
Part III - Queries and Packs:  http://securitysynapse.blogspot.com/2019/05/osquery-part-iii-queries-and-packs.html
Part IV - Fleet Control Using fleetctl - http://securitysynapse.blogspot.com/2019/05/osquery-part-iv-fleet-control-using-fleetctl.html


Even though we now have a centralized management platform that we can manage, reading the query output in the Kolide Fleet UI does not scale to hundreds of thousands of hosts -- thus we need to integrate with a big data analytics platform so we can stack and perform statistical analysis on the data.  In this article, we will examine Kolide Fleet output + Splunk integration.  As a bonus, we are releasing a Kolide Fleet App for Splunk -- free of charge in Splunkbase.  In the first version of the app, it will be able to parse, normalize, and display the following information:

  • Overview information
  • Status Log
  • osquery_info query
  • programs query
  • process_open_sockets query
  • users query
Screenshots of the app are shown below.  Splunk app is available here:  https://splunkbase.splunk.com/app/4518/

Figure 1:  Overview page

Figure 2:  Status Log

Figure 3:  osquery_info page


Expected Kolide Packs and Queries

The first version of the Kolide App for Splunk needs the pack, query names, and output to conform to what is shown in the fleetctl get commands below. For this reason, we are sharing our exported packs and queries here:


Remember in Part IV of this series, we covered how to import the queries and packs using fleetctl:

fleetctl apply -f kolide_splunk_app.yaml 
[+] applied 4 queries
[+] applied 4 packs


Pack and Query details

fleetctl get p
+---------------------------+----------+-------------------------------+
|           NAME            | PLATFORM |          DESCRIPTION          |
+---------------------------+----------+-------------------------------+
| users pack                |          | Query all users               |
+---------------------------+----------+-------------------------------+
| osquery_info pack         |          | Query the version of osquery  |
+---------------------------+----------+-------------------------------+
| process_open_sockets pack |          | Pack for process_open_sockets |
+---------------------------+----------+-------------------------------+
| programs pack             |          | pack for programs             |
+---------------------------+----------+-------------------------------+


fleetctl get q
+----------------------------+------------------------------+--------------------------------+
|            NAME            |         DESCRIPTION          |             QUERY              |
+----------------------------+------------------------------+--------------------------------+
| users query                | Query all users              | SELECT * FROM users            |
+----------------------------+------------------------------+--------------------------------+
| osquery_info query         | Query the version of osquery | SELECT * FROM osquery_info     |
+----------------------------+------------------------------+--------------------------------+
| process_open_sockets query | Query process_open_sockets   | SELECT DISTINCT proc.name,     |
|                            |                              | proc.path, proc.cmdline,       |
|                            |                              | pos.pid, pos.protocol,         |
|                            |                              | pos.local_address,             |
|                            |                              | pos.local_port,                |
|                            |                              | pos.remote_address,            |
|                            |                              | pos.remote_port FROM           |
|                            |                              | process_open_sockets AS pos    |
|                            |                              | JOIN processes AS proc ON      |
|                            |                              | pos.pid = proc.pid;            |
+----------------------------+------------------------------+--------------------------------+
| programs query             | query for programs           | SELECT * FROM programs         |
+----------------------------+------------------------------+--------------------------------+



Kolide Output

Once the Packs and Queries above are imported using the fleetctl apply command above, applied to targets, and scheduled to run, we need to gather the output and send it to Splunk. You might remember that in Part III of this series, we mentioned that we added a statement to our fleet.yaml configuration file to send the results and status output to the following path with log rotation:

filesystem:
  status_log_file: /data/osquery/status.log
  result_log_file: /data/osquery/results.log
  enable_log_rotation: true


This sets us up perfectly to use a Splunk forwarder to send the data to Splunk. If not already completed, download and install the Splunk forwarder here:

https://www.splunk.com/en_us/download/universal-forwarder.html

Once installed, configure the forwarder to send data to your indexers.

Install the Splunk App and Create the Index

In order to prepare for the data's arrival, we now install the Splunk app and create an index for the osquery data:

1) Install the following Kolide Fleet App For Splunk: https://splunkbase.splunk.com/app/4518/
2) Create an index called osquery

If you already had osquery data going to Splunk to a different index and sourcetype, all is not lost.  You can modify the eventtypes.conf file to account for it.

(Optional) Modify eventtypes.conf as Needed

If you already had Kolide setup and sending data to Splunk under a different index and sourcetype name, that's not a problem.  As long as the data is being parsed correctly, we can just modify eventtypes.conf within Splunk to still make all the dashboards function correctly for your index and sourcetype names.  Modify index=osquery to match your index.  Modify sourcetype=osquery:results and osquery:status to match your sourcetypes.


cat eventtypes.conf


[osquery_index]
search = index=osquery

[osquery_status]
search = eventtype=osquery_index sourcetype=osquery:status

[osquery_results]
search = eventtype=osquery_index sourcetype=osquery:results



(Optional) Modify props.conf as Needed

Currently the only two stanzas in props.conf that are used are osquery:results and osquery:status shown below.  Feel free to change the stanza to match the sourcetype if needed.  Minimal parsing is accomplished below:

cat props.conf

## Results log
[osquery:results]
TRUNCATE = 50000
KV_MODE = json
SHOULD_LINEMERGE = 0
category = osquery
pulldown_type = 1
MAX_TIMESTAMP_LOOKAHEAD = 10
TIME_FORMAT = %s
TIME_PREFIX = unixTime\"\:
EVAL-vendor_product = "osquery"
FIELDALIAS-user = decorations.username as user
FIELDALIAS-username = username as user
FIELDALIAS-dest = decorations.hostname as host


## Status log
[osquery:status]
KV_MODE = json
SHOULD_LINEMERGE = 0
category = osquery
pulldown_type = 1
MAX_TIMESTAMP_LOOKAHEAD = 10
TIME_FORMAT = %s
TIME_PREFIX = unixTime\"\:
EVAL-vendor_product = "osquery"
FIELDALIAS-user = decorations.username as user
FIELDALIAS-dest = host as dest

Send logs to Splunk via Splunk forwarder (inputs.conf)

Once our app is installed on the search head, Splunk forwarder is installed on the Kolide host and Kolide is writing the status and results logs to disk, we need to let the fowarder know where to gather the logs. For this, we use the following inputs.conf file:

cat inputs.conf 
[monitor:///data/osquery/results.log]
index = osquery
sourcetype = osquery:results
disabled = 1

[monitor:///data/osquery/status.log]
index = osquery
sourcetype = osquery:status
disabled = 1

If all went as planned, you should see data populating in the Splunk app.  :-)

Conclusion

This article covered how to import the required queries to populate the current version of the Splunk app.  It then explained where the Kolide Fleet logs should appear and how to forward those logs to Splunk.  We covered installing the newly created Kolide Fleet App for Splunk and optionally configure the eventtypes.conf and/or props.conf for any deviation in expected index or sourcetype.  At the end of this effort, you should have data flowing from Kolide Fleet to Splunk properly ingested, parsed, and displayed.  For any questions, please post in the comments section below.  Otherwise, stay tuned additional integration efforts!

Props to the osquery TA for getting us started.


Bonus for the curious reader -- Splunk Magic

Normally, JSON is not the prettiest of data to table in Splunk.  However, we discovered a series of tricks that makes panel and dashboard development scale a little easier.  Our searches in many cases end up looking something like this:

eventtype=osquery_results  name="pack/network_connection_listening/Windows_Process_Listening_Port" | dedup host, _time | spath output=data path=snapshot{} | mvexpand data | rename data as _raw | extract pairdelim="," kvdelim=":" | eval pname=mvindex(name,1) | table _time, host, pname, path, protocol, address, port 


It is a lot of digest all at once, so let's break it down:

  1. Find the data we want:  eventtype=osquery_results  name="pack/network_connection_listening/Windows_Process_Listening_Port"
  2. Get the latest result by host and time:  dedup host, _time
  3. Convert multivalue field into data field, one per line kv pair:  spath output=data path=snapshot{}
  4. Expand multivaue fields into separate events:  mvexpand data
  5. Rename data as _raw since extract only works on _raw:  rename data as _raw
  6. Extract key/value pair (regardless of key names):  extract pairdelim="," kvdelim=":"
  7. Avoid conflict for "name" variable:  eval pname=mvindex(name,1)
  8. Table remaining values extracted:  table <extracted field names>

Figure 4:  Pure joy of JSON data in Splunk

Figure 5:  Data is ready to table after our SPL trickery

Sunday, May 26, 2019

osquery - Part III - Queries and Packs

By Tony Lee and Matt Kemelhar

This series on osquery will take us on a journey from stand-alone agents, to managing multiple agents with Kolide Fleet, and then finally onto more advanced integrations and analysis.  So far, we have already covered the following topics:

Part I - Local Agent Interaction:  http://securitysynapse.blogspot.com/2019/05/osquery-part-i-local-agent-interaction.html
Part II - Kolide Centralized Management:  http://securitysynapse.blogspot.com/2019/05/osquery-part-ii-kolide-centralized.html


Even though we now have a centralized management platform, reading the query output in the Kolide Fleet UI does not scale to hundreds of thousands of hosts -- thus we need to integrate with a big data analytics platform so we can stack and perform statistical analysis on the data.  In order to do that, we first need to cover Query Packs and the resulting logs.

What is Query Pack?

The Kolide Fleet Web UI does an excellent job succinctly describing the query packs in the following manner:

"Osquery supports grouping of queries (called query packs) which run on a scheduled basis and log the results to a configurable destination.

Query Packs are useful for monitoring specific attributes of hosts over time and can be used for alerting and incident response investigations. By default, queries added to packs run every hour (interval = 3600s).

Queries can be run in two modes:

  1. Differential = Only record data that has changed.
  2. Snapshot = Record full query result each time.

Packs are distributed to specified targets. Targets may be individual hosts or groups of hosts called labels.

The results of queries run via query packs are stored in log files for your convenience. We recommend forwarding this logs to a log aggregation tool or other actionable tool for further analysis. These logs can be found in the following locations:

    Status Log: /path/to/status/logs
    Result Log: /path/to/result/logs"


Creating Saved Queries

Packs sound like a great step toward big data integration, but first we need to create a saved search by doing the following (our example below queries users):

In the Kolide Web UI, click Query on the left hand navigation > New Query
  • Query Title:  Users Query
  • SQL:  SELECT * FROM users
  • Description:  Query all users
  • Select Targets:  All Hosts
Click the Save button > Save as New

Figure 1:  Adding a new saved user query

Creating Query Packs

Now that we have a saved query, let's schedule it using a Pack.

Click Packs on the left hand navigation > New Pack

  • Query Pack Title:  Users Pack
  • Query Pack Description:  Query all users

Click the Save Query Pack button

Figure 2:  Creating a new users pack

In the next screen, on the far right hand side, select the Users Query that we created earlier and fill in the fields to define the pack properties:
  • Interval:  60  (Just so we get some data to play with)
  • Platform:  All
  • Minimum version:  All
  • Logging:  Snapshot  (Just so we get some data to play with)

Figure 3:  Defining the User Pack properties

Query Pack Output

With our current minimalist configuration (shown below in fleet.yaml), the packs logs are being sent by default to disk here:
  • /tmp/osquery_result
  • /tmp/osquery_status


cat /opt/fleet/conf/fleet.yaml 

mysql:
  address: 127.0.0.1:3306
  database: kolide
  username: root
  password: toor
redis:
  address: 127.0.0.1:6379
server:
  cert: /opt/fleet/ssl/fleetserver-cert.crt
  key: /opt/fleet/ssl/fleetserver-cert.key
  address: 0.0.0.0:443
auth:
  jwt_key: strong_key
logging:
  json: true


If we wanted to send the logs to a lager drive, we could add the following to our fleet.yaml configuration (the enable log rotation provides 500 Mb or 28 days of data):

filesystem:
  status_log_file: /path/to/drive/osquery/status.log
  result_log_file: /path/to/drive/osquery/result.log
  enable_log_rotation: true

(For our lab environment, we wrote it to:  /data/osquery/)

Just remember to restart the Kolide Fleet service using the following:

service fleet-service restart

For a full list of Fleet configuration options (such as sending to firehose, etc.):
https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md


Conclusion

This article covered how to create saved queries, configure and schedule query packs to run on a regular basis, and how to send this data to a specified file so we can pick up the results and send them to a big data analytics platform.  In the next couple of articles we will cover how to manage the fleet manager and advanced integrations.

Saturday, May 25, 2019

osquery - Part II - Kolide Centralized Management

By Tony Lee and Matt Kemelhar

This series on osquery will take us on a journey from stand-alone agents, to managing multiple agents with Kolide, and then finally onto more advanced integrations and analysis.  We already covered stand-alone local osquery interaction in Part I of this series:

http://securitysynapse.blogspot.com/2019/05/osquery-part-i-local-agent-interaction.html

However, we quickly noticed that it does not scale to hundreds of thousands of hosts -- thus we need a centralized management platform.  In this article, we will examine the freely available Kolide Fleet.

What is Kolide?

Kolide (https://kolide.com/) is a centralized osquery agent management platform.  As of the writing of this article, there are two versions:  Cloud and on-prem Fleet.  Currently Kolide Cloud runs about $6 per endpoint.  However, for our needs, we will kick the tires with the on-prem Kolide Fleet (https://kolide.com/fleet) which is offered free of charge.


Kolide Fleet Dependencies and Installation

Kolide Fleet has a few significant dependencies:
  • *nix based operating system
  • MySQL version 5.7 (or greater) - used as Fleet's primary database
  • Redis - "ingest and queue results of distributed queries, cache data, etc."

Due to these dependencies, setup can be a little painful and time consuming, however we found a pretty awesome Fleet installation script (https://github.com/deeso/fleet-deployment) from Adam Pridgen (https://www.linkedin.com/in/-dso-/) that works great for our lab environment running Ubuntu.

Installation

Follow these steps to get up and running quickly:

git clone https://github.com/deeso/fleet-deployment.git
cd fleet-deployment/fleet-server-install
cp passwords.example passwords.sh

** Using your favorite text editor (such as vim), update the MYSQL_PASS and JWT_KEY variable with the sql password:

vim passwords.sh

Now run the installer script:
bash install.sh

NOTE:  During the SSL certificate creation phase, you will be asked for a "Common Name" / server FQDN (see below) -- be sure to use the server name. 

Ex: Common Name (e.g. server FQDN or YOUR name) []:<ENTER IT HERE>


 It will matter later when you try to connect via fleetctl.  If you do not specify the server name, you will see the following error message upon login attempt:

"error logging in: POST /api/v1/kolide/login: Post https://<hostname>:443/api/v1/kolide/login: x509: certificate is not valid for any names, but wanted to match localhost"


Check on the status of the service:

service fleet-service status

When complete, open a browser and navigate to https://localhost to complete the Kolide setup to specify the user, organization, and Kolide URL.

Figure 1:  Kolide Fleet Setup Complete


Joining an agent to Kolide Fleet

If you installed osquery as a stand-alone during the part I article, feel free to uninstall it.  We now need to install some osquery agents and get them to connect to our Kolide server.

"To connect a host to Kolide Fleet, you have two general options. 

1)  You can install the osquery binaries on your hosts via the packages distributed at https://osquery.io/downloads 

- or -  

2)  You can use the Kolide osquery Launcher

The Launcher is a light wrapper that aims to make running and deploying osquery easier by adding a few features and minimizing the configuration interface. Some features of The Launcher are:

  • Secure autoupdates to the latest stable osqueryd
  • Remote communication via a strongly-typed, versioned, modern gRPC server API
  • a curated kolide_best_practices table which includes a curated set of standards for the modern enterprise"

Source:  https://github.com/kolide/fleet/blob/master/docs/infrastructure/adding-hosts-to-fleet.md


Using Kolide osqery Launcher

For this article, we will use the Kolide osquery Launcher to connect a host to our Kolide Fleet server.  The launcher can be obtained as source or pre-compiled binaries from here:  https://github.com/kolide/launcher/releases

Then you will need to obtain the enrollment secret from the Kolide Fleet Server web UI by clicking on the "Add New Host" link.

Figure 2:  Obtain enrollment secret from the Kolide Fleet web UI

Once you have the launcher binary and enrollment secret, run something similar to the following (where 192.168.21.129 is your Kolide server):

launcher.exe --hostname=192.168.21.129:443 --root_directory=c:\programdata\osquery --enroll_secret=6Ua**snip**rUc --insecure

The host will check in and you will be able to run queries from Kolide Fleet.

Figure 3:  Host checked into Kolide Fleet

To run queries, use the side navigator in the Kolide Fleet UI and click Query > New Query.  Type the SQL query you want to run (autocomplete is present), select the target(s), and click run.  The output from the hosts will be at the bottom of the screen.


Figure 4:  Running a query from Kolide Fleet


Conclusion

At this point you should have the basic building blocks for deploying osquery agents and having them check into Kolide Fleet.  This centralized management is quite powerful.  Being able to view (and export) the data from multiple hosts is also powerful, but viewing the results in this interface is a bit limiting--especially when processing results from thousands of hosts.  In the next couple of articles we will examine fleet control and integration possibilities that will allow processing and stacking the data using a big data analytics platform.