osquery - Part VI - Controlling fleetctl with fleet_helper

By Tony Lee and Matt Kemelhar

** Spoiler alert -- we are releasing fleet_helper.sh in this article **  The download is available here:
https://github.com/TonyLeeVT/fleet_helper/

This series on osquery has taken us on a journey from stand-alone agents, to managing multiple agents with Kolide Fleet, and then finally onto more advanced integrations and analysis.  We already covered the following topics:

Part I - Local Agent Interaction:  http://securitysynapse.blogspot.com/2019/05/osquery-part-i-local-agent-interaction.html
Part II - Kolide Centralized Management:  http://securitysynapse.blogspot.com/2019/05/osquery-part-ii-kolide-centralized.html
Part III - Queries and Packs:  http://securitysynapse.blogspot.com/2019/05/osquery-part-iii-queries-and-packs.html
Part IV - Fleet Control Using fleetctl - http://securitysynapse.blogspot.com/2019/05/osquery-part-iv-fleet-control-using-fleetctl.html
Part V - Integration - http://securitysynapse.blogspot.com/2019/05/osquery-part-v-integration.html


However, back in Part IV (Fleet Control Using fleetctl), we covered the fleetctl command line tool used to interface with the Fleet API.  We also mentioned the following issue:

Exporting All Queries and Packs (Previous Statement)

Unfortunately, we cannot find a single command or option within fleetctl to export all queries and packs to re-import them elsewhere.  The documentation states that fleetctl functions similarly to kubectl, but the -o yaml option does not appear to be implemented yet...

We also mentioned we would expand on this, so we are by releasing a helper tool called fleet_helper.

Exporting packs and queries is critical for sharing them and enabling faster integration with our Kolide Fleet App for Splunk - https://splunkbase.splunk.com/app/4518/#/details

We submitted a feature request (https://github.com/kolide/fleet/issues/2052), but figured we could prototype the desired functionality in bash and release it in the meantime.

fleet_helper Usage

The feet_helper tool (https://github.com/TonyLeeVT/fleet_helper/) is relatively simple and just over 200 lines (including comments).  Two of the functions are straight from the fleetctl tool, but the rest perform some sort of output manipulation or iteration over data to achieve the desired output.

Requirements:

• fleetctl must be setup and authenticated (post fleetctl login)
• sed, grep, tail, cut, echo commands

Help Menu:

Activate the help menu by providing no options, -h, or --help.

Usage:  ./fleet_helper.sh <function>

Note: All functions output to stdout.  Redirect to .yaml files as needed.

Possible functions:

 listpacks - Lists pack names only within the Kolide Fleet instance

 listqueries - Lists query names only within the Kolide Fleet instance

 exportpacks - Exports all packs in yaml format

 exportqueries - Exports all queries in yaml format

 exportpack <pack_name> - Exports a specified pack in yaml format

 exportquery <query_name> - Exports a specified query in yaml format

 exportpackquery <pack_name> - Exports a pack and all associated queries in yaml format

 exportall - Warning!  This exports all packs and all queries in yaml format

Listing Packs and Queries

The feet_helper can list packs and queries in simple text format using the following two functions:

 listpacks - Lists pack names only within the Kolide Fleet instance

 listqueries - Lists query names only within the Kolide Fleet instance

./fleet_helper.sh listpacks

users pack                   
osquery_info pack            
process_open_sockets pack    
programs pack                
network_connection_listening 

Exporting (All or select) Packs and Queries

The feet_helper can export all or select packs and queries using the following four functions:

 exportpacks - Exports all packs in yaml format

 exportqueries - Exports all queries in yaml format

 exportpack <pack_name> - Exports a specified pack in yaml format

 exportquery <query_name> - Exports a specified query in yaml format

./fleet_helper.sh exportqueries
apiVersion: v1
kind: query
spec:
  description: Query all users
  name: users query
  query: SELECT * FROM users
---
apiVersion: v1
kind: query
spec:
  description: Query the version of osquery
  name: osquery_info query
  query: SELECT * FROM osquery_info
--SNIP--

Exporting a Specific Pack and All Associated Queries

This feature is critical to sharing queries within a pack.  The feet_helper can export a pack and all associated queries as well using the following function:

 exportpackquery <pack_name> - Exports a pack and all associated queries in yaml format


./fleet_helper.sh exportpackquery "network_connection_listening"
apiVersion: v1
kind: pack
spec:
  id: 14
  name: network_connection_listening
  queries:

--SNIP--

---

Snaphost_Windows_Process_Listening_Port

apiVersion: v1

kind: query

spec:

  description: Returns the Listening port List - ATT&CK T1043,T1090,T1094,T1205,T1219,T1105,T1065,T1102

  name: Snaphost_Windows_Process_Listening_Port

  query: select p.name, p.path, lp.port, lp.address, lp.protocol  from listening_ports

    lp LEFT JOIN processes p ON lp.pid = p.pid WHERE lp.port != 0 AND p.name != '';

Exporting All Packs and Queries

This feature is useful for backing up packs and queries or sharing an entire environment with someone else.  The feet_helper can export all packs and queries to yaml using the following function:

 exportall - Warning!  This exports all packs and all queries in yaml format

./fleet_helper.sh exportall > exportall.yaml

We won't show the output on this function because the output is quite lengthy -- hence why we recommend redirecting the output to a file.

Conclusion

While fleetctl is a great tool, some features that assist in pack and query sharing are missing.  This quick prototype should help provide a stop gap and model new features.  Feel free to let us know about any bugs over at github.  Feel free to leave any comments below.  Thanks for reading!

osquery - Part IV - Fleet Control Using fleetctl

By Tony Lee and Matt Kemelhar

This series on osquery will take us on a journey from stand-alone agents, to managing multiple agents with Kolide Fleet, and then finally onto more advanced integrations and analysis.  We already covered the following topics:

Part I - Local Agent Interaction:  http://securitysynapse.blogspot.com/2019/05/osquery-part-i-local-agent-interaction.html
Part II - Kolide Centralized Management:  http://securitysynapse.blogspot.com/2019/05/osquery-part-ii-kolide-centralized.html
Part III - Queries and Packs:  http://securitysynapse.blogspot.com/2019/05/osquery-part-iii-queries-and-packs.html

Now that we have a centralized osquery management platform using Kolide Fleet, we need to learn how to manage the manager.  This may not be difficult with a single Kolide Fleet instance, but keep in mind you could also run multiple load balanced Kolide Fleet instances.  Plus exporting and importing queries and packs are helpful tasks if you want to share or consume queries and packs with/from others.

Managing Fleets using fleetctl

Fortunately, there is a command-line tool that interacts with the Fleet API called fleetctl.  While it isn't perfect, at least it is a start.  This article will hopefully provide some additional tips on usage and limitations.


Official documentation can be found at the link below, but we will give you a primer with examples:  

https://github.com/kolide/fleet/blob/master/docs/cli/README.md


In order to use fleetctl, we first need to set it up (replace <hostname> with the hostname of your server):

Setup
fleetctl config set --address https://<hostname>:443

fleetctl login
Log in using the standard Fleet credentials.
Email: <my email address>
Password: 
[+] Fleet login successful and context configured!

If you see the following error message while attempting to log in, it is probably because you are using a self-signed cert without a common name:
"error logging in: POST /api/v1/kolide/login: Post https://<hostname>:443/api/v1/kolide/login: x509: certificate is not valid for any names, but wanted to match localhost"

Please see our previous article for details, however hopefully this will correct this issue for a self-signed cert:
fleetctl config set --rootca /opt/fleet/ssl/fleetserver-cert.crt

Otherwise, now that you are logged in, try something easy (this value should be familiar from the GUI):

Quick Test
fleetctl get enroll-secret
6U**********rUc


To see other options, use -h for help.

Help Menu
fleetctl -h

NAME:
   fleetctl - CLI for operating Kolide Fleet

USAGE:
   fleetctl [global options] command [command options] [arguments...]

VERSION:
   2.1.0

COMMANDS:
     apply    Apply files to declaratively manage osquery configurations
     delete   Specify files to declaratively batch delete osquery configurations
     setup    Setup a Kolide Fleet instance
     login    Login to Kolide Fleet
     logout   Logout of Kolide Fleet
     query    Run a live query
     get      Get/list resources
     config   Modify how and which Fleet server to connect to
     convert  Convert osquery packs into decomposed fleet configs
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version


Get Command Syntax (Queries, Packs, Labels, Host, Enroll-Secret)
fleetctl get
NAME:
   fleetctl get - Get/list resources

USAGE:
   fleetctl get command [command options] [arguments...]

COMMANDS:
     queries, query, q  List information about one or more queries
     packs, pack, p     List information about one or more packs
     labels, label, l   List information about one or more labels
     options            Retrieve the osquery configuration
     hosts, host, h     List information about one or more hosts
     enroll-secret      Retrieve the osquery enroll secret

OPTIONS:
   --help, -h  show help

Get A List of All Packs -- NOTE:  fleetctl does not currently provide an option to change the format
fleetctl get p
+---------------------------+----------+-------------------------------+
|           NAME            | PLATFORM |          DESCRIPTION          |
+---------------------------+----------+-------------------------------+
| users pack                |          | Query all users               |
+---------------------------+----------+-------------------------------+
| osquery_info pack         |          | Query the version of osquery  |
+---------------------------+----------+-------------------------------+
| process_open_sockets pack |          | Pack for process_open_sockets |
+---------------------------+----------+-------------------------------+
| programs pack             |          | pack for programs             |
+---------------------------+----------+-------------------------------+

(Note:  For a list of the queries, change the p to a q)

Get a Pack as Config File

For information about the config file format, please see this link:
https://github.com/kolide/fleet/blob/master/docs/cli/file-format.md

By specifying the exact query or pack in the "get" command, we get the properly formatted output, but we cannot find a way to say all packs or all queries (*).

fleetctl get p "users pack"

apiVersion: v1
kind: pack
spec:
  description: Query all users
  id: 1
  name: users pack
  queries:
  - description: ""
    interval: 60
    name: Users Query
    platform: ""
    query: users query
    removed: false
    snapshot: true
    version: ""
  targets:
    labels: null

(Note:  This also works for query names too)

Exporting All Queries and Packs

Unfortunately, we cannot find a single command or option within fleetctl to export all queries and packs to re-import them elsewhere.  The documentation states that fleetctl functions similarly to kubectl, but the -o yaml option does not appear to be implemented yet... so we had to perform some Linux trickery to get a list of all packs and queries in a non-tabled format.

To get a list of all of the packs, we used:

fleetctl get p | grep -v '+' | tail -n +2 | cut -f 2 -d '|' | sed 's/^.//' | sed -r '/^\s*$/d'

users pack                
osquery_info pack         
process_open_sockets pack 
programs pack           

To get a list of all of the queries, we just changed the first p to a q:

fleetctl get q | grep -v '+' | tail -n +2 | cut -f 2 -d '|' | sed 's/^.//' | sed -r '/^\s*$/d'

users query                
osquery_info query         
process_open_sockets query 
programs query           

We will try to expand on this when we get more time...

Import a Simple Query

If you happen to have a simple query such as the one below, you can import it using the fleetctl apply command:

cat querytoimport.yaml 

apiVersion: v1
kind: query
spec:
  description: Query the version of osquery
  name: osquery_info query
  query: SELECT * FROM osquery_info

fleetctl apply -f querytoimport.yaml
[+] applied 1 queries

This should now show up in the Web UI.

Importing Third Party Query Packs

Now that we know how to create a query pack, it does not mean that we need to create them all by ourselves.  There is already a significant amount of content being freely shared that we can leverage, such as the following:

• Mapping MITRE Att&ck Framework in osquery - https://github.com/teoseller/osquery-attck
• Palantir Security - https://github.com/palantir/osquery-configuration 
• osquery packs - https://github.com/facebook/osquery/tree/experimental/packs

Note:  Some of the content above is provided as an osquery pack and not a fleet config that can be directly imported using fleetctl apply.  For these, use fleetctl convert first.  For example:

wget https://raw.githubusercontent.com/teoseller/osquery-attck/master/network_connection_listening.conf

fleetctl convert -f network_connection_listening.conf > network_connection_listening.yaml

fleetctl apply -f network_connection_listening.yaml 
[+] applied 4 queries
[+] applied 1 packs

 More on this later... but we wanted to introduce the concept here.

Conclusion

This article covered how to manage Kolide Fleet using a tool called fleetctl.  We covered how to setup fleetctl, how to get a list of packs, queries, labels, users, and even how to import third part query packs.  You may have noticed that each article lays the groundwork to build an integration which will really weaponize Kolide Fleet...  As such, our next article we will use all of the information from the first four articles to create an integration with Splunk!  :-)